Chapter 32: GitOps and Progressive Delivery

32.1 Why.

Welcome to Chapter 32.

I am Ravindra Bagale.

Raju had no engineering degree. He had a cluster and a habit of editing a live deployment because the change was small. By evening the live cluster and the Git repository disagreed, and he could not say which one was true. Guru-ji pointed at Git.

"That file is the desired state," Guru-ji said. "A controller reconciles the world toward it. If you change the world by hand, the controller will either undo you or, worse, learn to follow a lie. Progressive delivery is how a new desired state arrives in slices, with a way back."

Raju liked the speed of a click. "Suppose we are Netflix and a bad digest reaches one percent, then stops, because the error budget moved. Suppose we are Amazon and a controller's credential can change every cluster because the install was done as a superuser. Suppose we are Swiggy or Zomato and two teams share a repo, and one team's pull request can retarget the other's production namespace. Suppose we are Paytm or PayFast and a secret is committed next to the manifest so the sync would be easy. Suppose we are Infosys and the client asks who merged the desired state and whether the artifact was signed. Suppose we are BHIM UPI or WhatsApp-shaped systems and a rollback is a guess instead of the previous Git revision. The controller is a production admin. Treat it like one."

32.2 What.

GitOps principles. The desired state is declarative and versioned. Changes arrive as reviewed commits. A controller inside the cluster, or beside it, continuously makes the running state match Git. Drift is a bug you can see. You do not shell into production to "just fix" a replica count unless break-glass is logged and then returned to Git. The benefit for security is auditability. The risk is that Git and the controller become the new crown jewels. Protect them as you protected the pipeline in Chapter 11.

Controllers, in concept. Tools such as Argo CD and Flux implement this loop. You do not need every feature. You need the idea. An application or a kustomization points at a path and a revision. The controller has credentials to apply to the cluster. Those credentials must be limited to the namespaces that application owns. A controller that is cluster-admin for every team is a single key to the city. Project or tenant boundaries in the controller are part of multi-tenancy from Chapter 26.

RBAC and project isolation. Humans get access to Git. The controller gets access to the cluster. A developer who can push to a production branch can change production, so branch protection and a second reviewer stay. A developer who should not touch another team's namespace must not have a controller project that allows it. Separate repositories or clearly separated folders with separate permissions. Review who can edit the controller's own configuration. That meta-configuration is more powerful than any one app.

Sync permissions. The identity that applies manifests should not be the identity that builds images, and neither should be a personal laptop key. Use a workload identity for the controller. Scope it. If the controller can create roles and bind them, a malicious manifest can widen power. Restrict which kinds of resources each project may apply. Policy before reconciliation, using the same policy-as-code habit as Chapter 16, should reject privileged pods, host paths, and public services your standard forbids. The rejection happens before the sync, not in a sad report after.

Signed commits and signed artifacts. Git tells you who proposed a change if your identity provider and signed commits are real. A stolen laptop session can still push if you have not bound pushes to hardware-backed keys and reviews. Artifacts the manifest points at should be signed and pinned by digest, not by a floating tag. Chapter 13's provenance belongs in the deploy path. The controller or an admission policy checks the signature. A tag named stable is not a signature.

Secrets and Git. Do not commit raw secrets so the reconcile is convenient. Commit a reference: an external secret name, a sealed blob encrypted to a key the cluster holds, or a short-lived injection at runtime. The repository is copied to laptops and to CI logs. Treat it as public to insiders. Rotate anything that was committed anyway, and purge it from history with a plan, not with a shrug. The CircleCI-shaped lesson later in this chapter is the same shape: delivery systems store the keys to other systems.

Drift and out-of-band change. If someone scales by hand, the controller should put the scale back or raise a loud drift alert, depending on the mode you chose. Pick a mode on purpose. Auto-heal is safer when Git is trustworthy. A freeze during an incident is a conscious pause, recorded, with a return to Git before you go home. Two sources of truth are no source of truth.

Progressive delivery. Blue-green keeps two environments and switches. Canary sends a slice of traffic to the new digest. Feature flags hide a behavior without a new deploy, and the flag service is itself a control plane to secure. Automated rollback watches the indicators from Chapter 27: errors, latency, and a security indicator such as a sudden deny spike that means you broke good users or opened a bad path. Rollback targets the previous signed digest, which is still in the registry. Practice it when nothing is on fire.

Supply chain at deploy time. Build-time scans are not enough if the manifest can point at an unscanned tag. Verify at the door: signature, digest, and a policy result. Promotion across environments is a Git change from staging to production, reviewed, not a registry copy done by hand with admin credentials. Infosys-shaped evidence is the merge commit plus the sync status plus the running digest. Those three should match.

A teaching sample: a desired state note. Lab only.

app: lab-orders
repo: git path clusters/lab/orders, protected branch, two-person review
image: registry/lab-orders@sha256:... signed
controller: may apply only namespace lab-orders
forbidden: host network, privileged, cluster-admin role bindings
rollout: 10 percent canary, auto-rollback on error burn
secrets: external reference, not a literal value

Seminar notes. Raju asked whether Argo CD or Flux was the secure one. Guru-ji said the secure one is the one whose credential you can explain. Both families reconcile Git to a cluster. They differ in how you structure projects, how you separate tenants, and how you plug policy into the loop. Learn one in the lab. Learn the control objective so you can recognize the other. The objective is a reviewed desired state, a narrow apply identity, a check before apply, and a recorded sync. If a tool tempts you to click a sync from a laptop with a personal kubeconfig that is cluster-admin, the tool is not the problem. The kubeconfig is.

Signed commits bind a change to an identity your organization enrolled. They do not survive a stolen session unless the key is hardware-backed and the review is real. Signed images bind a digest to a build identity. Admission or the controller should refuse what is unsigned. Progressive delivery chooses how much of the city sees the new digest. A blue-green cutover is a single switch after the green side is healthy. A canary is a percentage and a watch on the indicators. Feature flags are a third control plane. Secure the flag service: who may turn on a payment-shaped behavior, whether the flag is reviewed, whether a kill switch exists that does not require a new image. Netflix-shaped rollouts and Swiggy-shaped meal rushes want the slice first. Paytm-shaped and PayFast-shaped changes want the slice and a second reviewer. The mechanism can be the same module with a stricter policy on the payment folder.

Drift is either healed or paged. Healing is right when Git is the authority and the live edit was a mistake. Paging is right during an incident freeze you declared. Write which mode the lab is in. An undeclared mix, where some namespaces heal and some silently accept console edits, will confuse the evidence packet. Infosys-shaped reviewers will ask which mode you run. Answer with the config in Git, not with a memory.

Secrets in GitOps fail in a familiar way. A sealed secret or an external reference keeps raw values out of the repo. The decryption key or the store role is then a crown jewel, scoped per namespace. A repo that everyone can read should not be able to request every secret in the account. Separate the secret paths by service. Automated rollback signals must be hard to fake and easy to trust: error rate, latency, and a synthetic probe you control. Do not let a single noisy log line roll the city back and forth. Do let a burned error budget stop the promotion. Supply-chain verification at deploy time closes the gap between a scan on Tuesday and a tag someone moved on Wednesday. Digest plus signature plus policy. The running object must show the digest you reviewed. If it shows a tag, you are not done.

The controller's own upgrade is a change to the control plane. Canary that too, in the sense of a lab cluster first, then production-shaped. A controller bug that applies too much is an architecture incident. Keep the version pinned. Read the changelog the way you read a base-image changelog in Chapter 25. When a known-exploited issue touches the controller, it is on the emergency lane, because the controller is an admin. That sentence alone justifies the hour you spend scoping its role until the policy looks dull.

One more pass. Promotion across environments is a pull request from a staging folder to a production folder, or a commit that changes a digest in a production path. It is not a person with cluster-admin copying an image and hoping. The pull request shows the old digest, the new digest, the signature status, and the policy result. A second person approves. The controller syncs. The canary watches. The evidence packet is those three lines again: merge, sync, running digest. If the canary burns the budget, Git reverts and the controller returns the city to the previous digest without a laptop session. Practice that revert when you are calm. A rollback you have never done is a runbook you have only typed.

Out-of-band changes during an incident get a ticket and a sunset. Before the incident channel closes, the live state is either returned to Git or Git is updated to match what you intentionally kept. Leaving the cluster drifted overnight is how the next sync becomes a surprise outage or a surprise undo of a hotfix. Write that sunset into the Chapter 27 incident card. GitOps does not replace incident judgment. It replaces forgotten judgment. Zomato-shaped and WhatsApp-shaped services can move quickly and still end the day with Git telling the truth. Speed that cannot be reconciled is not progressive delivery. It is drift with confidence.

Flags and slices. A feature flag can hide unfinished behavior, and it can also hide a security control that someone turned off. Treat a flag that disables signature checks or widens a role as a forbidden flag. The paved progressive path allows flags for product behavior and routes security behavior through policy and Git. Canary percentages should be explicit in the desired state so a "temporary" one hundred percent is visible in review. Blue-green is the right shape when you need a fast switch and a warm previous stack. Canary is the right shape when the indicator needs real traffic to tell the truth. Use one in the lab and write why. Do not use both in a way you cannot draw. The drawing is the control.

The running digest, the Git revision, and the approver are the three facts a later reader needs. If one is missing, the progressive story is only a feeling of speed.

32.3 How.

1. Create a lab Git repository for desired state and protect the main branch.
2. Put one namespace manifest and one deployment in a folder for a single app.
3. Pin the image by digest and sign it with the lab key from earlier chapters.
4. Install a GitOps controller in the lab with a role scoped to that namespace only.
5. Prove the controller cannot create objects in a second namespace.
6. Apply a policy that rejects privileged pods. Commit a privileged pod on a branch and show the sync or the admission refuses it.
7. Remove the bad manifest.
8. Store a fake secret outside Git and reference it from the manifest.
9. Search the repo for accidental literals and keep the search in CI.
10. Require a second lab identity to approve a production-folder change.
11. Sync a good version and record the Git revision and the running digest.
12. Change a replica by hand and show drift detection or auto-heal.
13. Return the change through Git so the two match again.
14. Configure a canary or a blue-green step that starts at a small slice.
15. Point the rollback signal at an error indicator you can trip with a harmless bad config.
16. Trip it, watch the slice stop, and confirm the previous digest serves traffic.
17. Revert the Git commit so desired state matches what is running.
18. Document who can edit the controller configuration.
19. Log a break-glass apply, then undo it via Git before you finish the lab.
20. Export the three-line evidence: merge, sync result, running digest.
21. Stop. Do not point a controller, a canary, or a policy test at Swiggy, Paytm, Amazon, BHIM UPI, WhatsApp, Zomato, Netflix, Infosys, PayFast, or any cluster you do not own.

32.4 A real public incident.

Name. CircleCI security incident disclosed on 4 January 2023.

Date. CircleCI publicly warned customers on 4 January 2023. The company's later incident report said customer secrets were taken on 22 December 2022, and that the investigation traced the path to an engineer's compromised machine in mid-December 2022.

Source. CircleCI, "CircleCI security alert: Rotate any secrets stored in CircleCI," 4 January 2023, and "CircleCI incident report for January 4, 2023 security incident," published after the initial alert. This chapter does not describe the malware or the session theft, and it does not repeat customer-impact indicators.

What happened. CircleCI is a delivery system. Customers stored environment variables, tokens, and keys there so pipelines could deploy. CircleCI reported that an unauthorized party reached production data, including those secrets, and told customers to rotate them and to review their own logs for misuse. A delivery platform had become a vault. GitOps controllers and progressive-delivery tools sit in the same chair. They hold, or they can mint, the power to change production and often the power to read the secrets they inject. A compromise of that chair is not a leak of one repository. It is a leak of every environment the chair can touch. The customer action that mattered was rotation and review, not curiosity about the intrusion. The design action that matters here is to keep that chair's power small, to avoid long-lived cloud keys in the delivery system, to alert on unexpected syncs and secret reads, and to make sure Git history is not a second copy of the vault.

How to stop.

1. Rotate secrets that the delivery system could see, starting with production deploy credentials.
2. Revoke tokens and keys the vendor tells you were in scope.
3. Review logs on the systems those secrets open, for the window the vendor names.
4. Pause unattended production syncs until credentials are fresh, if your risk says so.
5. Preserve your logs. Do not probe the vendor.

How to prevent it again.

1. Scope controller and pipeline identities to one namespace or one account role.
2. Keep raw secrets out of Git and out of overly broad delivery variables.
3. Prefer short-lived workload identity over long-lived keys stored in the delivery product.
4. Alert on drift, on new credentials, and on syncs outside change windows.
5. Require signed artifacts and reviewed Git before production reconcile.
6. Practice rotation and rollback in a lab you own.

This is for learning only, and only inside a system you are allowed to test.

32.5 A simple picture.

A reviewed Git commit is the desired state. Policy runs before sync. The controller's credential is narrow. A canary watches the budget and can return to the previous digest.

32.6 A short animation.

A hand edit drifts the live box. The controller pulls it back to the Git revision. The digest matches again.

32.7 Conclusion.

Raju reverted the hand edit and let the controller finish. The canary died small. Guru-ji marked the controller role in the notebook. "You have the pieces. In Chapter 33 we draw the whole city: identity, network, data, and control, and we decide what is shared and what each team owns."

• Git is the desired state. The controller is an admin to be scoped.
• Policy and signatures happen before reconcile, not as a story afterward.
• Progressive delivery spends error budget in slices and keeps a previous digest.
• CircleCI's alert on 4 January 2023 is the delivery-system-as-vault lesson.
• Own lab only.

END OF CHAPTER 32 MASTER CLASS