Chapter 64: Kubernetes Updates and Releases: Rolling Updates, Rollback, Blue-Green, Canary and Argo Rollouts

64.3 Rollback: undo a bad release in seconds

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Even with good tests, some releases go wrong in production. A new build of the PhonePe payments page may pass every check and still throw errors for one bank's customers. When that happens the first goal is not to find the bug. The first goal is to stop the damage, and the fastest way is to go back to the version that worked. Kubernetes makes this a one-line command because it keeps old ReplicaSets around. kubectl rollout undo deployment/NAME takes the pod template of the previous revision and applies it again. Under the hood this is just another rolling update: the old ReplicaSet is scaled up and the bad one down, with the same maxSurge, maxUnavailable and readiness rules from 64.1. The revision number does not go back; instead the old revision is renumbered as the newest one. So if revision 5 was bad and you undo to 4, the history shows 1, 2, 3, 5 and a new 6 that holds the old template. --to-revision=N lets you jump further back. A rollback only restores the pod template. It does not undo changes to ConfigMaps, Secrets, Services, database schemas or anything outside the Deployment. If the bad release also changed a ConfigMap that the old code cannot read, rolling back the Deployment alone will not help. That is why config should be versioned together with the code, for example as a ConfigMap whose name includes a hash, and why database migrations must stay backward compatible. There is also a GitOps view. If your manifests live in git and Argo CD or Flux applies them, a kubectl rollout undo will be overwritten by the next sync. In that world you roll back by reverting the commit in git. kubectl undo is still useful as an emergency brake, but the git revert must follow quickly. Finally, rollbacks should be fast and boring. Teams decide in advance what triggers one: error rate above a set level, latency too high, or a failing smoke test. In this lesson we roll back a broken release, jump to an older revision, see why a ConfigMap change breaks a naive rollback, and write a small script that rolls back automatically when a smoke test fails.

What: every term in plain English

• Rollback — Returning a workload to a previous known-good version.
• rollout undo — kubectl command that re-applies the previous pod template.
• --to-revision — Rolls back to a specific revision number.
• Revision renumbering — The restored revision becomes the newest number.
• rollout history — Lists revisions and change-cause.
• Smoke test — A quick check right after deploy that the app basically works.
• Automatic rollback — Script or tool that undoes a release when checks fail.
• Versioned ConfigMap — ConfigMap whose name changes with its content, so old pods keep the old one.
• Expand and contract — Migration pattern that keeps the schema usable by old and new code.
• Git revert — GitOps rollback: undo the commit that made the change.
• MTTR — Mean time to recovery; rollbacks are the biggest lever on it.
• rollout pause — Freezes a rollout half-way, useful while you decide.

How: step by step

1. Deploy three good revisions. History with change-cause.
2. Release a broken version. Errors in the loop.
3. Undo. Back to the last good version.
4. Read history. See renumbering.
5. Jump to an older revision. --to-revision.
6. Break a rollback with a ConfigMap change. See the limit of undo.
7. Fix with a versioned ConfigMap. Rollback works again.
8. Script an automatic rollback. Smoke test decides.
9. Clean up. Ready for 64.4.

Example 1: Undo a broken release

Goal. Undo a broken release.

pay.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pay
  namespace: undo
  annotations:
    kubernetes.io/change-cause: "pay v1: first release"
spec:
  replicas: 3
  revisionHistoryLimit: 5
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: pay
  template:
    metadata:
      labels:
        app: pay
    spec:
      containers:
        - name: web
          image: hashicorp/http-echo:1.0
          args:
            - -listen=:8080
            - -text=pay v1
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            periodSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: pay
  namespace: undo
spec:
  selector:
    app: pay
  ports:
    - port: 80
      targetPort: 8080

ver.sh

V=$1; MSG=$2
kubectl patch deploy pay -n undo --type=json -p "[{\"op\":\"replace\",\"path\":\"/spec/template/spec/containers/0/args/1\",\"value\":\"-text=$V\"}]"
kubectl annotate deploy pay -n undo kubernetes.io/change-cause="$MSG" --overwrite
kubectl rollout status deploy/pay -n undo

What each line does

• change-cause — Every revision gets a message, so history is readable.
• revisionHistoryLimit 5 — Keeps enough history to roll back several steps.
• ver.sh — Changes the reply text, which is a template change, and labels it.
• Broken release — We simulate it with a reply text 'ERROR 500', which our loop treats as a failure.
• undo — Re-applies the previous template through a normal rolling update.

Green output

$ kubectl create ns undo && kubectl apply -f pay.yaml
$ bash ver.sh "pay v2" "pay v2: UPI intent"
$ bash ver.sh "pay v3" "pay v3: new bank list"
$ bash ver.sh "ERROR 500" "pay v4: refactor"
$ kubectl run c -n undo --rm -i --image=busybox:1.36 --restart=Never -- wget -q -O- http://pay
ERROR 500
$ kubectl rollout undo deploy/pay -n undo
deployment.apps/pay rolled back
$ kubectl rollout status deploy/pay -n undo
deployment "pay" successfully rolled out
$ kubectl run c -n undo --rm -i --image=busybox:1.36 --restart=Never -- wget -q -O- http://pay
pay v3
$ kubectl rollout history deploy/pay -n undo
REVISION  CHANGE-CAUSE
1         pay v1: first release
2         pay v2: UPI intent
4         pay v4: refactor
5         pay v3: new bank list

Expected result. Users went back to v3 within seconds. Revision 3 disappeared from the list and came back as 5, because a rollback is itself a new revision built from the old template.

Example 2: Jump to an older revision, and see what undo does not restore

Goal. Jump to an older revision, and see what undo does not restore.

cm-v1.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: pay-config
  namespace: undo
data:
  BANKS: "sbi,hdfc,icici"

use-cm.sh

kubectl set env deploy/pay -n undo --from=configmap/pay-config
kubectl annotate deploy pay -n undo kubernetes.io/change-cause="pay: read banks from config" --overwrite
kubectl rollout status deploy/pay -n undo
kubectl create configmap pay-config -n undo --from-literal=BANK_LIST_V2='[{"id":"sbi"}]' --dry-run=client -o yaml | kubectl apply -f -

What each line does

• --to-revision=2 — Goes straight to v2, skipping v3.
• pay-config — A shared ConfigMap the app reads as env vars.
• Changing it in place — Old and new revisions both point to the same name.
• Undo after a config change — Restores the template, but the ConfigMap content stays new.
• Lesson — Version config with the code, or a rollback is only half a rollback.

Green output

$ kubectl rollout undo deploy/pay -n undo --to-revision=2
$ kubectl run c -n undo --rm -i --image=busybox:1.36 --restart=Never -- wget -q -O- http://pay
pay v2
$ kubectl apply -f cm-v1.yaml && bash use-cm.sh
$ kubectl rollout undo deploy/pay -n undo
$ POD=$(kubectl get pod -n undo -l app=pay -o name | head -1)
$ kubectl exec -n undo $POD -- env | grep -i bank
BANK_LIST_V2=[{"id":"sbi"}]

Expected result. The second undo restored the template that reads BANKS, but the ConfigMap had been overwritten with a new key, so the pod sees BANK_LIST_V2 and no BANKS. Old code plus new config is a common cause of failed rollbacks.

Example 3: Versioned config and an automatic rollback script

Goal. Versioned config and an automatic rollback script.

cm-hash.sh

H=$(printf 'BANKS=sbi,hdfc,icici' | sha256sum | cut -c1-8)
kubectl create configmap pay-config-$H -n undo --from-literal=BANKS=sbi,hdfc,icici
kubectl set env deploy/pay -n undo --from=configmap/pay-config-$H --overwrite
kubectl annotate deploy pay -n undo kubernetes.io/change-cause="pay: config $H" --overwrite
kubectl rollout status deploy/pay -n undo

safe-release.sh

V=$1
bash ver.sh "$V" "pay $V"
OK=$(kubectl run smoke -n undo --rm -i --image=busybox:1.36 --restart=Never -- sh -c 'n=0; for i in 1 2 3 4 5 6 7 8 9 10; do wget -q -O- http://pay | grep -q ERROR || n=$((n+1)); done; echo $n' | tail -1)
echo "smoke passed $OK/10"
if [ "$OK" -lt 10 ]; then
  echo "smoke failed, rolling back"
  kubectl rollout undo deploy/pay -n undo
  kubectl rollout status deploy/pay -n undo
  exit 1
fi

What each line does

• pay-config-HASH — Each config version has its own name; old revisions keep pointing at the old one.
• sha256sum — Short content hash so the name changes only when content changes.
• safe-release.sh — Releases, runs 10 smoke requests, and undoes on any failure.
• -lt 10 — Shell test for 'less than', written as a flag, not a symbol.
• exit 1 — Lets CI mark the pipeline as failed.

Green output

$ bash cm-hash.sh
$ bash safe-release.sh "pay v5"
smoke passed 10/10
$ bash safe-release.sh "ERROR 500"
smoke passed 0/10
smoke failed, rolling back
deployment.apps/pay rolled back
deployment "pay" successfully rolled out
$ kubectl run c -n undo --rm -i --image=busybox:1.36 --restart=Never -- wget -q -O- http://pay
pay v5

Expected result. With versioned config, rollbacks bring back both code and config together. The script turned a bad release into a short blip with no human in the loop, which is the idea that Argo Rollouts automates in 64.6.

Every command, step by step (kind lab with kubectl)

1. Namespace

   kubectl create ns undo
   

   What it does: Lab space.

   namespace/undo created
   

2. Deploy

   kubectl apply -f pay.yaml
   

   What it does: v1.

   deployment.apps/pay created
   

3. Releases

   bash ver.sh "pay v2" "pay v2: UPI intent"
   

   What it does: v2 and v3.

   deployment "pay" successfully rolled out
   

4. Bad

   bash ver.sh "ERROR 500" "pay v4: refactor"
   

   What it does: Broken.

   deployment.apps/pay annotated
   

5. Undo

   kubectl rollout undo deploy/pay -n undo
   

   What it does: Back to v3.

   deployment.apps/pay rolled back
   

6. History

   kubectl rollout history deploy/pay -n undo
   

   What it does: Renumbered.

   5         pay v3: new bank list
   

7. Older

   kubectl rollout undo deploy/pay -n undo --to-revision=2
   

   What it does: v2.

   deployment.apps/pay rolled back
   

8. Config trap

   bash use-cm.sh
   

   What it does: Shared config changed.

   configmap/pay-config configured
   

9. Versioned

   bash cm-hash.sh
   

   What it does: Hash name.

   configmap/pay-config-3b1d9e2a created
   

10. Auto

    bash safe-release.sh "ERROR 500"
    

    What it does: Rolled back.

    smoke failed, rolling back
    

11. Clean up

    kubectl delete ns undo
    

    What it does: Ready for 64.4.

    namespace "undo" deleted
    

When it is red 1

error: unable to find specified revision 3 in history

Cause. Revision pruned or renumbered.

Fix, step by step

1. kubectl rollout history to see real numbers.

When it is red 2

skipped rollback (current template already matches revision N)

Cause. You asked for the version that is already running.

Fix, step by step

1. Pick another revision.

When it is red 3

Rollback succeeded but errors continue

Cause. Config, schema or another object changed too.

Fix, step by step

1. Check ConfigMaps, Secrets and migrations.

When it is red 4

ImagePullBackOff after rollback

Cause. Old image deleted from the registry.

Fix, step by step

1. Keep images for every revision you might need.

When it is red 5

Argo CD puts the bad version back

Cause. GitOps sync overwrote the manual undo.

Fix, step by step

1. Revert the commit in git, or pause auto-sync first.

When it is red 6

Rollback is slow

Cause. maxSurge 1 with many replicas.

Fix, step by step

1. Allow a larger surge for rollbacks, or keep the old ReplicaSet scaled during the release.

Practice tasks

kubectl get pods -n undo

kubectl rollout history deploy/pay -n undo

wget from a pod

kubectl rollout status deploy/pay -n undo

kubectl rollout history deploy/pay -n undo

wget from a pod

kubectl exec env

kubectl get cm -n undo

kubectl rollout history deploy/pay -n undo

bash safe-release.sh

Recap

• kubectl rollout undo re-applies an old pod template as a new revision.
• --to-revision jumps further back; history must still hold it.
• Undo restores only the template, not config, schema or other objects.
• Version config with code and keep migrations backward compatible.
• Automate rollback on failed smoke tests; in GitOps, revert in git.

FAQ

Q. How fast is a rollback?

As fast as a rolling update, since it is one. With images cached on nodes and a quick readiness probe, a few seconds per batch of pods.

Q. Can I roll back a StatefulSet or DaemonSet?

Yes, kubectl rollout undo works for both because they keep ControllerRevisions. Watch out for data written in a new format by the new version.

Q. Can I roll back while a rollout is still running?

Yes. Undo during a stalled or running rollout simply points the Deployment at the old template again.

Q. Should rollback be automatic in production?

For clear signals such as a failed smoke test or a large error spike, yes. For subtle problems, alert a person and let them decide quickly.

Q. How does Helm rollback compare?

helm rollback restores a whole release, including ConfigMaps, Services and other objects in the chart, not just the Deployment. That covers the config trap, but not database changes.

Q. What if the bad version wrote bad data?

Rolling back the code does not clean it up. You need a data fix or restore. Feature flags and canaries reduce how much data a bad version can touch.

Q. Does the kustomize configMapGenerator help?

Yes. It adds a content hash to the ConfigMap name automatically and updates references, which gives you versioned config for free.

Q. How many revisions should I keep?

Enough to cover a few days of releases. 5 to 10 is common, and each old ReplicaSet with zero replicas costs almost nothing.

Interview questions

1. How do you roll back a Deployment?

   Model answer: kubectl rollout undo, optionally with --to-revision, then watch rollout status and verify traffic.

2. Why can a rollback fail to fix the problem?

   Model answer: Because only the pod template is restored; changed config, secrets, schema or external services stay changed.

3. How do you make config rollback-safe?

   Model answer: Use hash-named ConfigMaps, through kustomize or Helm, so each revision points to its own config.

4. How do you roll back in GitOps?

   Model answer: Revert the commit in git and let Argo CD or Flux sync; a manual undo will be overwritten.

5. How would you automate rollback?

   Model answer: Run smoke tests or watch metrics after deploy and undo on failure, or use Argo Rollouts analysis.

Samajla ka?