Chapter 64: Kubernetes Updates and Releases: Rolling Updates, Rollback, Blue-Green, Canary and Argo Rollouts

64.1 Rolling updates: replace pods without downtime

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Every team ships new versions all the time. At Swiggy a single frontend may be released many times a day. The default way Kubernetes does this is a rolling update. When you change anything in a Deployment's pod template, such as the image, an environment variable or a resource limit, the Deployment controller creates a new ReplicaSet for the new template. It then scales the new ReplicaSet up and the old one down, a few pods at a time, until all pods run the new version. Users keep getting answers the whole time because some pods are always serving. Two fields control the speed and safety of this dance. maxSurge says how many extra pods may exist above the desired replica count during the update. maxUnavailable says how many of the desired pods may be missing. Both accept a number or a percentage, and both default to 25%. With 4 replicas and the defaults, Kubernetes may run up to 5 pods and must keep at least 3 available. Setting maxUnavailable to 0 and maxSurge to 1 gives the safest rollout: a new pod must be ready before an old one goes away. Setting maxSurge to 0 and maxUnavailable to 1 is useful when the cluster has no spare capacity. "Available" depends on readiness, which you learned in 63.2. A new pod counts only when it is Ready and has stayed Ready for minReadySeconds. If the new pods never become ready, the rollout stops and waits. After progressDeadlineSeconds (600 by default) the Deployment gets a condition Progressing=False with reason ProgressDeadlineExceeded. Kubernetes does not roll back by itself; it simply stops and leaves the old pods serving. You will roll back in 64.3. The other strategy is Recreate. It kills all old pods first and only then starts the new ones. That means downtime, but it is the right choice when two versions must never run together, for example an app that holds an exclusive lock on a disk volume or a database schema change that old code cannot handle. You can also pause a rollout, make several changes and resume it, so the change becomes a single new revision instead of many. In this lesson we run a rolling update under live traffic, compare different surge settings, watch a broken release stall, and try the Recreate strategy to see the gap it causes.

What: every term in plain English

• Rolling update — Default Deployment strategy that replaces pods gradually.
• Recreate — Strategy that deletes all old pods before creating new ones.
• ReplicaSet — Object that keeps a set number of identical pods; each template version gets one.
• pod-template-hash — Label that ties pods to their ReplicaSet.
• maxSurge — Extra pods allowed above replicas during an update; default 25%.
• maxUnavailable — Desired pods allowed to be missing during an update; default 25%.
• minReadySeconds — Time a new pod must stay ready before it counts as available.
• progressDeadlineSeconds — Time after which a stuck rollout is marked failed; default 600.
• ProgressDeadlineExceeded — Reason on the Progressing condition when the deadline passes.
• rollout status — kubectl command that waits for a rollout to finish or fail.
• rollout pause and resume — Stop a rollout, batch several changes, then continue.
• rollout restart — Triggers a new rollout with the same spec by adding an annotation.

How: step by step

1. Create kind cluster ch64. Three nodes.
2. Deploy version 1 of the shop frontend with four replicas. Four ready pods.
3. Start a request loop. Every reply shows the version.
4. Update to version 2. Pods replace a few at a time.
5. Watch ReplicaSets. Old scales down while new scales up.
6. Try different surge settings. See the speed and pod count change.
7. Release a broken version. Rollout stalls; old pods serve.
8. Try Recreate. See the downtime gap.
9. Clean up. Ready for 64.2.

Example 1: Rolling update under live traffic

Goal. Rolling update under live traffic.

kind-ch64.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker

shop-v1.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop
  namespace: roll
spec:
  replicas: 4
  minReadySeconds: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: shop
  template:
    metadata:
      labels:
        app: shop
    spec:
      containers:
        - name: web
          image: hashicorp/http-echo:1.0
          args:
            - -listen=:8080
            - -text=shop v1
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            periodSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: shop
  namespace: roll
spec:
  selector:
    app: shop
  ports:
    - port: 80
      targetPort: 8080

loop.sh

kubectl run loop -n roll --image=busybox:1.36 --restart=Never -- sh -c '
while true; do
  r=$(wget -q -T 1 -O- http://shop 2>/dev/null || echo ERROR)
  echo "$(date +%T) $r"
  sleep 0.2
done'

What each line does

• maxSurge 1, maxUnavailable 0 — At most 5 pods, never fewer than 4 ready: the safest setting.
• minReadySeconds 3 — Each new pod must stay ready 3 seconds before the next step.
• http-echo — Tiny server that returns the -text argument, so we can see the version.
• readinessProbe — Lets the Deployment know when a new pod is really serving.
• loop.sh — Prints a timestamp and the reply five times a second, or ERROR.

Green output

$ kind create cluster --name ch64 --config kind-ch64.yaml
$ kubectl create ns roll
$ kubectl apply -f shop-v1.yaml
$ bash loop.sh
$ kubectl patch deploy shop -n roll --type=json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/args/1","value":"-text=shop v2"}]'
deployment.apps/shop patched
$ kubectl rollout status deploy/shop -n roll
Waiting for deployment "shop" rollout to finish: 1 out of 4 new replicas have been updated...
Waiting for deployment "shop" rollout to finish: 2 out of 4 new replicas have been updated...
Waiting for deployment "shop" rollout to finish: 3 out of 4 new replicas have been updated...
Waiting for deployment "shop" rollout to finish: 1 old replicas are pending termination...
deployment "shop" successfully rolled out
$ kubectl logs -n roll loop --tail=12
10:41:02 shop v1
10:41:02 shop v2
10:41:03 shop v1
10:41:03 shop v2
10:41:04 shop v2

Expected result. During the rollout both versions answer, then only v2. There is no ERROR line, because a new pod was always ready before an old one left.

Example 2: Compare surge settings and watch ReplicaSets

Goal. Compare surge settings and watch ReplicaSets.

fast.yaml

spec:
  strategy:
    rollingUpdate:
      maxSurge: 100%
      maxUnavailable: 0

tight.yaml

spec:
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1

watch-rs.sh

kubectl get rs -n roll -l app=shop -w -o custom-columns=NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas

What each line does

• maxSurge 100% — Creates all new pods at once, then removes old ones: fast but doubles pods for a moment.
• maxSurge 0, maxUnavailable 1 — No extra pods; one old pod goes away before each new one starts. Good for a full cluster.
• watch-rs.sh — Shows desired and ready counts of both ReplicaSets live.
• custom-columns — Keeps the output short and readable.
• Revision — Each patch here creates a new ReplicaSet with a new pod-template-hash.

Green output

$ kubectl patch deploy shop -n roll --patch-file fast.yaml
$ bash watch-rs.sh &
$ kubectl set env deploy/shop -n roll RELEASE=3
NAME              DESIRED   READY
shop-5d8f7c9b6    4         4
shop-7b9c6d4f8    4         (none)
shop-7b9c6d4f8    4         4
shop-5d8f7c9b6    0         4
shop-5d8f7c9b6    0         (none)
$ kubectl patch deploy shop -n roll --patch-file tight.yaml
$ kubectl set env deploy/shop -n roll RELEASE=4
shop-7b9c6d4f8    3         3
shop-66c4d9f7b    1         (none)
shop-66c4d9f7b    1         1
shop-7b9c6d4f8    2         2

Expected result. With 100% surge, four new pods appeared together and the old set dropped to zero in one step. With surge 0 the old set shrank first, one pod at a time, and capacity briefly dropped to three.

Example 3: A broken release stalls, and Recreate shows a gap

Goal. A broken release stalls, and Recreate shows a gap.

bad.sh

kubectl patch deploy shop -n roll --type=json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port","value":9999}]'
kubectl patch deploy shop -n roll -p '{"spec":{"progressDeadlineSeconds":60}}'
kubectl rollout status deploy/shop -n roll --timeout=90s

recreate.yaml

spec:
  strategy:
    type: Recreate
    rollingUpdate: null

What each line does

• Port 9999 — Nothing listens there, so new pods never become ready.
• progressDeadlineSeconds 60 — Marks the rollout failed after one minute instead of ten.
• Old pods — Stay up and serve traffic the whole time.
• Recreate — Kills all old pods, then starts new ones.
• rollingUpdate null — Must be removed when switching to Recreate, or the API rejects the change.

Green output

$ bash bad.sh
Waiting for deployment "shop" rollout to finish: 1 out of 4 new replicas have been updated...
error: deployment "shop" exceeded its progress deadline
$ kubectl get deploy shop -n roll -o jsonpath='{.status.conditions[?(@.type=="Progressing")].reason}'
ProgressDeadlineExceeded
$ kubectl logs -n roll loop --tail=3
10:52:10 shop v2
10:52:10 shop v2
10:52:11 shop v2
$ kubectl rollout undo deploy/shop -n roll
$ kubectl patch deploy shop -n roll --patch-file recreate.yaml
$ kubectl set env deploy/shop -n roll RELEASE=5
$ kubectl logs -n roll loop --tail=20 | grep -c ERROR
14

Expected result. The broken release never got past one new pod, and users only ever saw v2. Recreate gave about three seconds of ERROR lines, which is why it is used only when two versions must not run at the same time.

Every command, step by step (kind lab with kubectl)

1. Cluster

   kind create cluster --name ch64 --config kind-ch64.yaml
   

   What it does: Three nodes.

   Creating cluster "ch64" ...
   

2. Namespace

   kubectl create ns roll
   

   What it does: Lab space.

   namespace/roll created
   

3. Deploy v1

   kubectl apply -f shop-v1.yaml
   

   What it does: Four pods.

   deployment.apps/shop created
   service/shop created
   

4. Loop

   bash loop.sh
   

   What it does: Request loop.

   pod/loop created
   

5. Update

   kubectl patch deploy shop -n roll --type=json -p '...'
   

   What it does: v2 rolls out.

   deployment.apps/shop patched
   

6. Status

   kubectl rollout status deploy/shop -n roll
   

   What it does: Waits.

   deployment "shop" successfully rolled out
   

7. ReplicaSets

   kubectl get rs -n roll
   

   What it does: Old at 0.

   shop-5d8f7c9b6   0   0   0
   shop-7b9c6d4f8   4   4   4
   

8. Fast

   kubectl patch deploy shop -n roll --patch-file fast.yaml
   

   What it does: 100% surge.

   deployment.apps/shop patched
   

9. Tight

   kubectl patch deploy shop -n roll --patch-file tight.yaml
   

   What it does: No surge.

   deployment.apps/shop patched
   

10. Broken

    bash bad.sh
    

    What it does: Stalls.

    error: deployment "shop" exceeded its progress deadline
    

11. Undo

    kubectl rollout undo deploy/shop -n roll
    

    What it does: Back.

    deployment.apps/shop rolled back
    

12. Recreate

    kubectl patch deploy shop -n roll --patch-file recreate.yaml
    

    What it does: Gap.

    deployment.apps/shop patched
    

13. Clean up

    kubectl delete ns roll
    

    What it does: Ready for 64.2.

    namespace "roll" deleted
    

When it is red 1

Rollout stuck at 1 out of 4 new replicas

Cause. New pods are not ready.

Fix, step by step

1. kubectl describe pod for probe errors.
2. Fix and re-apply or roll back.

When it is red 2

exceeded its progress deadline

Cause. progressDeadlineSeconds passed.

Fix, step by step

1. Check events.
2. kubectl rollout undo.

When it is red 3

spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy type is 'Recreate'

Cause. Old rollingUpdate block left in place.

Fix, step by step

1. Set rollingUpdate to null in the patch.

When it is red 4

maxUnavailable and maxSurge cannot both be 0

Cause. Rollout could never make progress.

Fix, step by step

1. Set one of them to at least 1.

When it is red 5

Pending new pods during rollout

Cause. No room for surge pods.

Fix, step by step

1. Lower maxSurge or add nodes.

When it is red 6

Users see errors during deploy

Cause. No readiness probe or no graceful shutdown.

Fix, step by step

1. Add readiness and preStop as in 63.2.

Practice tasks

kubectl get nodes

kubectl get pods -n roll

kubectl logs -n roll loop -f

kubectl rollout status deploy/shop -n roll

kubectl get rs -n roll

kubectl get pods -n roll

kubectl get pods -n roll

kubectl rollout history deploy/shop -n roll

kubectl get deploy shop -n roll -o yaml

kubectl rollout status deploy/shop -n roll

Recap

• A rolling update replaces pods gradually through a new ReplicaSet.
• maxSurge and maxUnavailable control speed and capacity.
• Readiness and minReadySeconds decide when a new pod counts.
• A broken release stalls; it does not roll back by itself.
• Recreate causes downtime and is for versions that cannot coexist.

FAQ

Q. Does changing replicas trigger a rollout?

No. Only pod template changes create a new revision. Scaling just changes the count of the current ReplicaSet.

Q. Does a ConfigMap change trigger a rollout?

No, unless the ConfigMap is referenced by a hash annotation in the template. Many teams add a checksum annotation, or use kubectl rollout restart.

Q. What does rollout restart actually do?

It adds a kubectl.kubernetes.io/restartedAt annotation to the template, which is a template change, so a normal rolling update follows.

Q. Can I roll out StatefulSets and DaemonSets the same way?

Yes, both support RollingUpdate. StatefulSets update one pod at a time in reverse order, and support partition for staged updates. DaemonSets support maxUnavailable and maxSurge.

Q. Why did my rollout create pods on a node that was full?

Surge pods need resources like any pod. If no node has room they stay Pending and the rollout waits. Lower maxSurge or let the Cluster Autoscaler from 63.7 add a node.

Q. How do I see which pods belong to which version?

kubectl get pods -L pod-template-hash, or kubectl get rs -o wide which shows images per ReplicaSet.

Q. What is a good default for production?

maxUnavailable 0 with maxSurge 25% or 1, a real readiness probe, minReadySeconds of a few seconds, and a PDB from 63.4 for drains.

Q. Is a rolling update enough for database changes?

Only if old and new code both work with the schema during the overlap. Use expand and contract migrations: add columns first, deploy, then remove old ones later.

Interview questions

1. How does a Deployment rolling update work?

   Model answer: A template change creates a new ReplicaSet; the controller scales it up and the old one down within maxSurge and maxUnavailable, waiting for readiness.

2. What do maxSurge and maxUnavailable mean?

   Model answer: Extra pods allowed above replicas, and desired pods allowed to be missing, during the update.

3. What happens if new pods never become ready?

   Model answer: The rollout stops, old pods keep serving, and after progressDeadlineSeconds the Deployment reports ProgressDeadlineExceeded.

4. RollingUpdate vs Recreate?

   Model answer: RollingUpdate has no downtime but runs two versions together; Recreate has downtime but never mixes versions.

5. How do you make rollouts zero-downtime?

   Model answer: Readiness probes, maxUnavailable 0, graceful shutdown with preStop, PDBs, and backward compatible changes.

Samajla ka?