Chapter 64: Kubernetes Updates and Releases: Rolling Updates, Rollback, Blue-Green, Canary and Argo Rollouts

64.4 Blue-green deployments: switch all traffic at once, switch back just as fast

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

A rolling update mixes old and new pods for a while. Sometimes that is not acceptable. Imagine IRCTC launching a new booking flow just before Tatkal opens at 10 am. Half the users on the old flow and half on the new one would be confusing, and if the new flow fails you want to go back instantly, not pod by pod. Blue-green deployment solves this. You run two complete copies of the app side by side. Blue is the version serving users now. Green is the new version, fully deployed with the same number of replicas, but receiving no user traffic. You test green directly, through its own Service or a preview URL, as long as you like. When you are happy you flip one switch, and every new request goes to green. Blue stays running, untouched. If something goes wrong you flip the switch back, and recovery takes one API call instead of a full rollout. In Kubernetes the simplest switch is the Service selector. Both Deployments carry app: booking, and each also has a label such as version: blue or version: green. The live Service selects app: booking plus version: blue. Changing that to version: green updates the EndpointSlices within a second or two, and kube-proxy sends new connections to green pods. A second Service, booking-preview, always points at whichever colour is idle, so testers can reach it. The same idea works at the Ingress level by pointing the Ingress at a different Service. Blue-green has costs. You need double the capacity during the release, which matters for large services. Long-lived connections such as WebSockets stay on blue until they close, so you must drain them. And both colours usually share one database, so schema changes still need the expand and contract pattern from 64.3: green must work with the schema blue uses, and the other way round, until blue is gone. In this lesson we run blue and green side by side, test green through a preview Service, switch traffic with one patch under a live request loop, switch back after a simulated incident, and then retire blue cleanly.

What: every term in plain English

• Blue-green — Two full environments; traffic switches from one to the other at once.
• Blue — The version serving users now.
• Green — The new version, deployed and tested but idle until the switch.
• Selector switch — Changing a Service selector to point at the other colour.
• Preview Service — Second Service that always points at the idle colour for testing.
• Cutover — The moment traffic moves to green.
• Switch back — Reverting the selector to blue after a problem.
• Connection draining — Letting existing connections finish on the old colour.
• Double capacity — Both colours run full replicas during a release.
• Smoke test — Quick checks on green before cutover.
• Expand and contract — Schema changes that both colours can live with.
• Retire — Scale blue to zero once green is stable.

How: step by step

1. Deploy blue with three replicas and the live Service. Users on blue.
2. Deploy green with three replicas. Idle, no user traffic.
3. Create the preview Service. Testers reach green.
4. Smoke test green. Passes.
5. Switch the live Service to green. All traffic moves at once.
6. Simulate an incident and switch back. Instant recovery.
7. Cut over again and retire blue. Clean finish.
8. Clean up. Ready for 64.5.

Example 1: Blue and green side by side

Goal. Blue and green side by side.

booking.sh

mk() {
C=$1
printf 'apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: booking-%s\n  namespace: bg\nspec:\n  replicas: 3\n  selector:\n    matchLabels:\n      app: booking\n      version: %s\n  template:\n    metadata:\n      labels:\n        app: booking\n        version: %s\n    spec:\n      containers:\n        - name: web\n          image: hashicorp/http-echo:1.0\n          args: [\"-listen=:8080\", \"-text=booking %s\"]\n          ports:\n            - containerPort: 8080\n          readinessProbe:\n            httpGet: {path: /, port: 8080}\n            periodSeconds: 2\n' $C $C $C $C > booking-$C.yaml
}
mk blue; mk green

svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: booking
  namespace: bg
spec:
  selector:
    app: booking
    version: blue
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: booking-preview
  namespace: bg
spec:
  selector:
    app: booking
    version: green
  ports:
    - port: 80
      targetPort: 8080

What each line does

• mk function — printf writes one Deployment per colour from the same template.
• version label — The only difference between the colours' selectors.
• booking Service — Live traffic; starts on blue.
• booking-preview — Always the idle colour; starts on green.
• Same replicas — Green must handle full load the moment it goes live.

Green output

$ kubectl create ns bg && bash booking.sh
$ kubectl apply -f booking-blue.yaml -f booking-green.yaml -f svc.yaml
$ kubectl get pods -n bg -L version
NAME                             READY   STATUS    VERSION
booking-blue-6f7d8c9b5-2kq8x     1/1     Running   blue
booking-blue-6f7d8c9b5-7mn4p     1/1     Running   blue
booking-blue-6f7d8c9b5-x9r2t     1/1     Running   blue
booking-green-5c8b7d6f4-4hj6w    1/1     Running   green
booking-green-5c8b7d6f4-k2p9d    1/1     Running   green
booking-green-5c8b7d6f4-v7s3m    1/1     Running   green
$ kubectl run t -n bg --rm -i --image=busybox:1.36 --restart=Never -- sh -c 'wget -q -O- http://booking; wget -q -O- http://booking-preview'
booking blue
booking green

Expected result. Six pods run, but users only reach blue. Testers can hit booking-preview to check green as long as they need, with real cluster networking and real config.

Example 2: Cut over and switch back under live traffic

Goal. Cut over and switch back under live traffic.

switch.sh

C=$1
kubectl patch svc booking -n bg -p "{\"spec\":{\"selector\":{\"app\":\"booking\",\"version\":\"$C\"}}}"
O=blue; [ "$C" = blue ] && O=green
kubectl patch svc booking-preview -n bg -p "{\"spec\":{\"selector\":{\"app\":\"booking\",\"version\":\"$O\"}}}"
kubectl get endpointslices -n bg -l kubernetes.io/service-name=booking -o jsonpath='{.items[0].endpoints[*].addresses[0]}{"\n"}'

loop.sh

kubectl run loop -n bg --image=busybox:1.36 --restart=Never -- sh -c '
while true; do echo "$(date +%T) $(wget -q -T 1 -O- http://booking || echo ERROR)"; sleep 0.2; done'

What each line does

• switch.sh green — Points live at green and preview at blue in two patches.
• EndpointSlice check — Prints the pod IPs now behind the live Service.
• loop.sh — Shows the exact second the replies change colour.
• Switch back — The same script with blue; no pods restart.
• No ERROR lines — Both colours were ready, so nothing was dropped.

Green output

$ bash loop.sh
$ bash switch.sh green
service/booking patched
service/booking-preview patched
10.244.1.9 10.244.2.8 10.244.1.10
$ kubectl logs -n bg loop --tail=6
11:02:14 booking blue
11:02:14 booking blue
11:02:15 booking green
11:02:15 booking green
$ bash switch.sh blue
$ kubectl logs -n bg loop --tail=3
11:03:40 booking blue
11:03:40 booking blue
$ kubectl logs -n bg loop | grep -c ERROR
0

Expected result. Traffic moved from blue to green within about a second and came back just as fast, with zero errors. Compare that with a rolling update rollback, which has to start pods and wait for readiness.

Example 3: Release the next version and retire the old colour

Goal. Release the next version and retire the old colour.

next.sh

IDLE=$(kubectl get svc booking-preview -n bg -o jsonpath='{.spec.selector.version}')
V=$1
kubectl patch deploy booking-$IDLE -n bg --type=json -p "[{\"op\":\"replace\",\"path\":\"/spec/template/spec/containers/0/args/1\",\"value\":\"-text=booking $V\"}]"
kubectl scale deploy booking-$IDLE -n bg --replicas=3
kubectl rollout status deploy/booking-$IDLE -n bg
echo "idle colour $IDLE now runs $V; test it on booking-preview, then: bash switch.sh $IDLE"

retire.sh

LIVE=$(kubectl get svc booking -n bg -o jsonpath='{.spec.selector.version}')
OLD=blue; [ "$LIVE" = blue ] && OLD=green
sleep 30
kubectl scale deploy booking-$OLD -n bg --replicas=0

What each line does

• IDLE — Read from the preview Service, so the script always updates the idle colour.
• Scale to 3 — The idle colour may be at zero after the last retire.
• Test on preview — Same as before: no user sees the new version yet.
• sleep 30 — Lets long connections on the old colour finish before it goes away.
• Scale to 0, not delete — Keeps the Deployment ready for the next release and a fast switch back.

Green output

$ bash next.sh v3
deployment "booking-blue" successfully rolled out
idle colour blue now runs v3; test it on booking-preview, then: bash switch.sh blue
$ bash switch.sh blue
$ kubectl logs -n bg loop --tail=2
11:10:05 booking v3
11:10:05 booking v3
$ bash retire.sh
deployment.apps/booking-green scaled
$ kubectl get deploy -n bg
NAME            READY   UP-TO-DATE   AVAILABLE
booking-blue    3/3     3            3
booking-green   0/0     0            0

Expected result. Colours simply take turns. The idle one gets the next version, is tested on preview, becomes live, and the old one scales to zero after a drain period. Capacity is doubled only during the release window.

Every command, step by step (kind lab with kubectl)

1. Namespace

   kubectl create ns bg
   

   What it does: Lab space.

   namespace/bg created
   

2. Generate

   bash booking.sh
   

   What it does: Two YAML files.

3. Deploy

   kubectl apply -f booking-blue.yaml -f booking-green.yaml -f svc.yaml
   

   What it does: Six pods, two Services.

   service/booking created
   

4. Check

   kubectl get pods -n bg -L version
   

   What it does: Three of each.

   booking-green-5c8b7d6f4-4hj6w   1/1   Running   green
   

5. Preview

   wget http://booking-preview
   

   What it does: Green.

   booking green
   

6. Loop

   bash loop.sh
   

   What it does: Live traffic.

   pod/loop created
   

7. Cut over

   bash switch.sh green
   

   What it does: All green.

   service/booking patched
   

8. Switch back

   bash switch.sh blue
   

   What it does: All blue.

   service/booking patched
   

9. Next

   bash next.sh v3
   

   What it does: Idle colour updated.

   deployment "booking-blue" successfully rolled out
   

10. Retire

    bash retire.sh
    

    What it does: Old colour at 0.

    deployment.apps/booking-green scaled
    

11. Clean up

    kubectl delete ns bg
    

    What it does: Ready for 64.5.

    namespace "bg" deleted
    

When it is red 1

Service has no endpoints after switch

Cause. Selector typo or idle colour scaled to 0.

Fix, step by step

1. kubectl get endpointslices.
2. Scale the colour up before switching.

When it is red 2

Some users still see the old version

Cause. Long-lived or keep-alive connections.

Fix, step by step

1. Wait for drain; lower keep-alive timeouts.

When it is red 3

Green fails right after cutover

Cause. Not tested under real load.

Fix, step by step

1. Load test on preview first (63.8).
2. Switch back.

When it is red 4

Pods Pending for green

Cause. No room for double capacity.

Fix, step by step

1. Add nodes or let the autoscaler from 63.7 handle it.

When it is red 5

Green breaks blue's data

Cause. Shared database, incompatible schema.

Fix, step by step

1. Expand and contract migrations.

When it is red 6

The field selector is immutable

Cause. You patched the Deployment selector instead of the Service.

Fix, step by step

1. Switch the Service selector only.

Practice tasks

kubectl get pods -n bg

kubectl get pods -n bg -L version

kubectl get svc -n bg

kubectl logs -n bg loop

kubectl logs -n bg loop

kubectl get endpointslices -n bg

kubectl get deploy -n bg

kubectl get deploy -n bg

kubectl get ingress -n bg

kubectl logs -n bg loop

Recap

• Blue-green runs two full versions and switches all traffic at once.
• In Kubernetes the switch is a Service selector or Ingress backend change.
• A preview Service lets you test the idle colour safely.
• Switching back is instant because the old colour is still running.
• It costs double capacity and still needs backward compatible data changes.

FAQ

Q. Is blue-green better than rolling updates?

Not always. It gives instant switch and switch back and no mixed versions, but costs double capacity. Rolling updates are cheaper and fine for most services.

Q. Can I do blue-green across clusters?

Yes, by switching DNS or a global load balancer between two clusters. It protects against cluster-level failures but DNS caching makes the switch slower.

Q. Does the switch drop in-flight requests?

No. Existing connections to blue keep working; only new connections go to green. That is why blue must stay up for a drain period.

Q. How do I test green with real users before cutover?

Route internal staff to the preview Service, for example with a header-based Ingress rule. That moves toward canary, covered in 64.5.

Q. What about background workers?

Workers that read from a queue do not use Services, so both colours would consume messages. Scale green workers to zero until cutover, or use separate queues.

Q. Can Helm or Argo CD do blue-green?

Helm needs a custom chart with two releases. Argo Rollouts has a built-in blueGreen strategy with activeService and previewService, shown briefly in 64.6.

Q. How long should I keep blue after cutover?

Long enough to be confident: often one hour to a day for important services. Scaling it to zero after that frees the capacity but keeps a fast path back.

Q. Does blue-green work with StatefulSets?

It is harder, because each colour would need its own copy of the data. For stateful systems, teams usually do blue-green on the stateless tier only and change the database with careful migrations.

Interview questions

1. What is blue-green deployment?

   Model answer: Running the current and new version side by side and switching all traffic from one to the other in one step.

2. How do you implement it in Kubernetes?

   Model answer: Two Deployments with a version label and a Service whose selector you patch, plus a preview Service for testing.

3. Blue-green vs rolling update?

   Model answer: Blue-green: no mixed versions, instant rollback, double capacity. Rolling: cheaper, gradual, mixed versions during rollout.

4. What are the risks?

   Model answer: Shared database schema, long-lived connections, background workers and the cost of double capacity.

5. How do you roll back?

   Model answer: Switch the selector back to the old colour, which is still running.

Samajla ka?