Chapter 64: Kubernetes Updates and Releases: Rolling Updates, Rollback, Blue-Green, Canary and Argo Rollouts

64.5 Canary releases: send a small share of users to the new version first

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Blue-green moves everyone at once. A canary moves a few people first. The name comes from miners who carried a canary into the mine: if the bird got sick, the miners left before they got hurt. In software, a small slice of real users gets the new version. If error rates and latency stay healthy, you send more; if not, you pull back and only that slice was affected. Flipkart, Swiggy and most large Indian apps release this way, because their traffic is big enough that 5% of users already gives a clear signal within minutes. The simplest canary in plain Kubernetes uses replica counts. Two Deployments, stable and canary, carry the same app label, and one Service selects that label. kube-proxy spreads connections roughly evenly over all ready pods, so 9 stable pods and 1 canary pod send about 10% of traffic to the canary. It needs no extra tools, but the split is coarse, depends on the number of pods, and you cannot pin a given user to one version. An ingress controller gives finer control. ingress-nginx supports canary annotations on a second Ingress for the same host: nginx.ingress.kubernetes.io/canary: "true" plus canary-weight: "10" sends 10% of requests to the canary Service, independent of pod counts. canary-by-header sends requests with a chosen header, such as X-Canary: always, to the canary every time, which is perfect for testers and internal staff. canary-by-cookie does the same with a cookie, so a user stays on one version across requests. The Gateway API offers the same idea through weighted backendRefs in an HTTPRoute, and service meshes such as Istio and Linkerd do it with their own routing rules. A canary is only useful if you watch it. Before you start, decide which numbers matter (error rate, p95 latency, business metrics like orders per minute), compare canary against stable rather than against an absolute value, and decide the steps, for example 5%, 25%, 50%, 100% with a wait at each. Promotion means updating stable to the new version and removing the canary; abort means setting the weight to zero. In this lesson we build a replica-based canary, install ingress-nginx on kind, split traffic by weight and by header, measure the split with a request loop, and then promote or abort.

What: every term in plain English

• Canary release — Send a small share of real traffic to a new version before everyone.
• Stable — The current version that serves most users.
• Replica-based canary — Traffic split by the ratio of canary to stable pods behind one Service.
• canary annotation — nginx.ingress.kubernetes.io/canary: "true" marks an Ingress as the canary for a host.
• canary-weight — Percentage of requests the canary Ingress gets, 0 to 100.
• canary-by-header — Header name that forces requests to the canary.
• canary-by-cookie — Cookie name that pins a user to the canary.
• Promotion — Moving the new version to stable and removing the canary.
• Abort — Setting canary traffic to zero and removing it.
• Analysis — Comparing canary and stable metrics to decide.
• HTTPRoute weights — Gateway API way to split traffic between backends.
• Blast radius — How many users a bad release can hurt.

How: step by step

1. Deploy stable with 9 replicas and canary with 1. About 10% to canary.
2. Measure the split. Count replies.
3. Install ingress-nginx on kind. Controller ready.
4. Create the main and canary Ingresses. Weight-based split.
5. Change weight to 30 and 50. Split follows.
6. Use a header for testers. Always canary.
7. Promote or abort. Clean finish.
8. Clean up. Ready for 64.6.

Example 1: Replica-based canary behind one Service

Goal. Replica-based canary behind one Service.

canary-replicas.sh

mk() {
N=$1; V=$2; R=$3
printf 'apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: cart-%s\n  namespace: can\nspec:\n  replicas: %s\n  selector:\n    matchLabels: {app: cart, track: %s}\n  template:\n    metadata:\n      labels: {app: cart, track: %s}\n    spec:\n      containers:\n        - name: web\n          image: hashicorp/http-echo:1.0\n          args: [\"-listen=:8080\", \"-text=%s\"]\n          ports: [{containerPort: 8080}]\n' $N $R $N $N $V | kubectl apply -f -
}
kubectl create ns can
mk stable v1 9
mk canary v2 1
kubectl create svc clusterip cart -n can --tcp=80:8080
kubectl set selector svc cart -n can app=cart

count.sh

URL=$1; H=$2
kubectl run count -n can --rm -i --image=busybox:1.36 --restart=Never -- sh -c "
for i in \$(seq 1 200); do wget -q -O- $H $URL; done | sort | uniq -c"

What each line does

• track label — stable or canary; the Service ignores it and selects only app: cart.
• 9 and 1 — Ten pods in total, so roughly one request in ten hits v2.
• set selector — Makes sure the Service selects app=cart only.
• count.sh — Sends 200 requests and counts each reply.
• New connection each time — wget opens a new connection per request, so the spread is per request.

Green output

$ bash canary-replicas.sh
$ bash count.sh http://cart
    181 v1
     19 v2
$ kubectl scale deploy cart-canary -n can --replicas=3
$ bash count.sh http://cart
    150 v1
     50 v2

Expected result. With 9 to 1 the canary got about 10%, and with 9 to 3 about 25%. It works with nothing but a Service, but the only knob is pod count, and to send 1% you would need 99 stable pods.

Example 2: Weighted canary with ingress-nginx

Goal. Weighted canary with ingress-nginx.

ingress-install.sh

# ingress-nginx-kind-v1.11.3.yaml: the kind provider manifest saved from the official ingress-nginx docs, pinned and reviewed
kubectl label node ch64-worker ingress-ready=true
kubectl apply -f ingress-nginx-kind-v1.11.3.yaml
kubectl wait -n ingress-nginx --for=condition=ready pod -l app.kubernetes.io/component=controller --timeout=180s
kubectl scale deploy cart-canary -n can --replicas=0
kubectl create svc clusterip cart-stable -n can --tcp=80:8080
kubectl set selector svc cart-stable -n can app=cart,track=stable
kubectl create svc clusterip cart-canary -n can --tcp=80:8080
kubectl set selector svc cart-canary -n can app=cart,track=canary
kubectl scale deploy cart-canary -n can --replicas=2

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cart
  namespace: can
spec:
  ingressClassName: nginx
  rules:
    - host: cart.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: cart-stable
                port:
                  number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cart-canary
  namespace: can
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
    nginx.ingress.kubernetes.io/canary-by-header: "X-Canary"
spec:
  ingressClassName: nginx
  rules:
    - host: cart.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: cart-canary
                port:
                  number: 80

What each line does

• Pinned manifest — Saved from the official docs and kept in the repo; no live URL in scripts.
• ingress-ready label — The kind provider manifest schedules the controller only on that node.
• Two Services — Each track gets its own Service so the Ingress can weigh them.
• canary-weight 10 — 10% of requests to canary, no matter how many pods each side has.
• canary-by-header — X-Canary: always forces canary, never forces stable.

Green output

$ bash ingress-install.sh && kubectl apply -f ingress.yaml
$ C=http://ingress-nginx-controller.ingress-nginx.svc.cluster.local
$ bash count.sh $C "--header=Host:cart.local"
    179 v1
     21 v2
$ kubectl annotate ingress cart-canary -n can nginx.ingress.kubernetes.io/canary-weight=50 --overwrite
$ bash count.sh $C "--header=Host:cart.local"
    102 v1
     98 v2

Expected result. Now the split follows the annotation, not the pod count: 10% then 50% with two canary pods throughout. Changing the weight is one annotation and takes effect in a second or two.

Example 3: Testers by header, then promote or abort

Goal. Testers by header, then promote or abort.

tester.sh

C=http://ingress-nginx-controller.ingress-nginx.svc.cluster.local
kubectl annotate ingress cart-canary -n can nginx.ingress.kubernetes.io/canary-weight=0 --overwrite
bash count.sh $C "--header=Host:cart.local"
bash count.sh $C "--header=Host:cart.local --header=X-Canary:always"

promote.sh

kubectl patch deploy cart-stable -n can --type=json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/args/1","value":"-text=v2"}]'
kubectl rollout status deploy/cart-stable -n can
kubectl delete ingress cart-canary -n can
kubectl scale deploy cart-canary -n can --replicas=0

abort.sh

kubectl annotate ingress cart-canary -n can nginx.ingress.kubernetes.io/canary-weight=0 --overwrite
kubectl scale deploy cart-canary -n can --replicas=0

What each line does

• Weight 0 plus header — Real users see only stable; testers with the header see only canary.
• promote.sh — Rolls stable to v2, then removes the canary Ingress and pods.
• abort.sh — One annotation stops canary traffic at once; then free the pods.
• Order matters — Promote stable first, then delete the canary, so traffic never hits a missing backend.
• Next step — Argo Rollouts in 64.6 does these steps for you, with checks between them.

Green output

$ bash tester.sh
    200 v1
    200 v2
$ bash promote.sh
deployment "cart-stable" successfully rolled out
ingress.networking.k8s.io "cart-canary" deleted
$ bash count.sh $C "--header=Host:cart.local"
    200 v2

Expected result. With weight 0, ordinary traffic was 100% stable while testers using the header always reached canary. After promotion everyone is on v2 and the canary objects are gone. An abort would have been the same single annotation followed by scaling to zero.

Every command, step by step (kind lab with kubectl)

1. Namespace and Deployments

   bash canary-replicas.sh
   

   What it does: 9 stable, 1 canary.

   deployment.apps/cart-canary created
   

2. Split

   bash count.sh http://cart
   

   What it does: About 10%.

   19 v2
   

3. Scale

   kubectl scale deploy cart-canary -n can --replicas=3
   

   What it does: About 25%.

   50 v2
   

4. Ingress

   bash ingress-install.sh
   

   What it does: Controller ready.

   pod/ingress-nginx-controller-6d9f7c8b5-q2x7m condition met
   

5. Rules

   kubectl apply -f ingress.yaml
   

   What it does: Main and canary.

   ingress.networking.k8s.io/cart-canary created
   

6. Weight 10

   bash count.sh $C "--header=Host:cart.local"
   

   What it does: About 10%.

   21 v2
   

7. Weight 50

   kubectl annotate ingress cart-canary -n can nginx.ingress.kubernetes.io/canary-weight=50 --overwrite
   

   What it does: About half.

   98 v2
   

8. Header

   bash tester.sh
   

   What it does: Testers only.

   200 v2
   

9. Promote

   bash promote.sh
   

   What it does: All v2.

   ingress.networking.k8s.io "cart-canary" deleted
   

10. Clean up

    kubectl delete ns can
    

    What it does: Keep ingress-nginx for 64.6.

    namespace "can" deleted
    

When it is red 1

All traffic goes to stable despite canary-weight

Cause. Canary Ingress host or path does not match the main one.

Fix, step by step

1. Use the same host and path on both.

When it is red 2

404 from the ingress controller

Cause. Host header missing.

Fix, step by step

1. Send Host: cart.local.

When it is red 3

Controller pod Pending

Cause. No node has the ingress-ready label.

Fix, step by step

1. kubectl label node ch64-worker ingress-ready=true.

When it is red 4

Replica split far from expected

Cause. Keep-alive connections stick to one pod.

Fix, step by step

1. Count with new connections, or use ingress weights.

When it is red 5

Canary annotations ignored

Cause. Different controller, such as Traefik or a cloud one.

Fix, step by step

1. Use that controller's own weighting or Gateway API.

When it is red 6

503 right after promote

Cause. Canary Service deleted before stable was ready.

Fix, step by step

1. Promote stable first, then remove canary.

Practice tasks

kubectl get pods -n can

bash count.sh http://cart

bash count.sh http://cart

kubectl get pods -n ingress-nginx

bash count.sh

bash count.sh

bash count.sh

bash count.sh

kubectl get ingress -n can

bash count.sh

Recap

• A canary sends a small share of real traffic to the new version first.
• Replica-based canaries need no tools but the split depends on pod counts.
• ingress-nginx canary-weight splits by percentage; canary-by-header serves testers.
• Decide metrics and steps before starting, and compare canary with stable.
• Promote by updating stable then removing canary; abort with weight 0.

FAQ

Q. What percentage should a canary start at?

Big enough to give a signal, small enough to limit harm. 1 to 5% is common for high-traffic apps; small apps may need 10 to 25% to get enough requests.

Q. How long should each step last?

Long enough to cover the metrics you watch, often 5 to 15 minutes, and longer for slow signals such as payment success or memory leaks.

Q. Is ingress-nginx still a good choice?

No longer for new setups. Kubernetes SIG Network and the Security Response Committee announced its retirement on 12 November 2025, and the project was retired on 24 March 2026. Since then it gets no releases, bug fixes or security patches. Existing installs keep working and the images stay available, so it is fine for this kind lab, but production clusters should move to a Gateway API implementation or another maintained Ingress controller. Gateway API supports weighted routing natively.

Q. How does Gateway API do canaries?

An HTTPRoute lists several backendRefs, each with a weight, for example stable 90 and canary 10. Header matches can send testers to the canary, just like canary-by-header.

Q. Can a canary use the same database as stable?

Usually yes, which is why both versions must be schema compatible. Canaries reduce how many users hit bad code but not how much a bad migration can break.

Q. What about canaries for background workers?

Run a small number of new-version workers alongside the old ones on the same queue. The share of messages they take is roughly their share of consumers.

Q. Is a canary the same as A/B testing?

No. A canary checks whether a release is safe; A/B testing compares product variants for business results. Tools overlap, but A/B tests need sticky users and longer runs.

Q. Can I canary a config change?

Yes. Deploy a canary Deployment that differs only in its ConfigMap, and route a small share to it. Risky config changes deserve this as much as code.

Interview questions

1. What is a canary release?

   Model answer: Sending a small share of real traffic to a new version, checking metrics, and increasing the share step by step or rolling back.

2. How do you canary without extra tools?

   Model answer: Two Deployments behind one Service; the replica ratio sets the split.

3. How do you get a precise split?

   Model answer: Ingress or Gateway weights, or a service mesh, which split by percentage regardless of pod counts.

4. What do you watch during a canary?

   Model answer: Error rate, latency percentiles and key business metrics, compared between canary and stable.

5. Canary vs blue-green?

   Model answer: Canary shifts traffic gradually and limits blast radius; blue-green switches everything at once with an instant switch back.

Samajla ka?