Chapter 64: Kubernetes Updates and Releases: Rolling Updates, Rollback, Blue-Green, Canary and Argo Rollouts

64.2 Image tags, digests and change-cause: know exactly what is running

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

A rollback is only as good as your knowledge of what you are rolling back to. Picture a Monday morning at Zomato. Someone says 'production runs orders:latest'. Which build is that? Friday's? Last night's hotfix? Nobody can tell, and if you roll back, Kubernetes may pull the same latest again. That is why release discipline starts with how you name images. A tag such as orders:1.4.2 is a human label that points to an image. Tags are mutable: anyone with push rights can move 1.4.2 to a different image tomorrow. The latest tag is the worst case, because it moves on every build and also changes imagePullPolicy to Always by default. A digest such as orders@sha256:9f2c... is the content hash of the image manifest. It can never change. If two pods use the same digest they run exactly the same bits. Good teams use both: an immutable, meaningful tag such as 1.4.2 or the git commit short SHA for humans, and the digest pinned in the Deployment for machines. CI builds the image, pushes it, reads the digest from the registry and writes image: registry/orders:1.4.2@sha256:... into the manifest. Kubernetes uses the digest and ignores the tag when both are present, but the tag still helps people reading the YAML. Many registries, including ECR, can also mark tags immutable so they cannot be overwritten. Kubernetes also keeps a revision history. Every ReplicaSet created by a Deployment has a deployment.kubernetes.io/revision annotation, and kubectl rollout history lists them. The CHANGE-CAUSE column comes from the kubernetes.io/change-cause annotation. The old --record flag that filled it automatically is deprecated, so today you set the annotation yourself, usually in CI, with a message such as 'orders 1.4.2: fix GST rounding, commit a1b2c3d'. revisionHistoryLimit (default 10) decides how many old ReplicaSets are kept for rollback. In this lesson we watch latest cause confusion, pin a digest, label each release with a change-cause, and read the history to see exactly what each revision contained. Because we must not push to a real registry in a lab, we run a local registry container next to kind and push our own small images there.

What: every term in plain English

• Tag — Human name for an image, such as 1.4.2; can be moved.
• latest — Default tag; moves on every push; avoid in production.
• Digest — sha256 hash of the image manifest; never changes.
• Immutable tag — Registry setting that blocks overwriting an existing tag.
• imagePullPolicy — IfNotPresent, Always or Never; Always is the default for latest.
• imageID — Field in pod status showing the digest actually pulled.
• Revision — Numbered version of a Deployment's pod template.
• change-cause — kubernetes.io/change-cause annotation shown in rollout history.
• revisionHistoryLimit — How many old ReplicaSets are kept; default 10.
• Local registry — registry:2 container used as a private registry for the lab.
• crane or skopeo — Tools that read digests from a registry without pulling.
• Semantic version — MAJOR.MINOR.PATCH scheme for tags.

How: step by step

1. Start a local registry and connect it to kind. Registry on localhost:5001.
2. Build two small images and push them with the same tag. See a tag move.
3. Deploy with the tag and see which image runs. Confusion shown.
4. Get the digest and pin it. Exact image every time.
5. Add change-cause for each release. Readable history.
6. Read rollout history. Revisions with messages.
7. Set revisionHistoryLimit. Old ReplicaSets pruned.
8. Clean up. Ready for 64.3.

Example 1: A moving tag runs two different images

Goal. A moving tag runs two different images.

registry.sh

docker run -d --restart=always -p 127.0.0.1:5001:5000 --name kind-registry registry:2
docker network connect kind kind-registry
printf 'apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: local-registry-hosting\n  namespace: kube-public\ndata:\n  localRegistryHosting.v1: |\n    host: \"localhost:5001\"\n' | kubectl apply -f -
for n in $(kind get nodes --name ch64); do
  docker exec "$n" mkdir -p /etc/containerd/certs.d/localhost:5001
  printf '[host.\"http://kind-registry:5000\"]\n' | docker exec -i "$n" sh -c 'cat > /etc/containerd/certs.d/localhost:5001/hosts.toml'
done

build.sh

mkdir -p img && cd img
printf 'FROM busybox:1.36\nARG MSG\nRUN mkdir /www && echo $MSG > /www/index.html\nCMD [\"httpd\",\"-f\",\"-p\",\"8080\",\"-h\",\"/www\"]\n' > Dockerfile
docker build -q --build-arg MSG="orders build A" -t localhost:5001/orders:1.0 .
docker push -q localhost:5001/orders:1.0

What each line does

• registry:2 — Official registry image, bound to 127.0.0.1 only.
• docker network connect — Lets kind nodes reach the registry by name.
• hosts.toml — Tells containerd on each node where localhost:5001 really is.
• printf — Builds config files and the Dockerfile without heredocs.
• Same tag twice — We push build A, deploy, then push build B under the same 1.0 tag.

Green output

$ bash registry.sh && bash build.sh
$ kubectl create ns tags
$ kubectl create deploy orders -n tags --image=localhost:5001/orders:1.0 --replicas=2 --port=8080
$ docker build -q --build-arg MSG="orders build B" -t localhost:5001/orders:1.0 img && docker push -q localhost:5001/orders:1.0
$ kubectl scale deploy orders -n tags --replicas=3
$ kubectl delete pod -n tags $(kubectl get pod -n tags -o name | head -1 | cut -d/ -f2)
$ for p in $(kubectl get pod -n tags -o name); do kubectl exec -n tags $p -- cat /www/index.html; done
orders build A
orders build B
orders build B

Expected result. The same tag 1.0 now runs two different builds inside one Deployment. New pods on nodes without the old image pulled build B. This is exactly the drift that makes incidents hard to debug.

Example 2: Pin the digest

Goal. Pin the digest.

digest.sh

D=$(docker inspect --format='{{index .RepoDigests 0}}' localhost:5001/orders:1.0)
echo "$D"
kubectl set image deploy/orders -n tags orders="localhost:5001/orders:1.0@${D#*@}"
kubectl rollout status deploy/orders -n tags

What each line does

• RepoDigests — Docker records the digest the registry returned on push.
• ${D#*@} — Shell cut that keeps only sha256:....
• tag plus digest — Humans read 1.0; Kubernetes uses the digest.
• imageID — Pod status confirms the exact digest pulled.
• Result — Every pod now runs build B, and always will, even if 1.0 moves again.

Green output

$ bash digest.sh
localhost:5001/orders@sha256:4c1e8a0b7d2f...e91
deployment "orders" successfully rolled out
$ kubectl get pod -n tags -o jsonpath='{range .items[*]}{.status.containerStatuses[0].imageID}{"\n"}{end}'
localhost:5001/orders@sha256:4c1e8a0b7d2f...e91
localhost:5001/orders@sha256:4c1e8a0b7d2f...e91
localhost:5001/orders@sha256:4c1e8a0b7d2f...e91
$ for p in $(kubectl get pod -n tags -o name); do kubectl exec -n tags $p -- cat /www/index.html; done
orders build B
orders build B
orders build B

Expected result. All three pods report the same imageID and the same content. Pinning the digest removed the drift, and a later push to 1.0 cannot change what runs.

Example 3: Readable history with change-cause

Goal. Readable history with change-cause.

release.sh

V=$1; MSG=$2
docker build -q --build-arg MSG="orders $V" -t localhost:5001/orders:$V img
docker push -q localhost:5001/orders:$V
D=$(docker inspect --format='{{index .RepoDigests 0}}' localhost:5001/orders:$V)
kubectl set image deploy/orders -n tags orders="localhost:5001/orders:$V@${D#*@}"
kubectl annotate deploy/orders -n tags kubernetes.io/change-cause="orders $V: $MSG" --overwrite
kubectl rollout status deploy/orders -n tags

What each line does

• New tag per release — 1.1, 1.2: never reuse a tag.
• set image then annotate — The annotation is copied to the new ReplicaSet as its change-cause.
• --overwrite — Replaces the previous message.
• revisionHistoryLimit — We set it to 3 to see pruning.
• Real CI — Would also add the git commit and pipeline link to the message.

Green output

$ bash release.sh 1.1 "add GST line"
$ bash release.sh 1.2 "fix rounding"
$ bash release.sh 1.3 "new coupon banner"
$ kubectl rollout history deploy/orders -n tags
REVISION  CHANGE-CAUSE
1         (none)
2         (none)
3         orders 1.1: add GST line
4         orders 1.2: fix rounding
5         orders 1.3: new coupon banner
$ kubectl rollout history deploy/orders -n tags --revision=4 | grep Image
    Image:  localhost:5001/orders:1.2@sha256:7aa0...c3
$ kubectl patch deploy orders -n tags -p '{"spec":{"revisionHistoryLimit":3}}'
$ kubectl get rs -n tags --no-headers | wc -l
4

Expected result. Each revision now says what changed and which digest it used. With revisionHistoryLimit 3 only three old ReplicaSets plus the current one remain, so you can roll back at most three steps.

Every command, step by step (kind lab with kubectl)

1. Registry

   bash registry.sh
   

   What it does: Local registry.

   configmap/local-registry-hosting created
   

2. Build A

   bash build.sh
   

   What it does: Push 1.0.

   sha256:1f0e...
   

3. Deploy

   kubectl create deploy orders -n tags --image=localhost:5001/orders:1.0 --replicas=2 --port=8080
   

   What it does: Two pods.

   deployment.apps/orders created
   

4. Move tag

   docker push localhost:5001/orders:1.0
   

   What it does: Build B under same tag.

   1.0: digest: sha256:4c1e... size: 735
   

5. Drift

   kubectl exec ... cat /www/index.html
   

   What it does: Mixed builds.

   orders build A
   orders build B
   

6. Pin

   bash digest.sh
   

   What it does: Digest in spec.

   deployment "orders" successfully rolled out
   

7. imageID

   kubectl get pod -n tags -o jsonpath='...'
   

   What it does: Same digest.

   localhost:5001/orders@sha256:4c1e...
   

8. Releases

   bash release.sh 1.1 "add GST line"
   

   What it does: Revision with message.

   deployment.apps/orders annotated
   

9. History

   kubectl rollout history deploy/orders -n tags
   

   What it does: Messages shown.

   5         orders 1.3: new coupon banner
   

10. Limit

    kubectl patch deploy orders -n tags -p '{"spec":{"revisionHistoryLimit":3}}'
    

    What it does: Pruned.

    deployment.apps/orders patched
    

11. Clean up

    kubectl delete ns tags
    

    What it does: Ready for 64.3. Keep the registry.

    namespace "tags" deleted
    

When it is red 1

ErrImagePull: http: server gave HTTP response to HTTPS client

Cause. containerd does not know the registry is plain HTTP.

Fix, step by step

1. Check hosts.toml on every node.
2. Re-run registry.sh.

When it is red 2

manifest unknown

Cause. Tag or digest not in the registry.

Fix, step by step

1. docker push again.
2. Check the digest string.

When it is red 3

CHANGE-CAUSE shows (none)

Cause. Annotation not set before the revision was created.

Fix, step by step

1. Annotate right after set image.

When it is red 4

rollout history shows only a few revisions

Cause. revisionHistoryLimit pruned older ReplicaSets.

Fix, step by step

1. Raise the limit if you need deeper rollback.

When it is red 5

Pods run different code with the same tag

Cause. Tag was moved after some nodes cached it.

Fix, step by step

1. Pin digests; make tags immutable.

When it is red 6

flag --record has been deprecated

Cause. Old habit.

Fix, step by step

1. Use the change-cause annotation instead.

Practice tasks

curl -s localhost:5001/v2/_catalog

curl -s localhost:5001/v2/orders/tags/list

kubectl get pods -n tags

kubectl exec

kubectl get deploy orders -n tags -o yaml

kubectl get pods -n tags -o json

kubectl rollout history deploy/orders -n tags

kubectl rollout history deploy/orders -n tags --revision=3

kubectl get rs -n tags

kubectl rollout history deploy/orders -n tags

Recap

• Tags are for humans and can move; digests are exact and never change.
• Never deploy latest to production.
• Pin tag plus digest in the manifest from CI.
• Set kubernetes.io/change-cause for readable history.
• revisionHistoryLimit decides how far back you can roll.

FAQ

Q. Does pinning a digest slow down deploys?

No. The image is pulled once per node like any other. It only removes the chance of pulling something unexpected.

Q. How do I get a digest without docker?

crane digest or skopeo inspect read it straight from the registry. Most CI systems also print it after a push.

Q. Should I use git SHA or semantic versions as tags?

Both work. Many teams tag every build with the short SHA and add a semantic version tag for releases. The important part is never reusing a tag.

Q. What about multi-arch images?

A multi-arch image has an index digest that points to one manifest per platform. Pin the index digest and each node pulls the right architecture.

Q. Can Kubernetes block images without a digest?

Yes, with an admission policy such as Kyverno or the built-in ValidatingAdmissionPolicy you can reject pods whose image has no @sha256.

Q. Does GitOps change this?

Tools like Argo CD and Flux apply what is in git, so the digest in git becomes the record of truth and history lives in git as well as in ReplicaSets.

Q. Why is imagePullPolicy Always for latest?

Because Kubernetes assumes latest moves, it checks the registry on every pod start. That makes startup slower and depends on the registry being up.

Q. Is it safe to delete old images from the registry?

Only after no revision you may roll back to uses them. If revisionHistoryLimit is 10, keep at least the last 10 released digests, or a rollback will fail with ImagePullBackOff.

Q. Can I sign images too?

Yes. Tools like cosign sign a digest, and an admission policy can allow only signed digests. Signing works on digests, which is one more reason to pin them.

Interview questions

1. Tag vs digest?

   Model answer: A tag is a movable name; a digest is the immutable content hash of the manifest.

2. Why avoid latest in production?

   Model answer: You cannot tell what runs, rollbacks can pull the same bad image, and pods may run different code.

3. How do you record why a release happened?

   Model answer: Set the kubernetes.io/change-cause annotation, and keep the full history in git and CI.

4. What does revisionHistoryLimit control?

   Model answer: How many old ReplicaSets are kept, and so how far back kubectl rollout undo can go.

5. How do you guarantee every pod runs the same build?

   Model answer: Pin the digest, use immutable tags in the registry, and enforce it with an admission policy.

Samajla ka?