Ravindra BagaleCourses & study guides मराठी

Chapter 76: DevOps Projects Part 6: Kubernetes Canary Helm RBAC NetworkPolicy Quotas EKS Microservices Velero Upgrade

76.9 Project 48: Cluster upgrade with zero downtime

By Last updated:

Friends, upgrades move Kubernetes one minor version at a time without dropping user traffic. Today we write PDBs, practice cordon drain uncordon, note EKS and kubeadm order, and save a checklist.

What you will build in this project

  1. PodDisruptionBudget YAML.
  2. cordon drain uncordon notes.
  3. EKS and kubeadm order tips.
  4. upgrade_checklist.yaml.

1. Start from a real company story

At Groww in Bengaluru, Neha skipped a minor version and broke the metrics API on Monday.

They now upgrade one step with Velero first. Rough past cost: about ₹35 lakh in engineer overtime.

Today we write that safe loop.

Velero backupcontrol planecordondrainupgrade kubeletuncordonsmoke testdone
Picture 1: backup → control plane → for each node: cordon drain upgrade uncordon → smoke test

Ravindra Bagale's Tip

Students use drain --force when PDB blocks them. That can drop the last Pod. Fix capacity first. Keep that habit.

2. Start and prepare

Step 1: Why upgrade carefully

Why: Safe minor bumps. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: echo. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 1: a1

Type this:

echo 'Kubernetes upgrades bump control plane then nodes minor version by minor'

You will see:

Kubernetes upgrades bump control plane then nodes minor version by minor

What it means: What an upgrade is.

Example 2: a2

Type this:

echo 'Skip versions is unsafe; go 1.28 to 1.29 then 1.30'

You will see:

Skip versions is unsafe; go 1.28 to 1.29 then 1.30

What it means: No skipping.

Example 3: a3

Type this:

echo 'Zero downtime means Pods keep serving while nodes rotate'

You will see:

Zero downtime means Pods keep serving while nodes rotate

What it means: Goal.

Step 2: Version check

Why: Know current. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: kubectl version. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 4: b1

Type this:

echo 'kubectl version'

You will see:

kubectl version

What it means: Client and server versions.

Example 5: b2

Type this:

echo 'kubectl get nodes -o wide'

You will see:

kubectl get nodes -o wide

What it means: Node kubelet versions.

Example 6: b3

Type this:

echo 'Check deprecations with kubectl get --raw /metrics 2>/dev/null | head || echo use pluto or kubent offline'

You will see:

Check deprecations ... || echo use pluto or kubent offline

What it means: API deprecation tip.

Step 3: PDB YAML

Why: Protect apps. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: printf pdb. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 7: c1

Type this:

echo 'Precheck: all Deployments have PodDisruptionBudgets for critical apps'

You will see:

Precheck: all Deployments have PodDisruptionBudgets for critical apps

What it means: PDB tip.

Example 8: c2

Type this:

mkdir -p /tmp/k8s48 && printf '%s\n' 'apiVersion: policy/v1' 'kind: PodDisruptionBudget' 'metadata: {name: shop-pdb, namespace: default}' 'spec:' '  minAvailable: 1' '  selector: {matchLabels: {app: shop}}' > /tmp/k8s48/pdb.yaml && cat /tmp/k8s48/pdb.yaml

You will see:

kind: PodDisruptionBudget

What it means: PDB YAML.

Example 9: c3

Type this:

echo 'kubectl apply -f /tmp/k8s48/pdb.yaml'

You will see:

kubectl apply -f /tmp/k8s48/pdb.yaml

What it means: Apply PDB.

3. Build the main pieces

Step 4: Drain practice

Why: Move Pods. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: drain. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 10: d1

Type this:

echo 'Drain tip: kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data'

You will see:

Drain tip: kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

What it means: Drain a node.

Example 11: d2

Type this:

echo 'kubectl get pods -o wide | grep node-1 || echo empty-after-drain'

You will see:

kubectl get pods -o wide | grep node-1 || echo empty-after-drain

What it means: Pods left the node.

Example 12: d3

Type this:

echo 'Never reboot a node without drain in production'

You will see:

Never reboot a node without drain in production

What it means: Habit.

Step 5: Platform tips

Why: EKS kubeadm kind. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: platform tips. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 13: e1

Type this:

echo 'kind upgrade tip: recreate node image then join; or recreate cluster for labs'

You will see:

kind upgrade tip: recreate node image then join; or recreate cluster for labs

What it means: kind lab tip.

Example 14: e2

Type this:

echo 'EKS tip: update node group AMI then aws eks update-cluster-version'

You will see:

EKS tip: update node group AMI then aws eks update-cluster-version

What it means: EKS order.

Example 15: e3

Type this:

echo 'kubeadm tip: kubeadm upgrade plan then apply then upgrade kubelet on nodes'

You will see:

kubeadm tip: kubeadm upgrade plan then apply then upgrade kubelet on nodes

What it means: kubeadm order.

Step 6: Cordon loop

Why: Node loop. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: cordon. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 16: f1

Type this:

echo 'kubectl cordon node-1'

You will see:

kubectl cordon node-1

What it means: Mark unschedulable before drain.

Example 17: f2

Type this:

echo 'kubectl uncordon node-1'

You will see:

kubectl uncordon node-1

What it means: Allow Pods after upgrade.

Example 18: f3

Type this:

echo 'Cordon → drain → upgrade kubelet → uncordon is the node loop'

You will see:

Cordon → drain → upgrade kubelet → uncordon is the node loop

What it means: Node loop.

4. Automate and verify

Step 7: Add-ons backup

Why: System plane. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: addons. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 19: g1

Type this:

echo 'Add-ons: upgrade CNI CoreDNS kube-proxy before or with the control plane per vendor notes'

You will see:

Add-ons: upgrade CNI CoreDNS kube-proxy before or with the control plane per vendor notes

What it means: Add-on tip.

Example 20: g2

Type this:

echo 'kubectl get pods -n kube-system'

You will see:

kubectl get pods -n kube-system

What it means: System Pods healthy after bump.

Example 21: g3

Type this:

echo 'Backup etcd or Velero before the control plane change'

You will see:

Backup etcd or Velero before the control plane change

What it means: Backup first.

Step 8: Readyz watch

Why: Health. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: readyz. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 22: h1

Type this:

echo 'kubectl get cs 2>/dev/null || kubectl get --raw='/readyz?verbose'

You will see:

kubectl get --raw='/readyz?verbose'

What it means: API readiness.

Example 23: h2

Type this:

echo 'Watch Deployments during drain: kubectl get deploy -A -w'

You will see:

Watch Deployments during drain: kubectl get deploy -A -w

What it means: Watch tip.

Example 24: h3

Type this:

echo 'If PDB blocks drain, fix replicas — do not --force casually'

You will see:

If PDB blocks drain, fix replicas — do not --force casually

What it means: Force warning.

Step 9: Checklist smoke

Why: Runbook. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: checklist. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 25: i1

Type this:

printf '%s\n' 'upgrade_checklist:' '- backup Velero' '- check deprecations' '- upgrade add-ons' '- upgrade control plane' '- cordon drain upgrade uncordon each node' '- smoke test Ingress and payments' > /tmp/k8s48/checklist.yaml && cat /tmp/k8s48/checklist.yaml

You will see:

upgrade_checklist:

What it means: Written checklist.

Example 26: i2

Type this:

echo 'Smoke test: curl the shop Ingress and a payments health URL'

You will see:

Smoke test: curl the shop Ingress and a payments health URL

What it means: Smoke tip.

Example 27: i3

Type this:

echo 'Keep a rollback note: previous AMI id and control plane version'

You will see:

Keep a rollback note: previous AMI id and control plane version

What it means: Rollback tip.

5. Harden and accept

Step 10: Surge announce

Why: Capacity comms. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: surge. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 28: j1

Type this:

echo 'Surge tip: add a new node first so capacity exists before drain'

You will see:

Surge tip: add a new node first so capacity exists before drain

What it means: Surge capacity.

Example 29: j2

Type this:

echo 'Maintenance window still helps even with zero-downtime technique'

You will see:

Maintenance window still helps even with zero-downtime technique

What it means: Comms tip.

Example 30: j3

Type this:

echo 'Announce in Slack before the first cordon'

You will see:

Announce in Slack before the first cordon

What it means: Announce tip.

Step 11: Cleanup

Why: Cleanup. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: cleanup. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 31: k1

Type this:

echo 'Optional cleanup: kubectl delete pdb shop-pdb'

You will see:

Optional cleanup: kubectl delete pdb shop-pdb

What it means: Cleanup.

Example 32: k2

Type this:

echo 'Save the checklist.yaml in your runbook folder'

You will see:

Save the checklist.yaml in your runbook folder

What it means: Artifact.

Example 33: k3

Type this:

echo 'Kubernetes project block 31-48 complete — Terraform projects are next'

You will see:

Kubernetes project block 31-48 complete — Terraform projects are next

What it means: Bridge.

Step 12: Acceptance

Why: Neha signs off. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: All green. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 34: z1

Type this:

echo 'PDB applied; cordon drain uncordon loop written'

You will see:

PDB applied; cordon drain uncordon loop written

What it means: Checklist node loop.

Example 35: z2

Type this:

echo 'Backup before control plane noted'

You will see:

Backup before control plane noted

What it means: Checklist backup.

Example 36: z3

Type this:

echo 'Neha: PASS'

You will see:

Neha: PASS

What it means: Project 48 accepted.

Check yourself

Q1. Who fixed Groww?

Neha

Q2. Skip versions?

no

Q3. Mark unschedulable?

cordon

Q4. Empty node command?

drain

Q5. Allow Pods again?

uncordon

Q6. Before control plane?

backup

Friends, you wrote a zero-downtime upgrade runbook. Did you understand? Let us try it once more. Put the checklist next to your on-call doc.

Lab: try at home

  1. Check versions and deprecations.
  2. Apply PDB.
  3. Practice cordon drain uncordon.
  4. Note platform upgrade order.
  5. Write checklist and smoke tests.

Recap

  • No version skipping.
  • Backup first.
  • PDB for critical apps.
  • Surge capacity.
  • Announce maintenance.

Lab

  1. Finish all nine projects; keep notes of abort, can-i, and restore proofs.
  2. Keep a notebook of helm history, NetworkPolicy allow/deny tests, and upgrade checklist.yaml.
  3. Never apply untested YAML to a shared staging cluster.
  4. After each project, write one paragraph: what broke and how you fixed it.
  5. Next batch starts Terraform projects 49-60.

Practice task

Pick the canary abort drill and the Velero restore drill. Prove both before you call the chapter done.

Ravindra Bagale's Tip

Practice abort on canaries. Never bind cluster-admin to a team ServiceAccount. Restore Velero every month. Backup before every control-plane bump.

In short

  • Abort before promote pride.
  • Helm values without Secrets in git.
  • Least privilege RBAC.
  • Deny-all then allow NetworkPolicy.
  • Requests before quotas complain.
  • IRSA over node keys on EKS.
  • One Deployment per microservice.
  • Untested backup is a hope.
  • Cordon drain upgrade uncordon.