Chapter 63: Kubernetes Health Checks, Self-Healing and Autoscaling: Probes, PDB, HPA, VPA, Cluster Autoscaler and Karpenter

63.4 Self-healing and fault tolerance: pod failure, node failure, PDB, anti-affinity and topology spread

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Probes heal one container. Real outages are bigger: a whole pod is deleted, a node runs out of memory, an EC2 instance is retired by AWS, or an entire availability zone has trouble. Zomato's order API cannot go down because one machine died during dinner rush. Fault tolerance means the app keeps serving through these failures, and self-healing means Kubernetes puts things back the way you declared them without a human. Kubernetes gives you layers. First, controllers: a Deployment's ReplicaSet always works to keep the replica count, so a deleted or crashed pod is replaced in seconds. Second, node failure handling: when a node stops reporting, its Ready condition becomes Unknown, the node gets a NoExecute taint, and after the toleration time (300 seconds by default) its pods are evicted and recreated on healthy nodes. Third, placement: if all three replicas sit on the same node, one node failure takes all of them. Pod anti-affinity and topologySpreadConstraints spread replicas across nodes and zones so one failure only removes part of the capacity. Fourth, voluntary disruptions. Not every pod removal is an accident. kubectl drain during node upgrades, Cluster Autoscaler scale-down and Karpenter consolidation all evict pods on purpose. A PodDisruptionBudget (PDB) tells Kubernetes 'never let fewer than N of these pods be available during voluntary evictions'. The eviction API respects it; a drain simply waits until it is safe. In this lesson we kill pods and see them return, stop a kind node container to simulate a dead EC2 instance, tune how fast pods move off a dead node, spread replicas with topology spread and anti-affinity, and protect them during a drain with a PDB. Together these are the answers interviewers want when they ask 'what happens when a node dies?'.

What: every term in plain English

• Self-healing — Controllers restoring declared state after failures without humans.
• Fault tolerance — Continuing to serve while parts of the system fail.
• Node Ready Unknown — Node condition when the kubelet stops reporting.
• node.kubernetes.io/unreachable — NoExecute taint added to a node that stopped reporting.
• tolerationSeconds — How long a pod stays on a tainted node before eviction; default 300.
• Pod anti-affinity — Rule that keeps pods with matching labels off the same node or zone.
• topologySpreadConstraints — Rule that keeps pod counts balanced across nodes or zones within maxSkew.
• maxSkew — Maximum allowed difference in pod count between topology domains.
• topologyKey — Node label that defines a domain, like kubernetes.io/hostname or topology.kubernetes.io/zone.
• Voluntary disruption — Planned eviction: drain, autoscaler scale-down, upgrades.
• PodDisruptionBudget — Limits how many pods of an app voluntary evictions may remove at once.
• kubectl drain — Cordons a node and evicts its pods through the eviction API.

How: step by step

1. Deploy orders-api with 3 replicas. Baseline.
2. Delete pods and watch replacements. Controller healing.
3. Spread pods across nodes. topologySpreadConstraints.
4. Stop a kind node. Simulate a dead instance.
5. Tune tolerationSeconds. Faster failover.
6. Add a PDB and drain a node. Safe maintenance.
7. Clean up. Ready for HPA in 63.5.

Example 1: Pod failure and controller healing

Goal. Pod failure and controller healing.

orders-api.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  namespace: heal
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      containers:
        - name: api
          image: nginx:1.27
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
---
apiVersion: v1
kind: Service
metadata:
  name: orders-api
  namespace: heal
spec:
  selector:
    app: orders-api
  ports:
    - port: 80

What each line does

• replicas 3 — Desired state the ReplicaSet defends.
• readinessProbe — New pods get traffic only when ready.
• Delete a pod — The ReplicaSet sees 2 of 3 and creates one.
• Delete all pods — All three come back; the Service has a gap of a few seconds.
• Kill the process — kill 1 inside the container: restartPolicy restarts it in place.

Green output

$ kubectl create ns heal
$ kubectl apply -f orders-api.yaml
$ kubectl delete pod -n heal -l app=orders-api --wait=false
$ kubectl get pods -n heal -l app=orders-api
NAME                          READY   STATUS              AGE
orders-api-6f7d9c8b5-5mx2q    0/1     ContainerCreating   2s
orders-api-6f7d9c8b5-h8k4t    0/1     ContainerCreating   2s
orders-api-6f7d9c8b5-w9n3c    0/1     ContainerCreating   2s
$ kubectl get rs -n heal
NAME                    DESIRED   CURRENT   READY
orders-api-6f7d9c8b5    3         3         3

Expected result. Every deleted pod was replaced with a new name within seconds. Deleting all of them at once caused a short gap, which is why we spread pods and use PDBs next.

Example 2: Spread replicas and survive a dead node

Goal. Spread replicas and survive a dead node.

spread-patch.yaml

spec:
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: orders-api
      tolerations:
        - key: node.kubernetes.io/unreachable
          operator: Exists
          effect: NoExecute
          tolerationSeconds: 30
        - key: node.kubernetes.io/not-ready
          operator: Exists
          effect: NoExecute
          tolerationSeconds: 30

kill-node.sh

NODE=$(kubectl get pod -n heal -l app=orders-api -o jsonpath='{.items[0].spec.nodeName}')
echo "stopping $NODE"
docker stop $NODE
kubectl get nodes -w

What each line does

• maxSkew 1 — Node pod counts may differ by at most one, so 3 replicas land on 3 workers.
• DoNotSchedule — Hard rule; ScheduleAnyway makes it a preference.
• Zones — On EKS use topology.kubernetes.io/zone to spread across AZs.
• tolerationSeconds 30 — Leave a dead node after 30 seconds instead of the default 300.
• docker stop — A kind node is a container; stopping it is like an EC2 instance dying.

Green output

$ kubectl patch deploy orders-api -n heal --patch-file spread-patch.yaml
$ kubectl get pods -n heal -o wide
orders-api-7c8b9d6f4-2xk9m   1/1   Running   ch63-worker
orders-api-7c8b9d6f4-8hq3n   1/1   Running   ch63-worker2
orders-api-7c8b9d6f4-t5v7p   1/1   Running   ch63-worker3
$ bash kill-node.sh
stopping ch63-worker
ch63-worker   NotReady   45s
$ kubectl get pods -n heal -o wide
orders-api-7c8b9d6f4-2xk9m   1/1   Terminating   ch63-worker
orders-api-7c8b9d6f4-q4w8r   0/1   Pending       (none)

Expected result. One replica per node means the dead node took only a third of capacity. After the node went NotReady and 30 seconds passed, the pod was evicted. The replacement is Pending because maxSkew 1 with DoNotSchedule cannot place a second pod on a surviving node while the dead node still counts. Switch to ScheduleAnyway, or restart the node with docker start ch63-worker, and it schedules.

Example 3: Safe maintenance with a PodDisruptionBudget

Goal. Safe maintenance with a PodDisruptionBudget.

orders-pdb.yaml

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders-api
  namespace: heal
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: orders-api

What each line does

• minAvailable 2 — At least two pods must stay available during voluntary evictions.
• maxUnavailable — Alternative: maxUnavailable: 1, which scales with replica count.
• drain — Uses the eviction API, which checks the PDB.
• Blocked eviction — drain retries until another pod is ready elsewhere.
• Not for crashes — PDBs only govern voluntary disruptions; a dead node ignores them.

Green output

$ docker start ch63-worker
$ kubectl patch deploy orders-api -n heal --type=json -p '[{"op":"replace","path":"/spec/template/spec/topologySpreadConstraints/0/whenUnsatisfiable","value":"ScheduleAnyway"}]'
$ kubectl apply -f orders-pdb.yaml
$ kubectl get pdb -n heal
NAME         MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS
orders-api   2               N/A               1
$ kubectl drain ch63-worker2 --ignore-daemonsets --delete-emptydir-data
evicting pod heal/orders-api-7c8b9d6f4-8hq3n
pod/orders-api-7c8b9d6f4-8hq3n evicted
node/ch63-worker2 drained
$ kubectl uncordon ch63-worker2

Expected result. The PDB allowed exactly one disruption. Drain evicted one pod, waited for its replacement to be ready, and only then would it evict another. Users kept at least two serving pods the whole time.

Every command, step by step (kind lab with kubectl)

1. Namespace

   kubectl create ns heal
   

   What it does: Lab space.

   namespace/heal created
   

2. Deploy

   kubectl apply -f orders-api.yaml
   

   What it does: Three replicas.

   deployment.apps/orders-api created
   

3. Delete one pod

   kubectl delete pod -n heal $(kubectl get pod -n heal -l app=orders-api -o name | head -1 | cut -d/ -f2)
   

   What it does: Replaced.

   pod "orders-api-6f7d9c8b5-5mx2q" deleted
   

4. Owner

   kubectl get pod -n heal -l app=orders-api -o jsonpath='{.items[0].metadata.ownerReferences[0].kind}'
   

   What it does: ReplicaSet owns pods.

   ReplicaSet
   

5. Spread

   kubectl patch deploy orders-api -n heal --patch-file spread-patch.yaml
   

   What it does: One per node.

   deployment.apps/orders-api patched
   

6. Where

   kubectl get pods -n heal -o wide
   

   What it does: Three nodes.

   ch63-worker  ch63-worker2  ch63-worker3
   

7. Kill a node

   bash kill-node.sh
   

   What it does: NotReady.

   ch63-worker   NotReady
   

8. Taints

   kubectl describe node ch63-worker | grep Taints -A2
   

   What it does: unreachable NoExecute.

   Taints: node.kubernetes.io/unreachable:NoExecute
   

9. Restart node

   docker start ch63-worker
   

   What it does: Node returns.

   ch63-worker
   

10. PDB

    kubectl apply -f orders-pdb.yaml
    

    What it does: minAvailable 2.

    poddisruptionbudget.policy/orders-api created
    

11. Drain

    kubectl drain ch63-worker2 --ignore-daemonsets --delete-emptydir-data
    

    What it does: Respects the PDB.

    node/ch63-worker2 drained
    

12. Uncordon and clean

    kubectl uncordon ch63-worker2
    kubectl delete ns heal
    

    What it does: Ready for 63.5.

    node/ch63-worker2 uncordoned
    

When it is red 1

Cannot evict pod as it would violate the pod's disruption budget

Cause. Not enough healthy replicas to satisfy the PDB.

Fix, step by step

1. Wait for replacements, scale up first, or check for unready pods.

When it is red 2

Drain hangs forever

Cause. PDB with minAvailable equal to replicas, so zero disruptions are allowed.

Fix, step by step

1. Use maxUnavailable: 1 or more replicas.

When it is red 3

Pods stay on a dead node for 5 minutes

Cause. Default tolerationSeconds is 300.

Fix, step by step

1. Lower it per workload, for example 30.

When it is red 4

Replacement pod Pending after node failure

Cause. Hard spread or anti-affinity cannot be satisfied.

Fix, step by step

1. Use ScheduleAnyway or preferred anti-affinity, or add nodes.

When it is red 5

All replicas on one node

Cause. No spread rules, so the scheduler packed them.

Fix, step by step

1. Add topologySpreadConstraints on hostname and zone.

Practice tasks

kubectl get pods -n heal -w

kubectl delete pod -n heal -l app=orders-api

kubectl get pods -n heal -o wide

kubectl get nodes

kubectl get pod -n heal -o yaml

kubectl explain pod.spec.affinity.podAntiAffinity

kubectl get pdb -n heal

kubectl drain ch63-worker2 --ignore-daemonsets

kubectl label node ch63-worker topology.kubernetes.io/zone=ap-south-1a

kubectl drain ch63-worker3 --ignore-daemonsets

Recap

• Controllers replace failed pods; probes heal containers.
• Dead nodes are tainted and their pods evicted after tolerationSeconds.
• Spread replicas across nodes and zones so one failure removes only part of capacity.
• PDBs keep enough pods alive during drains, upgrades and autoscaler scale-down.
• Next lesson: Horizontal Pod Autoscaler.

FAQ

Q. Does Kubernetes move a running pod to another node?

No, it never moves pods. It deletes them and creates new ones elsewhere, which is why apps must tolerate restarts.

Q. What about StatefulSet pods on a dead node?

They are not replaced automatically until the node is confirmed gone, to avoid two pods with the same identity. Delete the node object or force-delete the pod after confirming the instance is dead.

Q. How many replicas for fault tolerance?

At least 3 for important services, spread across zones.

Q. Anti-affinity or topology spread?

Topology spread is more flexible and scales better; anti-affinity is fine for one-per-node rules.

Q. Does a PDB block deleting a Deployment?

No, PDBs only apply to evictions.

Q. Do I need a PDB with one replica?

It would block drains forever with minAvailable 1; run at least two replicas instead.

Q. Does EKS handle node health too?

Managed node groups and Karpenter replace unhealthy instances, while Kubernetes reschedules pods.

Interview questions

1. What happens when a worker node dies?

   Model answer: Its kubelet stops reporting, the node becomes NotReady, the unreachable NoExecute taint is applied, and after tolerationSeconds the pods are evicted and their controllers create replacements on healthy nodes.

2. What is a PodDisruptionBudget?

   Model answer: A policy that limits how many pods of an app can be voluntarily evicted at once, using minAvailable or maxUnavailable.

3. How do you spread pods across AZs?

   Model answer: topologySpreadConstraints with topologyKey topology.kubernetes.io/zone and a small maxSkew, often together with hostname spreading.

4. Voluntary vs involuntary disruption?

   Model answer: Voluntary are planned evictions like drain or autoscaling; involuntary are crashes, kernel panics and hardware failure. PDBs cover only voluntary ones.

5. Why can node maintenance cause an outage even with 3 replicas?

   Model answer: If all replicas are on the same node or there is no PDB, a drain can evict all of them together.

Samajla ka?