Chapter 63: Kubernetes Health Checks, Self-Healing and Autoscaling: Probes, PDB, HPA, VPA, Cluster Autoscaler and Karpenter

63.1 Liveness probes: restart a stuck container before customers notice

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

A container can be running and still be useless. A Java service at a company like Flipkart can hit a deadlock during a big sale: the process is alive, the port is open, but every request hangs forever. Docker would happily keep that container running, because from the outside the process has not exited. Customers see spinning pages, and someone gets paged at 2 AM to restart it by hand. Kubernetes solves this with a liveness probe. The kubelet on each node runs a small check against the container on a schedule. If the check fails a set number of times in a row, the kubelet kills the container and starts it again under the pod's restartPolicy. Nobody has to wake up. The restart count goes up, an event is written, and the app is back. What can a liveness probe check? There are four mechanisms. httpGet calls a path such as /healthz and treats any status from 200 to 399 as success. tcpSocket only checks that a port accepts a connection. exec runs a command inside the container and checks for exit code 0. grpc calls the standard gRPC health service. On top of the mechanism you tune timing: initialDelaySeconds (wait before the first check), periodSeconds (how often), timeoutSeconds (how long one check may take), failureThreshold (how many failures in a row before a restart) and successThreshold (always 1 for liveness). The big danger is a liveness probe that is too strict or checks the wrong thing. If /healthz calls the database and the database is slow, every pod fails liveness at the same time and Kubernetes restarts the whole fleet, making the outage worse. Liveness should answer one question only: is this process stuck beyond repair? Dependencies belong in readiness, which is the next lesson. In this lesson we build a small app that we can break on purpose, add an HTTP liveness probe, watch the kubelet restart it, then try exec and tcpSocket probes and finally see what a badly tuned probe does to a healthy app. Every step runs on the kind cluster ch63.

What: every term in plain English

• Probe — A periodic check that the kubelet runs against one container.
• Liveness probe — Answers 'is the process stuck?'; failure restarts the container.
• httpGet — Probe that sends an HTTP GET; status 200 to 399 means success.
• tcpSocket — Probe that only opens a TCP connection to a port.
• exec — Probe that runs a command inside the container; exit code 0 means success.
• grpc — Probe that calls the standard gRPC health checking service.
• initialDelaySeconds — Seconds to wait after the container starts before the first probe.
• periodSeconds — How often the probe runs; default 10.
• timeoutSeconds — How long one probe may take before it counts as failed; default 1.
• failureThreshold — Failures in a row before the kubelet acts; default 3.
• RESTARTS — Column in kubectl get pods that counts container restarts.
• CrashLoopBackOff — State when a container keeps failing and the kubelet waits longer between restarts.

How: step by step

1. Create the ch63 kind cluster with three workers. One control plane, three workers.
2. Deploy a breakable app without a probe. It hangs and nothing happens.
3. Add an httpGet liveness probe. The kubelet notices and restarts it.
4. Read events and the restart count. Proof of self-healing.
5. Try an exec probe on a file-based health check. Same idea for apps without HTTP.
6. Try a tcpSocket probe on Redis. Port-level check.
7. Break the timing on purpose. See a too-strict probe kill a healthy app.
8. Clean up the namespace. Ready for readiness in 63.2.

Example 1: A hanging app with and without an HTTP liveness probe

Goal. A hanging app with and without an HTTP liveness probe.

kind-ch63.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: ch63
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker

hang-app.yaml

apiVersion: v1
kind: Pod
metadata:
  name: hang-app
  namespace: probes
  labels:
    app: hang-app
spec:
  containers:
    - name: app
      image: busybox:1.36
      command:
        - sh
        - -c
        - |
          mkdir -p /www && echo ok > /www/healthz
          httpd -f -p 8080 -h /www &
          sleep 30
          echo "simulating a deadlock: health file removed"
          rm /www/healthz
          sleep 3600
      ports:
        - containerPort: 8080
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 1
        failureThreshold: 3

What each line does

• kind config — Three workers so later lessons can spread pods and kill nodes.
• busybox httpd — Serves /healthz from a folder; removing the file makes it return 404, just like a stuck app.
• sleep 30 — The app is healthy for 30 seconds, then breaks.
• httpGet /healthz — Any status from 200 to 399 is healthy; 404 is a failure.
• periodSeconds 5, failureThreshold 3 — After about 15 seconds of failures the kubelet restarts the container.
• Without the probe — Delete the livenessProbe block and the pod stays Running and broken forever.

Green output

$ kind create cluster --config kind-ch63.yaml
$ kubectl create ns probes
$ kubectl apply -f hang-app.yaml
$ kubectl get pod hang-app -n probes -w
NAME       READY   STATUS    RESTARTS      AGE
hang-app   1/1     Running   0             10s
hang-app   1/1     Running   1 (2s ago)    52s
hang-app   1/1     Running   2 (1s ago)    97s
$ kubectl describe pod hang-app -n probes | tail -5
  Warning  Unhealthy  Liveness probe failed: HTTP probe failed with statuscode: 404
  Normal   Killing    Container app failed liveness probe, will be restarted

Expected result. The restart count climbs every time the app 'deadlocks'. The events show exactly why: liveness failed with 404 and the kubelet restarted the container. This is self-healing at the container level.

Example 2: exec and tcpSocket probes for apps that do not speak HTTP

Goal. exec and tcpSocket probes for apps that do not speak HTTP.

exec-probe.yaml

apiVersion: v1
kind: Pod
metadata:
  name: worker
  namespace: probes
spec:
  containers:
    - name: worker
      image: busybox:1.36
      command:
        - sh
        - -c
        - |
          while true; do
            date +%s > /tmp/heartbeat
            sleep 5
          done
      livenessProbe:
        exec:
          command:
            - sh
            - -c
            - test $(( $(date +%s) - $(cat /tmp/heartbeat) )) -lt 30
        initialDelaySeconds: 10
        periodSeconds: 10

redis-tcp.yaml

apiVersion: v1
kind: Pod
metadata:
  name: cache
  namespace: probes
spec:
  containers:
    - name: redis
      image: redis:7.2
      ports:
        - containerPort: 6379
      livenessProbe:
        tcpSocket:
          port: 6379
        initialDelaySeconds: 5
        periodSeconds: 10

What each line does

• Heartbeat file — A queue worker at a company like Swiggy has no HTTP port, so it writes a timestamp every loop.
• exec test — The probe fails when the heartbeat is older than 30 seconds, which means the loop is stuck.
• tcpSocket 6379 — Only checks that Redis accepts connections; cheap and safe.
• When exec is costly — exec starts a process every period; keep the command tiny.
• grpc — For gRPC services use grpc: port: 9090 instead of exec with a grpc health binary.

Green output

$ kubectl apply -f exec-probe.yaml -f redis-tcp.yaml
$ kubectl get pods -n probes
NAME     READY   STATUS    RESTARTS   AGE
cache    1/1     Running   0          40s
worker   1/1     Running   0          40s
$ kubectl exec -n probes worker -- kill -STOP 1
$ kubectl get pod worker -n probes -w
worker   1/1     Running   1 (3s ago)   2m10s

Expected result. Pausing the main process with kill -STOP freezes the loop, the heartbeat goes stale, the exec probe fails and the kubelet restarts the container. Redis stays healthy because its port keeps answering.

Example 3: A badly tuned probe that kills a healthy but slow app

Goal. A badly tuned probe that kills a healthy but slow app.

slow-app.yaml

apiVersion: v1
kind: Pod
metadata:
  name: slow-app
  namespace: probes
spec:
  containers:
    - name: app
      image: busybox:1.36
      command:
        - sh
        - -c
        - |
          sleep 20
          mkdir -p /www && echo ok > /www/healthz
          httpd -f -p 8080 -h /www
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        initialDelaySeconds: 2
        periodSeconds: 3
        failureThreshold: 2

What each line does

• Slow boot — The app needs 20 seconds before it listens, like a Spring Boot service warming caches.
• Probe too early — The first check runs after 2 seconds and fails twice in 6 seconds.
• Restart loop — Each restart starts the 20-second boot again, so the app never comes up.
• CrashLoopBackOff — The kubelet backs off longer and longer between restarts.
• Real fix — Use a startup probe (63.3) or a larger initialDelaySeconds, not a looser liveness check.

Green output

$ kubectl apply -f slow-app.yaml
$ kubectl get pod slow-app -n probes -w
slow-app   1/1   Running            1 (1s ago)    10s
slow-app   1/1   Running            2 (1s ago)    20s
slow-app   0/1   CrashLoopBackOff   4 (12s ago)   75s

Expected result. A perfectly healthy app is stuck forever because the probe never gives it time to start. This is the most common liveness mistake in real clusters.

Every command, step by step (kind lab with kubectl)

1. Create the cluster

   kind create cluster --config kind-ch63.yaml
   

   What it does: Four nodes.

   Creating cluster "ch63" ...
    ✓ Joining worker nodes
   

2. Check nodes

   kubectl get nodes
   

   What it does: One control plane, three workers.

   NAME                 STATUS   ROLES
   ch63-control-plane   Ready    control-plane
   ch63-worker          Ready    worker
   ch63-worker2         Ready    worker
   ch63-worker3         Ready    worker
   

3. Namespace

   kubectl create ns probes
   

   What it does: Lab space.

   namespace/probes created
   

4. Apply the hang app

   kubectl apply -f hang-app.yaml
   

   What it does: Pod with a liveness probe.

   pod/hang-app created
   

5. Watch restarts

   kubectl get pod hang-app -n probes -w
   

   What it does: RESTARTS climbs.

   hang-app   1/1   Running   1 (2s ago)   52s
   

6. Read events

   kubectl get events -n probes --field-selector involvedObject.name=hang-app
   

   What it does: Unhealthy and Killing.

   Warning   Unhealthy   Liveness probe failed: HTTP probe failed with statuscode: 404
   

7. Read the probe

   kubectl get pod hang-app -n probes -o jsonpath='{.spec.containers[0].livenessProbe}'
   

   What it does: Settings with defaults filled in.

   {"failureThreshold":3,"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"periodSeconds":5}
   

8. Last state

   kubectl get pod hang-app -n probes -o jsonpath='{.status.containerStatuses[0].lastState}'
   

   What it does: Why it died.

   {"terminated":{"exitCode":137,"reason":"Error"}}
   

9. exec and tcp

   kubectl apply -f exec-probe.yaml -f redis-tcp.yaml
   

   What it does: Two more probe types.

   pod/worker created
   pod/cache created
   

10. Freeze the worker

    kubectl exec -n probes worker -- kill -STOP 1
    

    What it does: Simulate a stuck loop.

11. Bad timing

    kubectl apply -f slow-app.yaml
    

    What it does: See CrashLoopBackOff.

    slow-app   0/1   CrashLoopBackOff
    

12. Clean up

    kubectl delete ns probes
    

    What it does: Ready for 63.2.

    namespace "probes" deleted
    

When it is red 1

Liveness probe failed: connect: connection refused

Cause. The app is not listening yet or listens on another port.

Fix, step by step

1. Check containerPort and the app's listen address.
2. Add a startup probe for slow boots.

When it is red 2

Liveness probe failed: context deadline exceeded

Cause. The health endpoint takes longer than timeoutSeconds (default 1).

Fix, step by step

1. Make /healthz cheap.
2. Raise timeoutSeconds to 2 or 3.

When it is red 3

Whole Deployment restarts at once during a DB outage

Cause. The liveness endpoint checks the database.

Fix, step by step

1. Move dependency checks to readiness.
2. Keep liveness to the process itself.

When it is red 4

CrashLoopBackOff right after deploy

Cause. The probe starts before the app is ready.

Fix, step by step

1. Use a startupProbe.
2. Or raise initialDelaySeconds.

When it is red 5

exec probe uses lots of CPU

Cause. The command is heavy or the period is short.

Fix, step by step

1. Use a tiny command or switch to httpGet.

Practice tasks

kubectl get nodes

kubectl get pod hang-app -n probes -w

kubectl get events -n probes

kubectl exec -n probes hang-app -- wget -qO- localhost:8080/healthz

kubectl get pod worker -n probes

kubectl describe pod cache -n probes

kubectl describe pod hang-app -n probes

kubectl explain pod.spec.containers.livenessProbe.grpc

kubectl get deploy cart -n probes

kubectl get pod slow-app -n probes

Recap

• A liveness probe lets the kubelet restart a stuck container automatically.
• Use httpGet, tcpSocket, exec or grpc depending on the app.
• Tune periodSeconds, timeoutSeconds and failureThreshold for real behaviour, including GC pauses.
• Never check dependencies in liveness; that turns a small outage into a fleet-wide restart.
• Next lesson: readiness probes keep traffic away from pods that are not ready.

FAQ

Q. Does every container need a liveness probe?

No. Add one when the app can hang without exiting. If the app crashes cleanly on errors, restartPolicy already handles it, and a bad probe adds risk.

Q. Should /healthz and /ready be the same endpoint?

No. Liveness checks only the process; readiness may check dependencies. Many teams use /livez and /readyz, the same names the Kubernetes API server uses.

Q. Does a liveness failure reschedule the pod to another node?

No. The container restarts in place on the same node. Only controllers create new pods elsewhere.

Q. Can probes use HTTPS?

Yes, set scheme: HTTPS. The kubelet does not verify the certificate.

Q. Can I send headers?

Yes, httpHeaders with name and value pairs, for example a Host header.

Q. Do probes count against my app's traffic metrics?

Yes, they show up in access logs; filter the kube-probe user agent.

Q. What exit code does a liveness kill give?

Usually 137, because the container is killed after the grace period.

Interview questions

1. Explain a liveness probe.

   Model answer: A kubelet check that, after failureThreshold failures in a row, restarts the container. It detects deadlocks and stuck processes that have not exited.

2. Difference between liveness and readiness?

   Model answer: Liveness failure restarts the container; readiness failure only removes the pod from Service endpoints without restarting it.

3. Why can a liveness probe cause an outage?

   Model answer: If it checks a shared dependency, all pods fail together and restart together, so capacity drops to zero during the dependency problem.

4. Which probe mechanisms exist?

   Model answer: httpGet, tcpSocket, exec and grpc.

5. A pod is in CrashLoopBackOff right after a deploy, and logs show it was still starting. What do you do?

   Model answer: Add a startup probe or raise initialDelaySeconds so liveness does not kill the app during boot, then confirm with kubectl describe events.

Samajla ka?