Chapter 63: Kubernetes Health Checks, Self-Healing and Autoscaling: Probes, PDB, HPA, VPA, Cluster Autoscaler and Karpenter

63.2 Readiness probes: send traffic only to pods that can serve it

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Liveness answers 'is the process stuck?'. Readiness answers a different question: 'should this pod get traffic right now?'. Think of a payments API at Razorpay. During a deploy a new pod starts, but it still has to load its config, open its database pool and warm a cache. If the Service sends real payments to it in those first seconds, customers see errors. Later, the same pod may lose its connection to Redis for a minute. It is not stuck, so restarting it would not help, but it should stop taking requests until Redis is back. A readiness probe does exactly that. The kubelet runs it on a schedule, just like liveness. When it fails, the pod's Ready condition turns False and the endpoints controller removes the pod's IP from every Service that selects it (you saw EndpointSlices in 62.2). No restart happens. When the probe passes again, the pod goes back into rotation. kubectl get pods shows this in the READY column: 0/1 means running but not ready. Readiness is also what makes rolling updates safe. A Deployment counts a new pod as available only after it is Ready (plus minReadySeconds if set). If the new version never becomes ready, the rollout stops and the old pods keep serving. Without a readiness probe Kubernetes assumes a pod is ready the moment its container starts, which is how many teams ship broken releases to all users at once. Readiness probes use the same mechanisms and timing fields as liveness: httpGet, tcpSocket, exec, grpc, with periodSeconds, timeoutSeconds, failureThreshold and successThreshold (which can be above 1 for readiness). Unlike liveness, it is fine for readiness to check critical dependencies, but be careful: if every pod depends on the same database, they will all go unready together and the Service will have zero endpoints. In this lesson we watch a pod leave and rejoin a Service when its readiness changes, see a rolling update stall because the new version is never ready, and build a drain endpoint that lets a pod finish requests before shutdown.

What: every term in plain English

• Readiness probe — Answers 'can this pod take traffic?'; failure removes it from Service endpoints.
• Ready condition — Pod status condition that is True only when all containers pass readiness.
• READY column — Shows ready containers out of total, for example 0/1.
• EndpointSlice — Object that lists the ready pod IPs behind a Service.
• notReadyAddresses — Endpoints that exist but are not receiving traffic.
• successThreshold — Passes in a row needed to become ready again; may be above 1 for readiness.
• minReadySeconds — Extra seconds a new pod must stay ready before a Deployment counts it as available.
• Rollout stall — A rolling update that cannot continue because new pods never become ready.
• progressDeadlineSeconds — Time after which a stuck rollout is marked as failed; default 600.
• Drain endpoint — An app endpoint that flips readiness to false before shutdown.
• publishNotReadyAddresses — Service flag that keeps unready pods in DNS, used by StatefulSets.

How: step by step

1. Deploy three pods behind a Service with a readiness probe. All three in endpoints.
2. Make one pod unready. It leaves the endpoints; no restart.
3. Make it ready again. It rejoins.
4. Roll out a broken version. The rollout stalls; old pods keep serving.
5. Undo the rollout. Back to healthy.
6. Add a drain step with preStop. Zero errors during scale-down.
7. Clean up. Ready for startup probes in 63.3.

Example 1: Watch a pod leave and rejoin the Service

Goal. Watch a pod leave and rejoin the Service.

pay-api.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pay-api
  namespace: ready
spec:
  replicas: 3
  selector:
    matchLabels:
      app: pay-api
  template:
    metadata:
      labels:
        app: pay-api
    spec:
      containers:
        - name: api
          image: busybox:1.36
          command:
            - sh
            - -c
            - |
              mkdir -p /www && hostname > /www/index.html && echo ok > /www/ready
              httpd -f -p 8080 -h /www
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 3
            failureThreshold: 1
            successThreshold: 2
---
apiVersion: v1
kind: Service
metadata:
  name: pay-api
  namespace: ready
spec:
  selector:
    app: pay-api
  ports:
    - port: 80
      targetPort: 8080

What each line does

• /ready file — Deleting it makes readiness fail, like a lost Redis connection.
• failureThreshold 1 — Leave the Service after one failed check, so traffic moves away fast.
• successThreshold 2 — Rejoin only after two good checks in a row, to avoid flapping.
• Service — Selects all pay-api pods, but only ready ones get traffic.
• No liveness here — We want to see readiness alone: the pod is never restarted.

Green output

$ kubectl create ns ready
$ kubectl apply -f pay-api.yaml
$ kubectl get endpointslices -n ready -l kubernetes.io/service-name=pay-api
NAME            ADDRESSTYPE   PORTS   ENDPOINTS
pay-api-7xk2p   IPv4          8080    10.244.1.5,10.244.2.4,10.244.3.6
$ POD=$(kubectl get pod -n ready -l app=pay-api -o name | head -1)
$ kubectl exec -n ready $POD -- rm /www/ready
$ kubectl get pods -n ready
NAME                       READY   STATUS    RESTARTS
pay-api-6c9d8b7f5d-4hq2x   0/1     Running   0
pay-api-6c9d8b7f5d-8wz7n   1/1     Running   0
pay-api-6c9d8b7f5d-r2m9k   1/1     Running   0
$ kubectl get endpointslices -n ready -l kubernetes.io/service-name=pay-api
pay-api-7xk2p   IPv4   8080   10.244.2.4,10.244.3.6
$ kubectl exec -n ready $POD -- sh -c 'echo ok > /www/ready'

Expected result. The broken pod shows 0/1 and its IP disappears from the EndpointSlice, but RESTARTS stays 0. After the file returns and two checks pass, the IP is back.

Example 2: A broken release stalls instead of reaching users

Goal. A broken release stalls instead of reaching users.

bad-release.sh

kubectl patch deploy pay-api -n ready --type=json -p '[
 {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/does-not-exist"}]'
kubectl rollout status deploy/pay-api -n ready --timeout=60s

What each line does

• The patch — Simulates v2 of the app whose ready endpoint is broken.
• maxUnavailable 25% — Default strategy: at most one of three old pods may go down.
• New pod never ready — So the Deployment never counts it as available and does not remove more old pods.
• Customers — Still served by the two or three old pods.
• rollout undo — Returns to the previous ReplicaSet (64.3 covers it in depth).

Green output

$ bash bad-release.sh
Waiting for deployment "pay-api" rollout to finish: 1 out of 3 new replicas have been updated...
error: timed out waiting for the condition
$ kubectl get pods -n ready
pay-api-6c9d8b7f5d-8wz7n   1/1   Running
pay-api-6c9d8b7f5d-r2m9k   1/1   Running
pay-api-6c9d8b7f5d-4hq2x   1/1   Running
pay-api-85f4c6d9b8-tm5lq   0/1   Running
$ kubectl rollout undo deploy/pay-api -n ready
deployment.apps/pay-api rolled back

Expected result. Only one new pod was created and it never became ready, so the old pods were kept. Without a readiness probe the broken version would have replaced all three.

Example 3: Drain before shutdown: zero errors during scale-down

Goal. Drain before shutdown: zero errors during scale-down.

drain-patch.yaml

spec:
  template:
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: api
          lifecycle:
            preStop:
              exec:
                command:
                  - sh
                  - -c
                  - rm -f /www/ready; sleep 10

load.sh

kubectl run load -n ready --image=busybox:1.36 --restart=Never -- sh -c '
ok=0; bad=0
for i in $(seq 1 300); do
  if wget -q -T 1 -O- http://pay-api >/dev/null 2>&1; then ok=$((ok+1)); else bad=$((bad+1)); fi
  sleep 0.1
done
echo ok=$ok bad=$bad'

What each line does

• preStop — Runs before the container gets SIGTERM.
• rm /www/ready — Readiness fails, so the pod leaves endpoints while still serving in-flight requests.
• sleep 10 — Gives kube-proxy and load balancers time to stop sending new traffic.
• terminationGracePeriodSeconds — Must be longer than the preStop sleep.
• load.sh — Counts good and bad requests during the scale-down.

Green output

$ kubectl patch deploy pay-api -n ready --patch-file drain-patch.yaml
$ kubectl rollout status deploy/pay-api -n ready
$ bash load.sh &
$ kubectl scale deploy pay-api -n ready --replicas=1
$ kubectl logs -n ready load
ok=300 bad=0

Expected result. All 300 requests succeeded while two pods were removed. Without the preStop drain you usually see a few failed requests, because a pod can stop before every node has updated its rules.

Every command, step by step (kind lab with kubectl)

1. Namespace

   kubectl create ns ready
   

   What it does: Lab space.

   namespace/ready created
   

2. Deploy

   kubectl apply -f pay-api.yaml
   

   What it does: Three pods and a Service.

   deployment.apps/pay-api created
   service/pay-api created
   

3. Endpoints

   kubectl get endpointslices -n ready -l kubernetes.io/service-name=pay-api
   

   What it does: Three IPs.

   pay-api-7xk2p   IPv4   8080   10.244.1.5,10.244.2.4,10.244.3.6
   

4. Break readiness

   kubectl exec -n ready $POD -- rm /www/ready
   

   What it does: One pod unready.

5. Check READY

   kubectl get pods -n ready
   

   What it does: 0/1, RESTARTS 0.

   pay-api-6c9d8b7f5d-4hq2x   0/1   Running   0
   

6. Conditions

   kubectl get pod -n ready ${POD#pod/} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
   

   What it does: False.

   False
   

7. Fix it

   kubectl exec -n ready $POD -- sh -c 'echo ok > /www/ready'
   

   What it does: Rejoins after two passes.

8. Bad release

   bash bad-release.sh
   

   What it does: Stalls safely.

   error: timed out waiting for the condition
   

9. Undo

   kubectl rollout undo deploy/pay-api -n ready
   

   What it does: Back to good.

   deployment.apps/pay-api rolled back
   

10. Drain

    kubectl patch deploy pay-api -n ready --patch-file drain-patch.yaml
    

    What it does: preStop added.

    deployment.apps/pay-api patched
    

11. Scale under load

    bash load.sh & kubectl scale deploy pay-api -n ready --replicas=1
    

    What it does: Zero errors.

    ok=300 bad=0
    

12. Clean up

    kubectl delete ns ready
    

    What it does: Ready for 63.3.

    namespace "ready" deleted
    

When it is red 1

Service has no endpoints

Cause. All pods fail readiness, often because they share a broken dependency.

Fix, step by step

1. kubectl describe pod for the probe error.
2. Check the dependency; consider a softer readiness check.

When it is red 2

Pod is 0/1 forever after deploy

Cause. Wrong path or port in readinessProbe.

Fix, step by step

1. Compare probe port with containerPort.
2. curl the path from inside the pod.

When it is red 3

Readiness flaps between ready and not ready

Cause. Threshold too tight or a slow endpoint.

Fix, step by step

1. Raise successThreshold and timeoutSeconds.

When it is red 4

502 errors during every deploy

Cause. Pods stop before load balancers remove them.

Fix, step by step

1. Add a preStop sleep and longer grace period.

When it is red 5

Rollout marked ProgressDeadlineExceeded

Cause. New pods never became ready within progressDeadlineSeconds.

Fix, step by step

1. Fix readiness, then kubectl rollout undo.

Practice tasks

kubectl get pods -n ready

kubectl get pods -n ready

kubectl get endpointslices -n ready -w

kubectl get pods -n ready -w

kubectl rollout status deploy/pay-api -n ready

kubectl rollout history deploy/pay-api -n ready

kubectl rollout status deploy/pay-api -n ready

kubectl logs -n ready load

kubectl get pods -n ready

kubectl logs -n ready load

Recap

• Readiness decides whether a pod receives Service traffic; failure never restarts it.
• Rolling updates wait for readiness, so a probe stops broken releases.
• successThreshold and failureThreshold control how fast pods leave and rejoin.
• A preStop drain plus readiness gives zero-error shutdowns.
• Next lesson: startup probes for slow-booting apps.

FAQ

Q. Should readiness check the database?

It may, if the pod is useless without it. But if all pods share one database, all go unready together. Many teams check only local state and let the app return errors for dependency failures.

Q. Is readiness used by Ingress controllers?

Yes. ingress-nginx and the AWS Load Balancer Controller read endpoints, so unready pods get no traffic there either.

Q. What if I set no readiness probe?

The pod is ready as soon as its containers start. That is fine for trivial apps, risky for real services.

Q. Can readiness and liveness use the same endpoint?

They can, but separate endpoints give clearer behaviour.

Q. Does readiness affect StatefulSet ordering?

Yes, with OrderedReady the next pod waits until the previous one is ready.

Q. What is a readiness gate?

An extra condition, set by a controller such as the AWS Load Balancer Controller, that must be True before the pod is Ready.

Q. Do Jobs need readiness?

Rarely; Jobs do not serve traffic.

Interview questions

1. What does a readiness probe do?

   Model answer: It marks the pod Ready or not; unready pods are removed from Service endpoints but not restarted.

2. How does readiness protect rolling updates?

   Model answer: New pods count as available only when ready, so a broken version stalls the rollout and old pods keep serving.

3. How do you get zero-downtime shutdowns?

   Model answer: A preStop hook that fails readiness and sleeps, a long enough terminationGracePeriodSeconds, and an app that handles SIGTERM gracefully.

4. A Service suddenly has zero endpoints. How do you debug?

   Model answer: Check pod READY status, describe a pod for probe errors, test the readiness path inside the pod, and look for a shared dependency outage.

5. Why set successThreshold above 1?

   Model answer: To stop a pod flapping in and out of the load balancer when its health is unstable.

Samajla ka?