Chapter 59: Kubernetes Pods in Depth: Architecture, YAML Anatomy and Pod Types

59.8 Pod lifecycle: phases, restartPolicy, termination and graceful shutdown

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose we release a new version of the Flipkart checkout service during a sale. Kubernetes replaces the old pods. For a few seconds, some customers see 502 errors and some payments are cut in the middle. Nothing was wrong with the new code. The old pods were killed too fast: they got SIGTERM, did not handle it, and 30 seconds later got SIGKILL while requests were still running. Also, the Service kept sending traffic to pods that were already shutting down. To fix this we must understand the pod lifecycle: the phases Pending, Running, Succeeded, Failed and Unknown; container states; restartPolicy; and exactly what happens on delete: SIGTERM, preStop hook, the grace period, and SIGKILL. In this lesson we watch each phase on a kind cluster, compare the three restart policies, and build a pod that shuts down gracefully. Then we break it with an app that ignores SIGTERM and a preStop hook longer than the grace period.

What: every term in plain English

• Pod phase — A one-word summary: Pending, Running, Succeeded, Failed or Unknown.
• Pending — Accepted by the API, but not all containers are running yet: waiting for scheduling, image pull or init containers.
• Running — Bound to a node and at least one container is running or restarting.
• Succeeded / Failed — All containers have stopped: all with exit 0, or at least one with a non-zero code.
• Unknown — The node stopped reporting, often a network or node failure.
• Container state — Waiting, Running or Terminated, each with a reason like ContainerCreating, CrashLoopBackOff or Completed.
• restartPolicy — Always (default), OnFailure or Never. It applies to all containers in the pod.
• CrashLoopBackOff — The container keeps failing; the kubelet waits 10s, 20s, 40s and so on up to 5 minutes between restarts.
• SIGTERM — The polite stop signal sent first. The app should finish work and exit.
• terminationGracePeriodSeconds — Time between SIGTERM and SIGKILL. Default is 30.
• preStop hook — A command or HTTP call that runs before SIGTERM is sent.
• Conditions — PodScheduled, Initialized, ContainersReady and Ready, each True or False.

How: step by step

1. Watch phases. kubectl get pod -w and kubectl describe pod.
2. Pick restartPolicy. Always for servers, OnFailure for batch Jobs, Never for one-shot debug pods.
3. Handle SIGTERM in the app. Stop accepting new requests, finish current ones, then exit.
4. Add a preStop sleep. sleep 5 lets the Service and load balancer remove the pod first.
5. Set the grace period. terminationGracePeriodSeconds longer than preStop plus the longest request.
6. Add readiness. Ready=False removes the pod from Service endpoints.
7. Test with delete. kubectl delete pod and read the app logs and events.

Example 1: Watch every phase and condition

Goal. Watch every phase and condition.

phases.yaml

apiVersion: v1
kind: Pod
metadata:
  name: short-task
  namespace: shop
spec:
  restartPolicy: Never
  containers:
    - name: task
      image: busybox:1.36
      command: ["sh", "-c", "echo start; sleep 15; echo done"]
---
apiVersion: v1
kind: Pod
metadata:
  name: bad-task
  namespace: shop
spec:
  restartPolicy: Never
  containers:
    - name: task
      image: busybox:1.36
      command: ["sh", "-c", "echo oops; exit 3"]
---
apiVersion: v1
kind: Pod
metadata:
  name: no-image
  namespace: shop
spec:
  containers:
    - name: task
      image: busybox:9.99-does-not-exist

What each line does

• short-task — Goes Pending, then Running for 15 seconds, then Succeeded.
• restartPolicy: Never — No restart, so the final phase is visible.
• exit 3 — Non-zero exit, so bad-task ends Failed.
• busybox:9.99-does-not-exist — Stays Pending with ErrImagePull, then ImagePullBackOff.

Green output

$ kubectl apply -f phases.yaml
$ kubectl get pods -n shop -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,REASON:.status.containerStatuses[0].state.*.reason
NAME         PHASE     REASON
bad-task     Failed    Error
no-image     Pending   ImagePullBackOff
short-task   Running   none
$ kubectl get pod short-task -n shop -o jsonpath='{range .status.conditions[*]}{.type}={.status} {end}{"\n"}'
PodReadyToStartContainers=True Initialized=True Ready=True ContainersReady=True PodScheduled=True
$ sleep 15; kubectl get pod short-task -n shop
NAME         READY   STATUS      RESTARTS   AGE
short-task   0/1     Completed   0          20s

Expected result. STATUS in kubectl get is a friendly mix of phase and container reason. The real phase is in .status.phase.

Example 2: Compare restartPolicy Always, OnFailure and Never

Goal. Compare restartPolicy Always, OnFailure and Never.

restart.yaml

apiVersion: v1
kind: Pod
metadata:
  name: rp-always
  namespace: shop
spec:
  restartPolicy: Always
  containers:
    - name: c
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3; exit 0"]
---
apiVersion: v1
kind: Pod
metadata:
  name: rp-onfailure
  namespace: shop
spec:
  restartPolicy: OnFailure
  containers:
    - name: c
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3; exit 1"]
---
apiVersion: v1
kind: Pod
metadata:
  name: rp-never
  namespace: shop
spec:
  restartPolicy: Never
  containers:
    - name: c
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3; exit 1"]

What each line does

• Always + exit 0 — Restarts even after success. That is why servers use it, and why a finishing script loops in CrashLoopBackOff.
• OnFailure + exit 1 — Restarts only on failure. Jobs use this.
• Never + exit 1 — Stays Failed with the logs kept for reading.

Green output

$ kubectl apply -f restart.yaml
$ sleep 60; kubectl get pods -n shop -l '!app' | grep rp-
rp-always      0/1   CrashLoopBackOff   4 (20s ago)   60s
rp-onfailure   0/1   CrashLoopBackOff   4 (18s ago)   60s
rp-never       0/1   Error              0             60s
$ kubectl get pod rp-always -n shop -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'
0

Expected result. rp-always exits 0 but still shows CrashLoopBackOff. A server pod must never exit. Use a Job for work that finishes.

Example 3: Graceful shutdown with preStop and a grace period

Goal. Graceful shutdown with preStop and a grace period.

graceful.yaml

apiVersion: v1
kind: Pod
metadata:
  name: checkout
  namespace: shop
  labels:
    app: checkout
spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c"]
      args:
        - trap 'echo "SIGTERM: finishing 3 open orders"; sleep 10; echo "clean exit"; exit 0' TERM;
          echo "checkout ready";
          while true; do sleep 1; done
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "echo preStop: waiting for LB > /proc/1/fd/1; sleep 5"]
      readinessProbe:
        exec:
          command: ["true"]
        periodSeconds: 2

What each line does

• terminationGracePeriodSeconds: 45 — Total time allowed: preStop 5s + app 10s fits easily.
• trap ... TERM — The shell catches SIGTERM, like a real app's shutdown handler.
• sleep 10 in trap — Stands in for finishing open requests.
• preStop sleep 5 — Runs first. While it runs, the pod is already removed from Service endpoints.
• /proc/1/fd/1 — Writes to the main process stdout, so the preStop line shows in kubectl logs.

Green output

$ kubectl apply -f graceful.yaml
$ kubectl logs -n shop checkout -f &
checkout ready
$ time kubectl delete pod checkout -n shop
preStop: waiting for LB
SIGTERM: finishing 3 open orders
clean exit
pod "checkout" deleted
real    0m16.4s

Expected result. Order on delete: endpoints removed and preStop runs, then SIGTERM, then the app exits cleanly. SIGKILL never came.

Every command, step by step (kind lab with kubectl)

1. Watch a pod's life

   kubectl apply -f phases.yaml
   kubectl get pod short-task -n shop -w
   

   What it does: Pending, ContainerCreating, Running, Completed.

   short-task   0/1   Pending             0   0s
   short-task   0/1   ContainerCreating   0   0s
   short-task   1/1   Running             0   2s
   short-task   0/1   Completed           0   17s
   

2. Read the phase field

   kubectl get pod short-task -n shop -o jsonpath='{.status.phase}{"\n"}'
   

   What it does: The real phase, not the friendly STATUS.

   Succeeded
   

3. Read the last exit

   kubectl get pod bad-task -n shop -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}{"\n"}'
   

   What it does: 3, as the script returned.

   3
   

4. Logs of a crashed run

   kubectl logs -n shop rp-onfailure --previous
   

   What it does: --previous shows the run before the last restart.

   (output of the last failed run)
   

5. See back-off in events

   kubectl get events -n shop --field-selector involvedObject.name=rp-always | tail -2
   

   What it does: Back-off restarting failed container.

   Warning   BackOff   pod/rp-always   Back-off restarting failed container c
   

6. Graceful delete

   kubectl apply -f graceful.yaml
   kubectl delete pod checkout -n shop
   

   What it does: Takes about 15 seconds, not instantly.

   pod "checkout" deleted
   

7. Ungraceful delete

   kubectl apply -f graceful.yaml
   kubectl delete pod checkout -n shop --grace-period=0 --force
   

   What it does: Skips everything. Only for stuck pods.

   Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated.
   pod "checkout" force deleted
   

8. Watch endpoints during shutdown

   printf '%s\n' \
     'apiVersion: v1' \
     'kind: Service' \
     'metadata:' \
     '  name: checkout' \
     '  namespace: shop' \
     'spec:' \
     '  selector:' \
     '    app: checkout' \
     '  ports:' \
     '    - port: 80' \
     '      targetPort: 80' \
     > checkout-svc.yaml
   kubectl apply -f checkout-svc.yaml -f graceful.yaml
   # in a second terminal: kubectl delete pod checkout -n shop
   kubectl get endpointslices -n shop -l kubernetes.io/service-name=checkout -w
   

   What it does: The full Service file is written by the first command; its selector app: checkout matches the pod in graceful.yaml. With a Service in front, the IP disappears the moment delete starts.

   checkout-abc12   IPv4   80   10.244.2.9   1m
   checkout-abc12   IPv4   80   none         1m
   

9. Clean up

   kubectl delete pod -n shop short-task bad-task no-image rp-always rp-onfailure rp-never
   kubectl delete svc checkout -n shop
   

   What it does: Ready for chapter 60.

   pod "short-task" deleted
   

When it is red 1

Last State: Terminated
  Reason:    Error
  Exit Code: 137
Normal  Killing  Stopping container app (grace period 30s exceeded)

Cause. The app ignored SIGTERM, so it was killed after the grace period.

Fix, step by step

1. Handle SIGTERM in code (Node: process.on('SIGTERM'), Spring: server.shutdown=graceful).
2. Run the app as PID 1 directly, or use tini, so the signal reaches it.
3. Use exec form in the Dockerfile CMD, not shell form.

When it is red 2

Exit Code: 137 after exactly 30s, preStop logs show sleep 40

Cause. preStop is longer than terminationGracePeriodSeconds. The grace timer includes preStop.

Fix, step by step

1. Make the grace period bigger than preStop plus shutdown time.
2. For example preStop 5s, shutdown 20s, grace 40s.

When it is red 3

pod stuck in Terminating for 20 minutes

Cause. Usually a finalizer, or the node is unreachable.

Fix, step by step

1. kubectl get pod -o jsonpath='{.metadata.finalizers}'.
2. If the node is gone, kubectl delete pod --grace-period=0 --force.
3. Check the node with kubectl get nodes.

When it is red 4

STATUS: Unknown   node ch59-worker2 NotReady

Cause. The kubelet on that node stopped reporting.

Fix, step by step

1. kubectl describe node ch59-worker2.
2. After about 5 minutes pods are evicted and controllers make new ones (lesson 63.4).
3. For kind: docker start ch59-worker2.

Practice tasks

kubectl get pods -n shop -o custom-columns=N:.metadata.name,P:.status.phase

kubectl get pod short-task -n shop -o jsonpath='{.status.conditions}'

kubectl logs -n shop rp-onfailure --previous

kubectl get pods -n shop | grep rp-

kubectl get events -n shop --field-selector involvedObject.name=rp-always

kubectl delete pod checkout -n shop

time kubectl delete pod checkout -n shop

preStop sleep 60

process.on('SIGTERM', () => server.close(() => process.exit(0)))

while true; do curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080; done

Recap

• Phases: Pending, Running, Succeeded, Failed, Unknown; containers have their own states.
• restartPolicy Always for servers, OnFailure for Jobs, Never for one-shot pods.
• CrashLoopBackOff means repeated failure with growing waits.
• On delete: endpoints removed and preStop, then SIGTERM, then SIGKILL after the grace period.
• Handle SIGTERM, add a short preStop sleep, and size the grace period.
• Chapter 60 starts the controllers that create and replace pods for you.

FAQ

Q. Why does my Completed pod show CrashLoopBackOff?

restartPolicy Always restarts even after exit 0. Use a Job for finishing work.

Q. Does preStop run on a crash?

No. It runs only when Kubernetes stops the container, such as delete, rollout or eviction.

Q. Is force delete safe?

Only when the node is gone. Otherwise two copies may run for a while.

Q. How long should preStop sleep be?

Usually 5 to 15 seconds, enough for load balancers to update.

Q. Why does my app not get SIGTERM?

A shell wrapper is PID 1 and does not forward signals. Use exec form CMD or tini.

Q. Can I change restartPolicy per container?

Only native sidecars may set Always; otherwise it is pod-wide.

Interview questions

1. Explain the pod lifecycle.

   Model answer: Pending while scheduling, pulling and init; Running once bound with a running container; then Succeeded or Failed when all containers stop; Unknown if the node stops reporting.

2. What happens when a pod is deleted?

   Model answer: It is marked Terminating, removed from endpoints, preStop runs, SIGTERM is sent, and after terminationGracePeriodSeconds remaining processes get SIGKILL.

3. How do you get zero-downtime shutdown?

   Model answer: Handle SIGTERM, use readiness probes, add a preStop sleep, and set a long enough grace period.

4. What is CrashLoopBackOff and how do you debug it?

   Model answer: Repeated container failure with exponential back-off; use describe, logs --previous, exit codes and events.

5. Difference between restart policies?

   Model answer: Always restarts every exit, OnFailure only non-zero exits, Never never restarts.

Samajla ka?