Chapter 63: Kubernetes Health Checks, Self-Healing and Autoscaling: Probes, PDB, HPA, VPA, Cluster Autoscaler and Karpenter

63.3 Startup probes: give slow apps time to boot without weakening liveness

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Some apps take a long time to start. A Java monolith at a bank like HDFC can need two minutes to load classes, run Flyway migrations and warm caches. An Elasticsearch node may take longer while it recovers shards. In 63.1 you saw what happens when liveness runs during that boot: the kubelet kills the container before it ever finishes starting, and the pod lands in CrashLoopBackOff. The old fix was a huge initialDelaySeconds on the liveness probe, for example 180. That works for the boot, but it has a cost: liveness now waits 180 seconds after every restart, and boot time is not always the same. If the app boots in 30 seconds today, you wasted 150 seconds of protection. If it boots in 200 seconds after a big migration, it still gets killed. The startup probe is the clean answer. While a startup probe is defined and has not yet succeeded, the kubelet does not run liveness or readiness at all. It runs only the startup probe, and the app gets failureThreshold multiplied by periodSeconds to start, for example 30 times 10 seconds, which is five minutes. The moment the startup probe succeeds once, it stops forever for that container, and liveness and readiness take over with their normal, strict timing. If startup never succeeds within the budget, the container is killed and restarted, just like a liveness failure. So the pattern is: startupProbe with a generous budget, livenessProbe with tight timing, readinessProbe for traffic. Usually all three hit endpoints in the same app, often the same /livez path for startup and liveness. In this lesson we fix the slow-app from 63.1 with a startup probe, measure the boot budget, combine all three probes in one realistic Deployment, and see what happens when an app takes longer than the budget.

What: every term in plain English

• Startup probe — A probe that runs first; liveness and readiness wait until it succeeds once.
• Boot budget — failureThreshold times periodSeconds: the maximum time allowed to start.
• Probe order — startup first, then liveness and readiness together.
• Started — Container status field that turns true when the startup probe passes.
• initialDelaySeconds — Still works on startup probes, but rarely needed.
• Kill on budget — If startup fails failureThreshold times, the container is restarted.
• Warm-up — Work an app does before serving: caches, migrations, JIT.
• Flyway / Liquibase — Database migration tools that often run at boot in Java apps.
• Three-probe pattern — startup with a long budget, liveness tight, readiness for traffic.
• kube-probe — User agent the kubelet sends with HTTP probes.

How: step by step

1. Re-run slow-app from 63.1 to see the crash loop. The problem.
2. Add a startup probe with a 60-second budget. The app boots.
3. Read the started field. Proof the probe passed.
4. Build a three-probe Deployment. Production pattern.
5. Push boot time over the budget. See the startup kill.
6. Compare with initialDelaySeconds. Why startup is better.
7. Clean up. Ready for self-healing in 63.4.

Example 1: Fix the slow app with a startup probe

Goal. Fix the slow app with a startup probe.

slow-app-fixed.yaml

apiVersion: v1
kind: Pod
metadata:
  name: slow-app
  namespace: startup
spec:
  containers:
    - name: app
      image: busybox:1.36
      command:
        - sh
        - -c
        - |
          sleep 20
          mkdir -p /www && echo ok > /www/healthz
          httpd -f -p 8080 -h /www
      startupProbe:
        httpGet:
          path: /healthz
          port: 8080
        periodSeconds: 5
        failureThreshold: 12
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        periodSeconds: 3
        failureThreshold: 2

What each line does

• Same app — Still needs 20 seconds to start, exactly as in 63.1.
• Budget — 12 failures times 5 seconds gives 60 seconds to boot.
• Liveness unchanged — Still tight: two failures in 6 seconds restarts the container, but only after startup.
• Started field — Turns true about 20 to 25 seconds in.
• No restarts — RESTARTS stays 0.

Green output

$ kubectl create ns startup
$ kubectl apply -f slow-app-fixed.yaml
$ kubectl get pod slow-app -n startup -w
NAME       READY   STATUS    RESTARTS   AGE
slow-app   0/1     Running   0          5s
slow-app   1/1     Running   0          24s
$ kubectl get pod slow-app -n startup -o jsonpath='{.status.containerStatuses[0].started}'
true
$ kubectl describe pod slow-app -n startup | grep -i startup
    Startup:   http-get http://:8080/healthz delay=0s timeout=1s period=5s #success=1 #failure=12
  Warning  Unhealthy  Startup probe failed: Get "http://10.244.2.7:8080/healthz": connect: connection refused

Expected result. The startup probe failed a few times while the app booted, which is expected and harmless. Once it passed, liveness took over and the pod never restarted.

Example 2: The three-probe pattern for a slow Java-style service

Goal. The three-probe pattern for a slow Java-style service.

loans-api.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: loans-api
  namespace: startup
spec:
  replicas: 2
  selector:
    matchLabels:
      app: loans-api
  template:
    metadata:
      labels:
        app: loans-api
    spec:
      containers:
        - name: api
          image: busybox:1.36
          env:
            - name: BOOT_SECONDS
              value: "45"
          command:
            - sh
            - -c
            - |
              mkdir -p /www
              sleep $BOOT_SECONDS
              echo ok > /www/livez
              sleep 5
              echo ok > /www/readyz
              httpd -f -p 8080 -h /www
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet:
              path: /livez
              port: 8080
            periodSeconds: 10
            failureThreshold: 18
          livenessProbe:
            httpGet:
              path: /livez
              port: 8080
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
            periodSeconds: 5
            failureThreshold: 2

What each line does

• BOOT_SECONDS 45 — Simulates class loading and migrations.
• Startup budget — 18 times 10 seconds gives three minutes, more than the slowest boot.
• Liveness — Restarts within 30 seconds if the app hangs after boot.
• Readiness — Separate /readyz that appears 5 seconds after /livez, like a cache warm-up.
• Why two paths — Startup and liveness ask 'alive?', readiness asks 'can I serve?'.

Green output

$ kubectl apply -f loans-api.yaml
$ kubectl get pods -n startup -l app=loans-api -w
loans-api-5b7c9d8f6-2kq8m   0/1   Running   0   10s
loans-api-5b7c9d8f6-2kq8m   0/1   Running   0   52s
loans-api-5b7c9d8f6-2kq8m   1/1   Running   0   58s
$ kubectl get pod -n startup -l app=loans-api -o jsonpath='{range .items[*]}{.status.containerStatuses[0].started}{" "}{.status.containerStatuses[0].ready}{"\n"}{end}'
true true
true true

Expected result. Around 50 seconds started turns true (startup passed), and about 5 seconds later the pod becomes ready. Zero restarts, and liveness is still strict for the rest of the pod's life.

Example 3: When boot time exceeds the budget

Goal. When boot time exceeds the budget.

over-budget.sh

kubectl set env deploy/loans-api -n startup BOOT_SECONDS=240
kubectl get pods -n startup -l app=loans-api -w

What each line does

• 240 seconds — Longer than the 180-second budget.
• Result — The kubelet kills the container after the 18th failure.
• Old pods — Still serve, because the new pods never become ready and the rollout stalls.
• Fix — Raise failureThreshold, or make the app boot faster.
• Lesson — Measure real boot times in production and set the budget with a margin, for example twice the slowest boot.

Green output

$ bash over-budget.sh
loans-api-7d9f8c6b4-x7v2n   0/1   Running   0             2m50s
loans-api-7d9f8c6b4-x7v2n   0/1   Running   1 (1s ago)    3m1s
$ kubectl describe pod -n startup loans-api-7d9f8c6b4-x7v2n | grep Killing
  Normal  Killing  Container api failed startup probe, will be restarted
$ kubectl rollout undo deploy/loans-api -n startup

Expected result. After three minutes of failed startup checks the container was restarted, and it would loop forever. The rolling update protected users, and undo returned the working version.

Every command, step by step (kind lab with kubectl)

1. Namespace

   kubectl create ns startup
   

   What it does: Lab space.

   namespace/startup created
   

2. Reproduce

   kubectl apply -f slow-app.yaml -n startup
   

   What it does: The 63.1 crash loop.

   slow-app   0/1   CrashLoopBackOff
   

3. Replace

   kubectl delete pod slow-app -n startup
   kubectl apply -f slow-app-fixed.yaml
   

   What it does: Startup probe added.

   pod/slow-app created
   

4. Watch

   kubectl get pod slow-app -n startup -w
   

   What it does: Ready at about 24 seconds.

   slow-app   1/1   Running   0   24s
   

5. Started

   kubectl get pod slow-app -n startup -o jsonpath='{.status.containerStatuses[0].started}'
   

   What it does: true.

   true
   

6. Three probes

   kubectl apply -f loans-api.yaml
   

   What it does: Realistic service.

   deployment.apps/loans-api created
   

7. Describe

   kubectl describe pod -n startup -l app=loans-api | grep -E 'Liveness|Readiness|Startup'
   

   What it does: All three listed.

   Liveness:   http-get http://:8080/livez delay=0s timeout=2s period=10s #success=1 #failure=3
   Readiness:  http-get http://:8080/readyz delay=0s timeout=1s period=5s #success=1 #failure=2
   Startup:    http-get http://:8080/livez delay=0s timeout=1s period=10s #success=1 #failure=18
   

8. Boot time

   kubectl get pod -n startup -l app=loans-api -o jsonpath='{.items[0].status.conditions}'
   

   What it does: Compare times.

   [{"type":"Ready","lastTransitionTime":"..."}]
   

9. Over budget

   bash over-budget.sh
   

   What it does: Startup kill.

   Container api failed startup probe, will be restarted
   

10. Undo

    kubectl rollout undo deploy/loans-api -n startup
    

    What it does: Recover.

    deployment.apps/loans-api rolled back
    

11. Clean up

    kubectl delete ns startup
    

    What it does: Ready for 63.4.

    namespace "startup" deleted
    

When it is red 1

Startup probe failed: connection refused (many times)

Cause. Normal during boot.

Fix, step by step

1. Ignore unless the container gets killed.

When it is red 2

Container failed startup probe, will be restarted

Cause. Boot takes longer than failureThreshold times periodSeconds.

Fix, step by step

1. Measure boot time and raise the budget.

When it is red 3

Liveness never seems to run

Cause. Startup probe never succeeds, or points at the wrong path.

Fix, step by step

1. Check the started field.

When it is red 4

App restarts every time a migration runs

Cause. Migrations run inside the main container at boot.

Fix, step by step

1. Move migrations to an init container or a Job (60.6).
2. Or raise the startup budget.

When it is red 5

Readiness true before the app is warm

Cause. Readiness uses the same cheap path as liveness.

Fix, step by step

1. Add a separate warm-up check.

Practice tasks

kubectl get pod slow-app -n startup

kubectl get pod slow-app -n startup -w

kubectl get pod slow-app -n startup -o jsonpath='{.status.containerStatuses[0].started}'

kubectl describe deploy loans-api -n startup

kubectl get pods -n startup -w

kubectl get pods -n startup -w

kubectl rollout status deploy/loans-api -n startup

kubectl delete pod -n startup -l app=loans-api

kubectl get pods -n startup

kubectl get pod -n startup -o wide

Recap

• A startup probe holds back liveness and readiness until the app has booted.
• Budget equals failureThreshold times periodSeconds; set it from measured boot times.
• After it passes, startup never runs again and strict liveness takes over.
• Use all three probes for slow, real-world services.
• Next lesson: self-healing and fault tolerance across pods and nodes.

FAQ

Q. Do fast apps need a startup probe?

No. If an app starts in a few seconds, readiness and liveness with sensible defaults are enough.

Q. Can the startup probe use exec?

Yes, any mechanism works, for example checking that a migration marker file exists.

Q. Does startup affect readiness?

Yes, readiness does not run until startup passes, so the pod stays 0/1 during boot.

Q. What about init containers?

They run before the main container even starts; startup probes run after it starts. Use both when needed.

Q. Does a startup probe slow down normal restarts?

No, it passes as soon as the app is up.

Q. Is there a maximum budget?

No hard limit, but very long budgets hide real hangs during boot.

Q. Which Kubernetes version added it?

It became stable in 1.20, so every current cluster supports it.

Q. How do I measure boot time in production?

Compare the container startedAt time with the time the Ready condition turned True, from kubectl get pod -o yaml or from kube-state-metrics in Prometheus. Track the slowest boot over a week, including boots right after a big migration or a cold node pull.

Q. Should the startup probe hit the same path as liveness?

Usually yes. Both ask whether the process is alive. Using the same cheap endpoint means that once startup passes, liveness will pass too, so there is no surprise restart at the handover.

Interview questions

1. What is a startup probe?

   Model answer: A probe that runs before liveness and readiness and gives slow apps failureThreshold times periodSeconds to start.

2. Why is a startup probe better than initialDelaySeconds?

   Model answer: It ends as soon as the app is up, so liveness protection starts early, and it adapts to varying boot times.

3. What happens when the startup probe fails its threshold?

   Model answer: The kubelet kills and restarts the container, like a liveness failure.

4. Describe the three-probe pattern.

   Model answer: Startup with a generous budget, liveness with tight timing on a cheap endpoint, and readiness on an endpoint that says the app can serve traffic.

5. A Java service restarts in a loop after a migration added two minutes to boot. Fix?

   Model answer: Raise the startup budget or move the migration to a Job or init container; do not loosen liveness.

Samajla ka?