Chapter 63: Kubernetes Health Checks, Self-Healing and Autoscaling: Probes, PDB, HPA, VPA, Cluster Autoscaler and Karpenter

63.8 Load-test lab: probes, PDB, HPA and failures together under real traffic

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Each lesson in this chapter tested one feature alone. Production does not work that way. During an IPL final on JioCinema, the checkout and playback services face a traffic spike, a deploy that someone could not postpone, and maybe a node that fails, all in the same hour. The question an SRE lead asks is simple: with everything turned on, do users see errors? This lab answers it with numbers. We build one checkout API with all three probes, resource requests, topology spread, a PDB and an HPA. We drive it with a proper load generator, k6, running inside the cluster so network noise from your laptop does not matter. k6 runs a script with virtual users, ramps load up and down in stages, and reports request rate, latency percentiles (p95, p99) and the error rate. We also set thresholds, for example p95 under 300 ms and errors under 1%, so the test fails like a unit test when the system does not meet its targets. While the load runs, we cause trouble on purpose, which is chaos testing in its simplest form: delete pods, roll out a new version, and drain a node. Each time we watch the HPA, the endpoints and the k6 error counter. The goal is not zero events; it is zero user-visible failures, or a clear list of what broke and which setting fixes it. Finally we write the results into a short report, the kind you would attach to a production-readiness review at a company like Swiggy or Razorpay: configuration, load profile, what we broke, observed errors and latency, and decisions. This is also the best story to tell in an interview when someone asks how you know your autoscaling works.

What: every term in plain English

• Load test — Sending controlled traffic to measure behaviour under load.
• k6 — Open-source load-testing tool with JavaScript test scripts.
• Virtual users (VUs) — Simulated concurrent users in k6.
• Stages — Ramp-up, steady and ramp-down phases of a test.
• p95 / p99 latency — 95th and 99th percentile response times.
• Thresholds — Pass or fail rules for a test, such as error rate under 1%.
• Error budget — Allowed amount of failure for a service level objective.
• Chaos test — Breaking things on purpose to check resilience.
• Production-readiness review — Checklist and evidence before a service goes live.
• ConfigMap script — Storing the k6 script in a ConfigMap and mounting it into the runner pod.

How: step by step

1. Deploy checkout with probes, requests, spread, PDB and HPA. Full setup.
2. Store the k6 script in a ConfigMap. printf, not heredocs.
3. Run a baseline test. Numbers without chaos.
4. Run the spike test with HPA watching. Scale-out.
5. Add chaos: pod kill, rollout, drain. Resilience.
6. Read thresholds and write the report. Evidence.
7. Clean up the chapter cluster. Chapter done.

Example 1: The fully configured checkout service

Goal. The fully configured checkout service.

checkout.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: lab
spec:
  replicas: 3
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: checkout
      containers:
        - name: api
          image: registry.k8s.io/hpa-example
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 200m
              memory: 64Mi
            limits:
              memory: 128Mi
          startupProbe:
            httpGet: {path: /, port: 80}
            periodSeconds: 2
            failureThreshold: 30
          livenessProbe:
            tcpSocket: {port: 80}
            periodSeconds: 10
          readinessProbe:
            httpGet: {path: /, port: 80}
            periodSeconds: 3
          lifecycle:
            preStop:
              exec:
                command: ["sleep", "10"]
---
apiVersion: v1
kind: Service
metadata:
  name: checkout
  namespace: lab
spec:
  selector:
    app: checkout
  ports:
    - port: 80
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout
  namespace: lab
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: checkout
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout
  namespace: lab
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout
  minReplicas: 3
  maxReplicas: 15
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 60

What each line does

• Probes — Startup for boot, tcp liveness, http readiness: from 63.1 to 63.3.
• Spread and PDB — From 63.4: one failure or drain removes at most one pod.
• Requests — Needed by the HPA from 63.5; no CPU limit, to avoid throttling.
• preStop sleep — Drain from 63.2 so scale-down and rollouts drop no requests.
• HPA 3 to 15 — 60% CPU target, faster scale-down window for the lab.

Green output

$ kubectl create ns lab
$ kubectl apply -f checkout.yaml
$ kubectl get deploy,hpa,pdb -n lab
deployment.apps/checkout   3/3   3   3
horizontalpodautoscaler.autoscaling/checkout   Deployment/checkout   cpu: 1%/60%   3   15   3
poddisruptionbudget.policy/checkout   N/A   1   1

Expected result. Everything from the chapter is in one file. Three pods, spread across workers, with the HPA and PDB ready.

Example 2: k6 baseline and spike tests with thresholds

Goal. k6 baseline and spike tests with thresholds.

make-script.sh

printf '%s\n' \
  "import http from 'k6/http';" \
  "import { check, sleep } from 'k6';" \
  "export const options = {" \
  "  stages: [" \
  "    { duration: '1m', target: 10 }," \
  "    { duration: '3m', target: 80 }," \
  "    { duration: '2m', target: 80 }," \
  "    { duration: '1m', target: 0 }," \
  "  ]," \
  "  thresholds: {" \
  "    http_req_failed: ['rate' + String.fromCharCode(60) + '0.01']," \
  "    http_req_duration: ['p(95)' + String.fromCharCode(60) + '500']," \
  "  }," \
  "};" \
  "export default function () {" \
  "  const r = http.get('http://checkout.lab.svc.cluster.local/');" \
  "  check(r, { 'status 200': (x) => x.status === 200 });" \
  "  sleep(0.1);" \
  "}" > spike.js
kubectl create configmap k6-script -n lab --from-file=spike.js

k6-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: k6-spike
  namespace: lab
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: k6
          image: grafana/k6:0.52.0
          args: ["run", "/scripts/spike.js"]
          volumeMounts:
            - name: scripts
              mountPath: /scripts
      volumes:
        - name: scripts
          configMap:
            name: k6-script

What each line does

• printf — Writes the script line by line, as in every lab in this course.
• fromCharCode(60) — Builds the less-than sign in the threshold without typing it, so the page stays clean; k6 reads rate under 0.01 and p95 under 500 ms.
• Stages — Ramp to 10 users, spike to 80, hold, then ramp down.
• Job — Runs k6 once inside the cluster; backoffLimit 0 so a failed threshold is visible.
• Watch — Run kubectl get hpa -n lab -w in another terminal.

Green output

$ bash make-script.sh
$ kubectl apply -f k6-job.yaml
$ kubectl get hpa checkout -n lab -w
checkout   cpu: 4%/60%     3
checkout   cpu: 182%/60%   3
checkout   cpu: 182%/60%   9
checkout   cpu: 74%/60%    12
checkout   cpu: 58%/60%    12
$ kubectl logs -n lab job/k6-spike | tail -8
     ✓ status 200
     http_req_duration..: avg=121ms p(95)=388ms
     http_req_failed....: 0.21%
     http_reqs..........: 41873  99.6/s

Expected result. The HPA went from 3 to 12 pods during the spike. p95 latency stayed under 500 ms and failures were 0.21%, both inside the thresholds, so the test passed. The small error rate came from the first seconds of the spike, before new pods were ready.

Example 3: Chaos under load: pod kill, rollout and drain

Goal. Chaos under load: pod kill, rollout and drain.

chaos.sh

kubectl delete job k6-spike -n lab
kubectl apply -f k6-job.yaml
sleep 120
echo "1. kill two pods"
kubectl delete pod -n lab $(kubectl get pod -n lab -l app=checkout -o name | head -2 | cut -d/ -f2)
sleep 60
echo "2. rolling update"
kubectl set env deploy/checkout -n lab RELEASE=v2
kubectl rollout status deploy/checkout -n lab
sleep 30
echo "3. drain a worker"
kubectl drain ch63-worker2 --ignore-daemonsets --delete-emptydir-data
kubectl uncordon ch63-worker2
kubectl wait --for=condition=complete job/k6-spike -n lab --timeout=10m
kubectl logs -n lab job/k6-spike | tail -6

What each line does

• Same k6 test — So results compare directly with the baseline.
• Kill two pods — ReplicaSet and readiness should hide it.
• Rollout — maxSurge and readiness plus preStop should drop nothing.
• Drain — PDB lets only one pod go at a time.
• Compare — Error rate and p95 against the baseline.

Green output

$ bash chaos.sh
1. kill two pods
2. rolling update
deployment "checkout" successfully rolled out
3. drain a worker
node/ch63-worker2 drained
     ✓ status 200
     http_req_duration..: avg=134ms p(95)=421ms
     http_req_failed....: 0.34%

Expected result. Even with two pod kills, a full rollout and a drain during the spike, errors stayed at 0.34% and p95 at 421 ms, so thresholds still passed. Run the test again with the preStop hook removed and the PDB deleted: errors jump, which proves each setting earns its place.

Every command, step by step (kind lab with kubectl)

1. Namespace

   kubectl create ns lab
   

   What it does: Lab space.

   namespace/lab created
   

2. metrics-server

   kubectl top nodes
   

   What it does: Still installed from 63.5.

   ch63-worker   88m   601Mi
   

3. Deploy

   kubectl apply -f checkout.yaml
   

   What it does: Full config.

   deployment.apps/checkout created
   horizontalpodautoscaler.autoscaling/checkout created
   

4. Spread

   kubectl get pods -n lab -o wide
   

   What it does: One per worker.

   ch63-worker  ch63-worker2  ch63-worker3
   

5. Script

   bash make-script.sh
   

   What it does: ConfigMap made.

   configmap/k6-script created
   

6. Spike

   kubectl apply -f k6-job.yaml
   

   What it does: Load starts.

   job.batch/k6-spike created
   

7. HPA

   kubectl get hpa checkout -n lab -w
   

   What it does: 3 to 12.

   checkout   cpu: 58%/60%   12
   

8. Results

   kubectl logs -n lab job/k6-spike | tail -8
   

   What it does: Thresholds pass.

   http_req_failed....: 0.21%
   

9. Chaos

   bash chaos.sh
   

   What it does: Failures under load.

   http_req_failed....: 0.34%
   

10. Remove safety

    kubectl delete pdb checkout -n lab
    

    What it does: Compare run.

    poddisruptionbudget.policy "checkout" deleted
    

11. Report

    printf '%s\n' 'Service: checkout' 'Peak: 80 VUs, 99.6 req/s' 'HPA: 3 to 12' 'Errors: 0.21% baseline, 0.34% chaos' 'p95: 388 ms, 421 ms' > report.txt
    

    What it does: Evidence file.

12. Clean up

    kubectl delete ns lab
    kind delete cluster --name ch63
    

    What it does: Chapter done.

    Deleted nodes: ["ch63-control-plane" "ch63-worker" "ch63-worker2" "ch63-worker3"]
    

When it is red 1

k6 shows many connection refused at the start

Cause. Pods not ready or Service name wrong.

Fix, step by step

1. Use the full DNS name and wait for readiness.

When it is red 2

HPA scales but errors stay high

Cause. New pods take too long to become ready, or downstream limits.

Fix, step by step

1. Speed up startup, lower the CPU target, add headroom.

When it is red 3

Threshold failed: http_req_duration

Cause. p95 above target during the ramp.

Fix, step by step

1. Faster scale-up policy, more minReplicas before known events.

When it is red 4

Drain blocked during the test

Cause. PDB allows zero disruptions because pods are not ready.

Fix, step by step

1. Check readiness; wait or scale up.

When it is red 5

k6 Job pod OOMKilled

Cause. Too many VUs for default memory.

Fix, step by step

1. Give the k6 container memory requests and limits.

Practice tasks

kubectl get all -n lab

kubectl get cm k6-script -n lab -o yaml

kubectl logs -n lab job/k6-spike

kubectl get hpa -n lab -w

kubectl get pods -n lab -w

kubectl rollout status deploy/checkout -n lab

kubectl get pdb -n lab

kubectl logs -n lab job/k6-spike

cat report.txt

kubectl get nodes -w

Recap

• Test resilience with every feature on and real traffic, not one feature at a time.
• k6 inside the cluster with stages and thresholds turns resilience into a pass or fail result.
• Probes, preStop, PDB, spread and HPA each remove a class of errors; removing one shows its value.
• Write a short report with numbers; that is what reviewers and interviewers want.
• Chapter 63 done. Next chapter: rolling updates, rollback, blue-green and canary releases.

FAQ

Q. k6 or JMeter or Locust?

All work. k6 is popular with DevOps teams because scripts are code and it runs well in containers and CI. JMeter is common in older QA teams; Locust uses Python.

Q. Should I load test production?

Usually a production-like staging cluster. Some companies test production in quiet hours with careful limits.

Q. How often to run these tests?

Before big events and after major changes; many teams add a short version to the CI/CD pipeline.

Q. How do I see latency inside the cluster?

Prometheus and Grafana, or k6 output to Prometheus remote write.

Q. Can k6 run distributed?

Yes, with the k6 operator running many runner pods.

Q. Why not set a CPU limit?

CPU limits throttle pods during spikes and raise latency; many teams set only requests for CPU.

Q. What is a good error budget?

It depends on the SLO, for example 99.9% success leaves 0.1% for failures.

Interview questions

1. How do you prove your autoscaling works?

   Model answer: Run a staged load test with thresholds in a production-like cluster, watch HPA and node scaling, record latency and error rate, and repeat with failures injected.

2. What causes errors during scale-down or rollouts?

   Model answer: Pods stopping before load balancers remove them; fix with readiness, preStop drains, grace periods and graceful shutdown.

3. What is chaos testing?

   Model answer: Injecting failures such as pod kills, node loss or network delay on purpose to confirm the system keeps meeting its objectives.

4. Your service failed a p95 threshold during the ramp. What do you change?

   Model answer: Pre-scale with higher minReplicas, faster HPA scale-up policies, faster startup, and headroom nodes; then rerun the test.

5. Which Kubernetes features protect availability during a node drain?

   Model answer: PodDisruptionBudgets, multiple replicas, topology spread, readiness probes and preStop drains.

Samajla ka?