Chapter 75: DevOps Projects Part 5: Kubernetes Cluster Deploy Ingress TLS HPA Probes StatefulSet Blue-Green

75.6 Project 36: Horizontal Pod Autoscaler under load

By Ravindra Bagale Last updated: 28 October 2026

Friends, fixed replica counts waste money or melt under traffic. Today we set CPU requests, create an HPA with min and max, generate load, and watch replicas rise and fall.

What you will build in this project

Deployment with CPU requests.
HPA min 1 max 5 at 50 percent CPU.
Load generator tip.
Watch SuccessfulRescale events.

1. Start from a real company story

At Swiggy in Bengaluru, dinner rush doubled orders. The cart API stayed at two Pods and timed out.

Refunds and vouchers cost about ₹40 lakh that weekend. Leadership asked for autoscaling.

Sana added an HPA on CPU with a hard maxReplicas cap and load-tested it on kind first.

Today we build that HPA lab.

Picture 1: load up then HPA adds Pods then load down then scale in

Ravindra Bagale's Tip

Many students create an HPA without CPU requests. Utilization is then unknown and the HPA sits idle. Always set requests first. Keep that habit.

2. Start and prepare

Step 1: metrics-server tip

Why: Need metrics API. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: metrics tip. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 1: m1

Type this:

echo 'HPA needs metrics-server on kind'

You will see:

HPA needs metrics-server on kind

What it means: metrics-server prerequisite for kubectl top and HPA.

Example 2: m2

Type this:

echo 'kubectl top nodes'

You will see:

kubectl top nodes

What it means: Proof metrics API works after install.

Example 3: m3

Type this:

echo 'HPA adds Pods when CPU rises'

You will see:

HPA adds Pods when CPU rises

What it means: Why HPA exists under traffic spikes.

Step 2: Deploy with requests

Why: Requests enable CPU percent. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: kubectl create deploy. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 4: d1

Type this:

kubectl create deploy hpa-demo --image=nginx:1.27-alpine --replicas=1 2>/dev/null || true; kubectl set resources deploy hpa-demo --requests=cpu=100m,memory=64Mi; kubectl expose deploy hpa-demo --port=80 --name=hpa-demo 2>/dev/null || true; kubectl get deploy hpa-demo -o name

You will see:

deployment.apps/hpa-demo

What it means: Demo Deployment ready for HPA.

Example 5: d2

Type this:

kubectl get deploy hpa-demo -o jsonpath='{.spec.template.spec.containers[0].resources.requests.cpu}'; echo

You will see:

100m

What it means: CPU request set so utilization has a denominator.

Example 6: d3

Type this:

echo 'Without requests, CPU percentage for HPA is undefined'

You will see:

Without requests, CPU percentage for HPA is undefined

What it means: Requests are mandatory for CPU HPA.

Step 3: Write HPA YAML

Why: v2 HPA object. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: printf hpa. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 7: h1

Type this:

mkdir -p /tmp/k8s36 && printf '%s\n' 'apiVersion: autoscaling/v2' 'kind: HorizontalPodAutoscaler' 'metadata: {name: hpa-demo}' 'spec:' '  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: hpa-demo}' '  minReplicas: 1' '  maxReplicas: 5' '  metrics:' '  - type: Resource' '    resource: {name: cpu, target: {type: Utilization, averageUtilization: 50}}' > /tmp/k8s36/hpa.yaml && grep maxReplicas /tmp/k8s36/hpa.yaml

You will see:

maxReplicas: 5

What it means: HPA scales between 1 and 5 on 50 percent CPU.

Example 8: h2

Type this:

kubectl apply -f /tmp/k8s36/hpa.yaml

You will see:

horizontalpodautoscaler.autoscaling/hpa-demo created

What it means: HPA object created.

Example 9: h3

Type this:

kubectl get hpa hpa-demo -o jsonpath='{.spec.minReplicas}-{.spec.maxReplicas}'; echo

You will see:

1-5

What it means: Min and max confirmed.

3. Build the main pieces

Step 4: Describe HPA

Why: Describe shows events. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: kubectl describe. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 10: v1

Type this:

kubectl describe hpa hpa-demo | head -12

You will see:

Name: hpa-demo
Min replicas: 1
Max replicas: 5

What it means: Describe shows targets.

Example 11: v2

Type this:

kubectl api-resources | grep horizontalpodautoscaler | head -1

You will see:

horizontalpodautoscalers  hpa  autoscaling/v2

What it means: HPA resource registered.

Example 12: v3

Type this:

echo 'autoscaling/v2 supports CPU memory and custom metrics'

You will see:

autoscaling/v2 supports CPU memory and custom metrics

What it means: Why v2 over v1.

Step 5: Load generator

Why: busybox loop makes load. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: load tip. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 13: l1

Type this:

echo 'kubectl run load --image=busybox:1.36 --restart=Never -- wget loop to hpa-demo'

You will see:

busybox loop hits the Service

What it means: Load generator shape.

Example 14: l2

Type this:

echo 'kubectl get hpa hpa-demo -w'

You will see:

kubectl get hpa hpa-demo -w

What it means: Watch TARGETS and REPLICAS climb.

Example 15: l3

Type this:

echo 'HPA sync is often 15s; wait a minute before calling it failed'

You will see:

HPA sync is often 15s; wait a minute before calling it failed

What it means: Patience tip.

Step 6: Scale-up under load

Why: Replicas climb. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: watch hpa. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 16: u1

Type this:

echo 'Under load REPLICAS should rise toward maxReplicas'

You will see:

Under load REPLICAS should rise toward maxReplicas

What it means: Expected scale-up behavior.

Example 17: u2

Type this:

echo 'kubectl get pods -l app=hpa-demo'

You will see:

kubectl get pods -l app=hpa-demo

What it means: Pods increase when HPA fires.

Example 18: u3

Type this:

echo 'SuccessfulRescale events appear on kubectl describe hpa'

You will see:

SuccessfulRescale events appear on kubectl describe hpa

What it means: Events show rescale reasons.

4. Automate and verify

Step 7: Scale-down window

Why: Stabilization slows scale-in. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: scale-down tip. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 19: s1

Type this:

echo 'Stop load; after stabilizationWindow replicas fall toward minReplicas'

You will see:

Stop load; after stabilizationWindow replicas fall toward minReplicas

What it means: Scale-down behavior.

Example 20: s2

Type this:

echo 'behavior.scaleDown.stabilizationWindowSeconds avoids flapping'

You will see:

behavior.scaleDown.stabilizationWindowSeconds avoids flapping

What it means: Anti-flapping tip.

Example 21: s3

Type this:

kubectl delete pod load --ignore-not-found; echo stopped

You will see:

stopped

What it means: Load pod removed.

Step 8: Memory and custom tips

Why: Memory optional. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: extra metrics. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 22: x1

Type this:

echo 'Add memory Utilization beside CPU in the same metrics list if needed'

You will see:

Add memory Utilization beside CPU in the same metrics list if needed

What it means: Memory metric tip.

Example 23: x2

Type this:

echo 'Custom metrics need an adapter; stick to CPU in this lab'

You will see:

Custom metrics need an adapter; stick to CPU in this lab

What it means: Scope tip.

Example 24: x3

Type this:

echo 'Event-driven scale for queues often uses KEDA later'

You will see:

Event-driven scale for queues often uses KEDA later

What it means: Preview.

Step 9: Read limits and min/max

Why: Caps protect the cluster. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: jsonpath. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 25: r1

Type this:

kubectl get deploy hpa-demo -o jsonpath='{.spec.template.spec.containers[0].resources.requests.cpu}'; echo

You will see:

100m

What it means: CPU request still set.

Example 26: r2

Type this:

kubectl get hpa hpa-demo -o jsonpath='{.spec.maxReplicas}'; echo

You will see:

What it means: Max replicas capped at 5.

Example 27: r3

Type this:

kubectl get hpa hpa-demo -o jsonpath='{.spec.minReplicas}'; echo

You will see:

What it means: Min replicas is 1.

5. Harden and accept

Step 10: Patch maxReplicas

Why: Patch proves control. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: kubectl patch. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 28: p1

Type this:

kubectl patch hpa hpa-demo --type merge -p '{"spec":{"maxReplicas":4}}' && kubectl get hpa hpa-demo -o jsonpath='{.spec.maxReplicas}'; echo

You will see:

What it means: Patched maxReplicas down to 4.

Example 29: p2

Type this:

kubectl patch hpa hpa-demo --type merge -p '{"spec":{"maxReplicas":5}}' && echo restored

You will see:

restored

What it means: Max restored to 5.

Example 30: p3

Type this:

echo 'Always set maxReplicas so a bad loop cannot create hundreds of Pods'

You will see:

Always set maxReplicas so a bad loop cannot create hundreds of Pods

What it means: Safety habit.

Step 11: Baseline before load

Why: Notebook baseline. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: baseline. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 31: b1

Type this:

echo 'Record baseline replicas before load in your notebook'

You will see:

Record baseline replicas before load in your notebook

What it means: Baseline habit for demos.

Example 32: b2

Type this:

kubectl get hpa hpa-demo -o jsonpath='{.status.currentReplicas}'; echo

You will see:

What it means: Current replicas before artificial load.

Example 33: b3

Type this:

echo 'Never demo HPA on a shared prod cluster'

You will see:

Never demo HPA on a shared prod cluster

What it means: Safety for real jobs.

Step 12: Acceptance checklist

Why: Sana signs off. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: All green. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 34: c1

Type this:

echo 'Optional: kubectl delete hpa hpa-demo deploy hpa-demo svc hpa-demo'

You will see:

Optional: kubectl delete hpa hpa-demo deploy hpa-demo svc hpa-demo

What it means: Cleanup command.

Example 35: c2

Type this:

echo 'Leave metrics-server installed for later monitoring projects'

You will see:

Leave metrics-server installed for later monitoring projects

What it means: Reuse tip.

Example 36: c3

Type this:

echo 'Sana: PASS'

You will see:

Sana: PASS

What it means: Project 36 accepted.

Check yourself

Q1. Who fixed Swiggy cart?

Sana

Q2. HPA API version?

autoscaling/v2

Q3. Min replicas?

Q4. Max replicas?

Q5. Target utilization?

Q6. Must set before CPU HPA?

requests

Friends, you capped an HPA and watched the rescale idea under load. Did you understand? Let us try it once more. Patch maxReplicas and explain why a cap is mandatory.

Lab: try at home

Install metrics-server tip.
Deploy with CPU requests.
Apply HPA v2.
Generate load and watch.
Stop load and confirm scale-down window.

Recap

metrics-server feeds HPA.
requests define 100 percent.
maxReplicas is a safety cap.
Stabilization stops flapping.
Test on kind before prod.