Chapter 63: Kubernetes Health Checks, Self-Healing and Autoscaling: Probes, PDB, HPA, VPA, Cluster Autoscaler and Karpenter

63.5 Horizontal Pod Autoscaler: scale pods on CPU, memory and custom metrics

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Traffic is never flat. Hotstar sees a few lakh viewers on a normal evening and crores during an India versus Pakistan match. BookMyShow spikes the minute a big film opens bookings. If you size a Deployment for the peak, you pay for idle pods most of the day. If you size it for the average, the app falls over at the peak. The Horizontal Pod Autoscaler (HPA) fixes this by changing the replica count of a Deployment, StatefulSet or ReplicaSet automatically. How does it decide? Every 15 seconds the HPA controller in kube-controller-manager reads metrics, usually average CPU utilisation from metrics-server, and applies one formula: desiredReplicas = ceil(currentReplicas x currentValue / targetValue). If 4 pods average 90% CPU against a 60% target, it wants ceil(4 x 90 / 60) = 6 pods. It then clamps the answer between minReplicas and maxReplicas and writes it to the scale subresource, just like kubectl scale. Three things must be true for this to work. First, metrics-server must be running so kubectl top returns numbers (you installed it in 59.3; we reuse that file). Second, the pods must have CPU and memory requests, because utilisation is measured as a percentage of the request. No request means the HPA shows unknown and never scales. Third, the app must actually be able to run as many copies, which means stateless pods behind a Service. autoscaling/v2 lets you go further: scale on memory, on several metrics at once (the HPA takes the largest answer), and on custom or external metrics such as requests per second from Prometheus or SQS queue depth through KEDA. The behavior section controls speed: scale up fast when load arrives, scale down slowly so a short dip does not remove pods you need a minute later. The default scale-down stabilisation window is 300 seconds. In this lesson we install metrics-server, run a CPU-hungry app, attach an HPA, generate load, watch replicas climb and fall, add a memory metric, tune behavior, and see how custom metrics plug in.

What: every term in plain English

• HPA — Controller that changes replica count based on metrics.
• metrics-server — Cluster add-on that collects CPU and memory from kubelets for kubectl top and HPA.
• Utilization target — Average usage as a percentage of the pod's request.
• Scaling formula — desired = ceil(current x currentValue / targetValue).
• minReplicas / maxReplicas — Floor and ceiling the HPA never crosses.
• Scale subresource — API endpoint that HPA and kubectl scale both write to.
• behavior — autoscaling/v2 section for scale-up and scale-down policies.
• Stabilization window — Period over which the HPA picks the safest recommendation; 300 seconds for scale-down by default.
• Custom metrics — Per-pod metrics from an adapter such as Prometheus Adapter.
• External metrics — Metrics outside the cluster, such as SQS queue length.
• KEDA — Event-driven autoscaler that feeds external metrics to HPA and can scale to zero.
• TARGETS column — Shows current and target value in kubectl get hpa.

How: step by step

1. Install metrics-server. kubectl top works.
2. Deploy a CPU app with requests. php-apache style test app.
3. Create an HPA at 50% CPU. kubectl autoscale or YAML.
4. Generate load. Replicas climb.
5. Stop load. Replicas fall after the window.
6. Add memory and behavior. autoscaling/v2.
7. Read how custom metrics plug in. Prometheus Adapter and KEDA.
8. Clean up. Ready for VPA.

Example 1: CPU autoscaling from 1 to 8 pods

Goal. CPU autoscaling from 1 to 8 pods.

search-api.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: search-api
  namespace: hpa
spec:
  replicas: 1
  selector:
    matchLabels:
      app: search-api
  template:
    metadata:
      labels:
        app: search-api
    spec:
      containers:
        - name: api
          image: registry.k8s.io/hpa-example
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 200m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi
---
apiVersion: v1
kind: Service
metadata:
  name: search-api
  namespace: hpa
spec:
  selector:
    app: search-api
  ports:
    - port: 80

search-hpa.yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: search-api
  namespace: hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: search-api
  minReplicas: 1
  maxReplicas: 8
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

What each line does

• hpa-example — Official test image that burns CPU on every request.
• requests.cpu 200m — 50% target means about 100m per pod.
• scaleTargetRef — The Deployment to scale.
• 1 to 8 — The HPA stays inside this range.
• Same as — kubectl autoscale deploy search-api -n hpa --cpu-percent=50 --min=1 --max=8.

Green output

$ kubectl apply -f metrics-server.yaml
$ kubectl top nodes
NAME           CPU(cores)   MEMORY(bytes)
ch63-worker    61m          412Mi
$ kubectl create ns hpa
$ kubectl apply -f search-api.yaml -f search-hpa.yaml
$ kubectl get hpa -n hpa
NAME         REFERENCE               TARGETS       MINPODS   MAXPODS   REPLICAS
search-api   Deployment/search-api   cpu: 1%/50%   1         8         1

Expected result. metrics-server is reporting, the HPA can read CPU (1% against a 50% target) and replicas stay at the minimum. Now we add load.

Example 2: Generate load and watch the HPA react

Goal. Generate load and watch the HPA react.

load.sh

kubectl run load -n hpa --image=busybox:1.36 --restart=Never -- sh -c '
while true; do wget -q -O- http://search-api >/dev/null; done'
kubectl get hpa search-api -n hpa -w

What each line does

• Endless loop — One busybox pod sends requests as fast as it can.
• Scale up — Within a minute CPU passes 50% and replicas jump.
• Formula — 1 pod at 250% gives ceil(1 x 250 / 50) = 5.
• Stop load — kubectl delete pod load -n hpa.
• Scale down — After about 5 minutes of low CPU, replicas return to 1.

Green output

$ bash load.sh
NAME         TARGETS         REPLICAS
search-api   cpu: 1%/50%     1
search-api   cpu: 248%/50%   1
search-api   cpu: 248%/50%   5
search-api   cpu: 71%/50%    7
search-api   cpu: 49%/50%    7
$ kubectl delete pod load -n hpa
$ kubectl get hpa search-api -n hpa -w
search-api   cpu: 0%/50%     7
search-api   cpu: 0%/50%     1
$ kubectl describe hpa search-api -n hpa | tail -3
  Normal  SuccessfulRescale  New size: 5; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  New size: 7; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  New size: 1; reason: All metrics below target

Expected result. Replicas went from 1 to 5 to 7 and settled where CPU was just under 50%. After the load stopped, the HPA waited the stabilisation window and scaled back to 1.

Example 3: Memory, several metrics and tuned behavior

Goal. Memory, several metrics and tuned behavior.

search-hpa-v2.yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: search-api
  namespace: hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: search-api
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
    - type: Resource
      resource:
        name: memory
        target:
          type: AverageValue
          averageValue: 100Mi
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 120
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60

What each line does

• Two metrics — The HPA computes replicas for each and uses the largest.
• AverageValue memory — Absolute target instead of a percentage.
• scaleUp 100% per 15s — Can double pods every 15 seconds during a spike.
• scaleDown 1 pod per 60s — Removes pods slowly, after a 2-minute window.
• minReplicas 2 — Never fewer than two, for fault tolerance from 63.4.

Green output

$ kubectl apply -f search-hpa-v2.yaml
$ kubectl get hpa -n hpa
NAME         TARGETS                          MINPODS   MAXPODS   REPLICAS
search-api   cpu: 2%/60%, memory: 9Mi/100Mi   2         10        2
$ bash load.sh
search-api   cpu: 230%/60%, memory: 11Mi/100Mi   4
search-api   cpu: 115%/60%, memory: 11Mi/100Mi   8

Expected result. Both metrics show in TARGETS. Scale-up doubled pods quickly; after the load stops the scale-down removes one pod a minute instead of dropping straight to 2.

Every command, step by step (kind lab with kubectl)

1. metrics-server

   kubectl apply -f metrics-server.yaml
   kubectl rollout status deploy/metrics-server -n kube-system
   

   What it does: The file from 59.3.

   deployment "metrics-server" successfully rolled out
   

2. Top

   kubectl top nodes
   

   What it does: Numbers, not errors.

   ch63-worker   61m   412Mi
   

3. App

   kubectl create ns hpa
   kubectl apply -f search-api.yaml
   

   What it does: With requests.

   deployment.apps/search-api created
   

4. HPA

   kubectl apply -f search-hpa.yaml
   

   What it does: 50% CPU.

   horizontalpodautoscaler.autoscaling/search-api created
   

5. Status

   kubectl get hpa -n hpa
   

   What it does: Targets known.

   search-api   cpu: 1%/50%   1   8   1
   

6. Load

   bash load.sh
   

   What it does: Climbs.

   search-api   cpu: 248%/50%   5
   

7. Pod CPU

   kubectl top pods -n hpa
   

   What it does: Spread over pods.

   search-api-7f9c6d8b5-2xk9m   98m   12Mi
   

8. Stop

   kubectl delete pod load -n hpa
   

   What it does: Load gone.

   pod "load" deleted
   

9. Events

   kubectl describe hpa search-api -n hpa
   

   What it does: Rescale reasons.

   New size: 1; reason: All metrics below target
   

10. v2

    kubectl apply -f search-hpa-v2.yaml
    

    What it does: Memory and behavior.

    horizontalpodautoscaler.autoscaling/search-api configured
    

11. Clean up

    kubectl delete ns hpa
    

    What it does: Keep metrics-server.

    namespace "hpa" deleted
    

When it is red 1

TARGETS shows unknown instead of a CPU percentage

Cause. metrics-server missing or pods have no CPU request.

Fix, step by step

1. kubectl top pods.
2. Add resources.requests.cpu.

When it is red 2

FailedGetResourceMetric: unable to get metrics

Cause. metrics-server not ready or kubelet TLS problem on kind.

Fix, step by step

1. Check its logs; on kind it needs --kubelet-insecure-tls.

When it is red 3

Replicas jump up and down every minute

Cause. Scale-down too fast.

Fix, step by step

1. Raise scaleDown stabilizationWindowSeconds.

When it is red 4

HPA stuck at maxReplicas

Cause. Real load above capacity or a target too low.

Fix, step by step

1. Raise maxReplicas, profile the app, check node capacity.

When it is red 5

Deployment replicas keep resetting after kubectl apply

Cause. replicas field in Git fights the HPA.

Fix, step by step

1. Remove replicas from the Deployment YAML when an HPA owns it.

Practice tasks

kubectl top nodes

kubectl get hpa -n hpa

kubectl get hpa -n hpa -w

kubectl get hpa -n hpa

kubectl describe hpa -n hpa

kubectl get hpa -n hpa

kubectl get hpa search-api -n hpa -o yaml

kubectl get deploy search-api -n hpa

kubectl describe hpa playback-api -n hpa

kubectl explain hpa.spec.metrics.external

Recap

• HPA changes replica count from metrics every 15 seconds.
• It needs metrics-server and CPU or memory requests on pods.
• desired = ceil(current x current value / target), clamped by min and max.
• autoscaling/v2 adds memory, multiple metrics, custom metrics and behavior tuning.
• Next lesson: VPA adjusts requests instead of replicas.

FAQ

Q. Can HPA scale to zero?

Not by default; minReplicas is at least 1. KEDA can scale to zero for event-driven workloads.

Q. HPA on memory for Java apps?

Be careful: JVM memory rarely goes down after load, so memory-based scale-down may never happen. Prefer CPU or request rate.

Q. How do I scale on requests per second?

Expose a metric to Prometheus, install Prometheus Adapter, and use type Pods with that metric name.

Q. Does HPA work with StatefulSets?

Yes, anything with a scale subresource.

Q. Can I use HPA and VPA together?

Not on the same CPU or memory metric; see 63.6.

Q. How fast does it react?

About 15 seconds to read metrics, plus pod start time, plus readiness.

Q. Does EKS include metrics-server?

It is available as an EKS add-on; install it once per cluster.

Q. Why did my HPA not scale even though one pod is at 100% CPU?

The HPA uses the average across all pods. If one pod is hot and four are idle, the average may sit under the target. Look for uneven load, such as sticky sessions from 62.8 or a single long-running request type, and fix the balancing first.

Q. Should minReplicas be 1 in production?

Rarely. One replica has no fault tolerance, so a single pod kill or node drain takes the service down. Use at least 2, and 3 across zones for important services, and raise it before known events like a sale or a match.

Q. Can I see what the HPA will do before it acts?

kubectl describe hpa shows current metrics, conditions such as AbleToScale and ScalingLimited, and recent events. Those conditions explain most surprises.

Interview questions

1. How does the HPA calculate replicas?

   Model answer: desired = ceil(currentReplicas x currentMetric / targetMetric), then bounded by min and max and smoothed by behavior policies and stabilisation windows.

2. Why does an HPA show unknown targets?

   Model answer: No metrics source, or pods lack resource requests for the metric.

3. What is the behavior section for?

   Model answer: Controlling scale-up and scale-down speed with policies and stabilisation windows to avoid flapping.

4. How do you autoscale on queue length?

   Model answer: External metrics via KEDA or a metrics adapter, for example SQS ApproximateNumberOfMessages.

5. HPA reached max but latency is still high. What next?

   Model answer: Check node capacity and Pending pods (cluster autoscaling, 63.7), downstream bottlenecks like the database, and whether the CPU target reflects the real limit.

Samajla ka?