Chapter 63: Kubernetes Health Checks, Self-Healing and Autoscaling: Probes, PDB, HPA, VPA, Cluster Autoscaler and Karpenter

63.6 Vertical Pod Autoscaler: right-size CPU and memory requests from real usage

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

HPA answers 'how many pods?'. It assumes each pod's requests are correct. In real companies they almost never are. A developer at Paytm copies requests from another service, asks for 2 CPUs and 4Gi 'to be safe', and the pod actually uses 150m and 300Mi. Multiply that by hundreds of services and the company pays for nodes that sit mostly empty. The opposite also happens: a request that is too small lets the scheduler pack too many pods on a node, and they get OOMKilled or throttled under load. The Vertical Pod Autoscaler (VPA) watches the real CPU and memory usage of pods over time and recommends better requests. It has three parts. The recommender reads metrics and history and calculates target, lowerBound and upperBound values for each container. The updater evicts pods whose requests are far from the recommendation, when the update mode allows it. The admission controller rewrites the requests of new pods as they are created, so replacement pods start with the new values. updateMode controls how brave VPA is. Off only produces recommendations, which you read and copy into Git yourself. This is the safest way to start and what most teams use for a long time. Initial applies recommendations only when pods are created, never evicting running ones. Recreate (and Auto, which currently behaves like Recreate) evicts pods to apply new values; newer clusters with in-place pod resize support can also apply some changes without a restart. resourcePolicy sets minAllowed and maxAllowed so VPA never asks for something absurd, and controlledResources can limit it to memory only. VPA is not part of Kubernetes core. You install it from the kubernetes autoscaler project, and it needs metrics-server. The main rule to remember: do not let HPA and VPA both act on the same CPU or memory metric for one workload, because VPA changes the request that HPA uses to compute utilisation, and the two loops fight. A common safe combination is HPA on CPU or request rate and VPA in Off mode for recommendations, or VPA on memory only. In this lesson we install VPA, run an over-provisioned app in Off mode and read recommendations, switch to Initial and Recreate to watch requests change, and set limits on what VPA may do.

What: every term in plain English

• VPA — Autoscaler that recommends or sets container requests from real usage.
• Recommender — VPA component that calculates target requests.
• Updater — VPA component that evicts pods whose requests are out of range.
• Admission controller — VPA webhook that rewrites requests on new pods.
• updateMode Off — Recommendations only; nothing changes.
• updateMode Initial — Apply only when pods are created.
• updateMode Recreate / Auto — Evict pods to apply new requests.
• target — Recommended request for normal operation.
• lowerBound / upperBound — Range outside which the updater acts.
• resourcePolicy — Per-container minAllowed, maxAllowed and controlledResources.
• Right-sizing — Matching requests to real usage to save cost and avoid OOM.
• In-place resize — Newer Kubernetes feature to change requests without restarting a pod.

How: step by step

1. Install VPA components. Recommender, updater, admission controller.
2. Deploy an over-provisioned app. 2 CPU request, tiny usage.
3. Create a VPA in Off mode. Read recommendations.
4. Switch to Initial. New pods get new requests.
5. Switch to Recreate with limits. VPA evicts and resizes.
6. Check HPA and VPA conflict rules. Safe combinations.
7. Clean up. Ready for node autoscaling in 63.7.

Example 1: Install VPA and get recommendations in Off mode

Goal. Install VPA and get recommendations in Off mode.

install-vpa.sh

# The VPA code lives in the kubernetes autoscaler project.
# Download a release archive of that project to your laptop, unpack it, then:
cd autoscaler/vertical-pod-autoscaler
./hack/vpa-up.sh
kubectl get pods -n kube-system -l 'app in (vpa-recommender,vpa-updater,vpa-admission-controller)'

wallet-api.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wallet-api
  namespace: vpa
spec:
  replicas: 2
  selector:
    matchLabels:
      app: wallet-api
  template:
    metadata:
      labels:
        app: wallet-api
    spec:
      containers:
        - name: api
          image: nginx:1.27
          resources:
            requests:
              cpu: "2"
              memory: 2Gi
---
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: wallet-api
  namespace: vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: wallet-api
  updatePolicy:
    updateMode: "Off"

What each line does

• vpa-up.sh — Script in the project that installs CRDs, RBAC, the three components and webhook certs.
• Over-provisioned — nginx idles at a few millicores but requests 2 CPU and 2Gi.
• On kind — Two replicas at 2 CPU may not even fit, which shows the real cost of bad requests.
• updateMode Off — Nothing changes; we just read.
• Wait — The recommender needs a few minutes of data; real decisions need days.

Green output

$ bash install-vpa.sh
vpa-admission-controller-6b8f9c7d5-xk2mq   1/1   Running
vpa-recommender-7c9d8b6f4-h3k8n            1/1   Running
vpa-updater-5d7f9c8b6-q9w2t                1/1   Running
$ kubectl create ns vpa
$ kubectl apply -f wallet-api.yaml
$ kubectl get vpa -n vpa
NAME         MODE   CPU   MEM       PROVIDED   AGE
wallet-api   Off    25m   262144k   True       4m
$ kubectl describe vpa wallet-api -n vpa | grep -A12 'Container Recommendations'
    Container Name:  api
    Lower Bound:     Cpu: 25m   Memory: 262144k
    Target:          Cpu: 25m   Memory: 262144k
    Upper Bound:     Cpu: 1     Memory: 1Gi

Expected result. VPA says the pod needs about 25m CPU and 256Mi memory, not 2 CPU and 2Gi. That is an 80-fold CPU over-request. In Off mode you copy these numbers, with a margin, into Git.

Example 2: Initial and Recreate: let VPA change requests

Goal. Initial and Recreate: let VPA change requests.

vpa-recreate.yaml

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: wallet-api
  namespace: vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: wallet-api
  updatePolicy:
    updateMode: "Recreate"
    minReplicas: 2
  resourcePolicy:
    containerPolicies:
      - containerName: api
        minAllowed:
          cpu: 50m
          memory: 128Mi
        maxAllowed:
          cpu: "1"
          memory: 1Gi
        controlledResources: ["cpu", "memory"]

What each line does

• Initial first — Set updateMode Initial and delete a pod: the new one starts with VPA values.
• Recreate — The updater evicts pods whose requests are outside the bounds.
• minReplicas 2 — Updater will not evict if fewer than 2 replicas would remain.
• minAllowed — Never below 50m and 128Mi, even if usage is lower.
• PDB — The updater uses the eviction API, so a PDB from 63.4 is respected.

Green output

$ kubectl patch vpa wallet-api -n vpa --type=merge -p '{"spec":{"updatePolicy":{"updateMode":"Initial"}}}'
$ kubectl delete pod -n vpa $(kubectl get pod -n vpa -o name | head -1 | cut -d/ -f2)
$ kubectl get pod -n vpa -o custom-columns=NAME:.metadata.name,CPU:.spec.containers[0].resources.requests.cpu,MEM:.spec.containers[0].resources.requests.memory
NAME                          CPU   MEM
wallet-api-6c8d9f7b5-2kq8m    2     2Gi
wallet-api-6c8d9f7b5-x9v3n    25m   262144k
$ kubectl apply -f vpa-recreate.yaml
$ kubectl get pod -n vpa -o custom-columns=NAME:.metadata.name,CPU:.spec.containers[0].resources.requests.cpu,MEM:.spec.containers[0].resources.requests.memory
wallet-api-6c8d9f7b5-x9v3n    50m   262144k
wallet-api-6c8d9f7b5-p4w7r    50m   262144k

Expected result. With Initial only the recreated pod changed. With Recreate the updater evicted the remaining old pod, and minAllowed lifted CPU from 25m to 50m. The Deployment YAML still says 2 CPU; the admission controller rewrites pods as they are created.

Example 3: Safe HPA plus VPA combinations

Goal. Safe HPA plus VPA combinations.

vpa-memory-only.yaml

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: search-api
  namespace: vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: search-api
  updatePolicy:
    updateMode: "Initial"
  resourcePolicy:
    containerPolicies:
      - containerName: "*"
        controlledResources: ["memory"]

check.sh

kubectl get hpa,vpa -n vpa
kubectl get hpa search-api -n vpa -o jsonpath='{.spec.metrics[*].resource.name}'; echo
kubectl get vpa search-api -n vpa -o jsonpath='{.spec.resourcePolicy.containerPolicies[0].controlledResources}'; echo

What each line does

• The conflict — HPA on CPU utilisation divides by the CPU request; if VPA changes that request, HPA's math shifts and both keep reacting.
• Pattern 1 — HPA on CPU, VPA in Off mode for advice only.
• Pattern 2 — HPA on CPU, VPA controlling memory only, as here.
• Pattern 3 — HPA on custom metrics such as requests per second, VPA on CPU and memory.
• check.sh — Proves the two do not overlap.

Green output

$ kubectl apply -f search-api.yaml -n vpa
$ kubectl autoscale deploy search-api -n vpa --cpu-percent=50 --min=2 --max=8
$ kubectl apply -f vpa-memory-only.yaml
$ bash check.sh
NAME                                             REFERENCE               TARGETS
horizontalpodautoscaler.autoscaling/search-api   Deployment/search-api   cpu: 2%/50%
NAME                                              MODE      CPU   MEM
verticalpodautoscaler.autoscaling.k8s.io/search-api   Initial         131072k
cpu
["memory"]

Expected result. HPA owns CPU and replica count; VPA only sets memory requests on new pods. Neither changes the number the other one uses.

Every command, step by step (kind lab with kubectl)

1. Install

   bash install-vpa.sh
   

   What it does: Three components.

   vpa-recommender-7c9d8b6f4-h3k8n   1/1   Running
   

2. CRDs

   kubectl get crd | grep verticalpodautoscaler
   

   What it does: API installed.

   verticalpodautoscalers.autoscaling.k8s.io
   

3. App

   kubectl create ns vpa
   kubectl apply -f wallet-api.yaml
   

   What it does: Over-provisioned.

   deployment.apps/wallet-api created
   

4. Pending?

   kubectl get pods -n vpa
   

   What it does: May not fit on kind.

   wallet-api-6c8d9f7b5-2kq8m   0/1   Pending
   

5. Recommendation

   kubectl get vpa -n vpa
   

   What it does: After a few minutes.

   wallet-api   Off   25m   262144k   True
   

6. Details

   kubectl describe vpa wallet-api -n vpa
   

   What it does: Bounds.

   Target: Cpu: 25m Memory: 262144k
   

7. Initial

   kubectl patch vpa wallet-api -n vpa --type=merge -p '{"spec":{"updatePolicy":{"updateMode":"Initial"}}}'
   

   What it does: On create only.

   verticalpodautoscaler.autoscaling.k8s.io/wallet-api patched
   

8. Recreate

   kubectl apply -f vpa-recreate.yaml
   

   What it does: Evicts and resizes.

   verticalpodautoscaler.autoscaling.k8s.io/wallet-api configured
   

9. Updater log

   kubectl logs -n kube-system deploy/vpa-updater | tail -3
   

   What it does: Evictions.

   evicting pod vpa/wallet-api-6c8d9f7b5-2kq8m
   

10. Annotation

    kubectl get pod -n vpa -o jsonpath='{.items[0].metadata.annotations.vpaUpdates}'
    

    What it does: Proof of rewrite.

    Pod resources updated by wallet-api: container 0: cpu request, memory request
    

11. Combine

    bash check.sh
    

    What it does: No overlap.

    cpu
    ["memory"]
    

12. Clean up

    kubectl delete ns vpa
    

    What it does: Keep VPA installed if you like.

    namespace "vpa" deleted
    

When it is red 1

VPA shows no recommendation (PROVIDED False)

Cause. Not enough data yet or metrics-server missing.

Fix, step by step

1. Wait a few minutes; check kubectl top pods.

When it is red 2

Pods restart unexpectedly every few hours

Cause. VPA in Recreate mode keeps evicting.

Fix, step by step

1. Use Initial or Off, widen bounds, add a PDB.

When it is red 3

HPA and VPA fight; replicas and requests keep changing

Cause. Both act on CPU.

Fix, step by step

1. Use one of the safe patterns.

When it is red 4

VPA recommendation is huge after a spike

Cause. Recommender uses peaks with a safety margin.

Fix, step by step

1. Set maxAllowed.

When it is red 5

Requests in Git differ from running pods

Cause. Admission controller rewrote them.

Fix, step by step

1. Expected; copy recommendations into Git if you want them to match.

Practice tasks

kubectl get pods -n kube-system | grep vpa

kubectl get vpa -n vpa

kubectl describe vpa wallet-api -n vpa

kubectl get pod -n vpa -o custom-columns=CPU:.spec.containers[0].resources.requests.cpu

kubectl get pods -n vpa -w

kubectl get vpa wallet-api -n vpa -o yaml

kubectl get pdb -n vpa

bash check.sh

kubectl get vpa -A

kubectl get hpa,vpa -n vpa -w

Recap

• VPA recommends or sets requests from real usage; HPA changes replica count.
• Off gives advice, Initial applies on creation, Recreate evicts to apply.
• Bound VPA with minAllowed and maxAllowed and protect apps with PDBs.
• Do not let HPA and VPA act on the same CPU or memory metric.
• Next lesson: Cluster Autoscaler and Karpenter add and remove nodes.

FAQ

Q. Is VPA built into Kubernetes?

No, it is an add-on from the Kubernetes autoscaler project. EKS, GKE and AKS document how to install it; GKE also offers it as a managed feature.

Q. Does VPA change limits?

It scales limits in proportion to requests by default, keeping your original ratio. You can set controlledValues: RequestsOnly.

Q. Is Off mode really useful?

Yes. Many FinOps teams run VPA only in Off mode, plus tools like Goldilocks that show recommendations in a dashboard.

Q. Does VPA work for StatefulSets?

Yes, with care, since evictions restart database pods.

Q. How long before trusting numbers?

At least a full business cycle, often a week, including peak days.

Q. Can VPA restart pods without eviction?

Newer Kubernetes versions with in-place resize let it change some requests live; check your version.

Q. What about JVM apps?

Set heap size relative to the container limit, or VPA memory changes will not help.

Interview questions

1. What does VPA do?

   Model answer: It analyses real CPU and memory usage and recommends or applies container requests through its recommender, updater and admission controller.

2. Explain VPA update modes.

   Model answer: Off: recommendations only. Initial: apply when pods are created. Recreate or Auto: evict pods whose requests are out of range so they are recreated with new values.

3. Can HPA and VPA run together?

   Model answer: Yes, if they do not use the same resource metric: HPA on CPU or custom metrics, VPA on memory or in Off mode.

4. How would you reduce cluster cost from over-provisioned pods?

   Model answer: Run VPA in Off mode, compare requests with recommendations, right-size through Git, then let the cluster autoscaler or Karpenter remove the freed nodes.

5. Why might VPA cause outages?

   Model answer: Recreate mode evicts pods; without enough replicas and PDBs, evictions reduce capacity.

Samajla ka?