Chapter 63: Kubernetes Health Checks, Self-Healing and Autoscaling: Probes, PDB, HPA, VPA, Cluster Autoscaler and Karpenter

63.7 Cluster Autoscaler and Karpenter: add and remove nodes automatically

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

HPA can ask for 30 pods, but if the nodes are full those pods simply sit in Pending. During a Big Billion Days sale at Flipkart, pods are not the bottleneck; machines are. And at 4 AM, when traffic is a tenth of the peak, half the nodes sit idle while the AWS bill keeps running. Node autoscaling closes the loop: add machines when pods cannot be scheduled, remove them when they are underused. There are two main tools on AWS. Cluster Autoscaler (CA) is the original. It watches for pods that are Pending because of insufficient CPU, memory or other constraints, finds an Auto Scaling group (node group) whose instance type would fit them, and raises that group's desired count. For scale-down, it looks for nodes whose requested resources stay below a threshold (50% by default) for about 10 minutes, checks that their pods can move elsewhere while respecting PDBs, drains them and shrinks the group. CA only works with node groups you define in advance, each with a fixed instance type or a small mix. Karpenter is the newer approach, created at AWS and now used widely on EKS. It skips node groups. It reads the Pending pods' requests, node selectors, affinities and tolerations, then calls EC2 directly to launch the cheapest instance type that fits, often within a minute. You describe what is allowed in a NodePool (instance families, sizes, zones, Spot or On-Demand, limits) and an EC2NodeClass (AMI, subnets, security groups, IAM role). Karpenter also consolidates: it keeps checking whether pods could be packed onto fewer or cheaper nodes and replaces nodes to save money, respecting PDBs and disruption budgets. Both depend on one thing you have already learned: accurate requests. Autoscalers schedule on requests, not actual usage. Inflated requests launch nodes you do not need; missing requests make nodes look empty. That is why VPA from 63.6 and node autoscaling go together. kind cannot add real machines, so this lesson simulates the trigger on kind (Pending pods and node capacity), then walks through Cluster Autoscaler and Karpenter configuration for EKS with eksctl, including the commands you would run and the output you would see.

What: every term in plain English

• Node autoscaling — Adding and removing worker nodes based on scheduling needs.
• Pending pod — A pod the scheduler cannot place, for example Insufficient cpu.
• Cluster Autoscaler — Scales cloud node groups up for Pending pods and down for idle nodes.
• Node group / ASG — Group of identical instances that CA resizes.
• Scale-down threshold — Node requested utilisation below which CA considers removal; default 0.5.
• Karpenter — Node provisioner that launches right-sized EC2 instances directly for Pending pods.
• NodePool — Karpenter resource with requirements, limits and disruption settings.
• EC2NodeClass — Karpenter resource with AMI, subnets, security groups and IAM role.
• Consolidation — Karpenter replacing or removing nodes to pack pods more cheaply.
• Spot — Discounted EC2 capacity that AWS can reclaim with two minutes notice.
• Over-provisioning — Low-priority placeholder pods that keep spare capacity ready.
• safe-to-evict — Pod annotation that tells CA or Karpenter whether a pod may be moved.

How: step by step

1. Fill the kind cluster until pods go Pending. The autoscaling trigger.
2. Read the scheduler's reason. Insufficient cpu.
3. Review Cluster Autoscaler setup on EKS. Node group plus CA deployment.
4. Review Karpenter NodePool and EC2NodeClass. Groupless provisioning.
5. Watch scale-up and consolidation on EKS. Commands and output.
6. Add over-provisioning pods. Instant headroom.
7. Clean up. Ready for the load-test lab.

Example 1: The trigger on kind: Pending pods and Insufficient cpu

Goal. The trigger on kind: Pending pods and Insufficient cpu.

fill.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sale-api
  namespace: nodes
spec:
  replicas: 3
  selector:
    matchLabels:
      app: sale-api
  template:
    metadata:
      labels:
        app: sale-api
    spec:
      containers:
        - name: api
          image: nginx:1.27
          resources:
            requests:
              cpu: "1"
              memory: 256Mi

capacity.sh

for n in $(kubectl get nodes -o name); do
  printf '%s ' $n
  kubectl get $n -o jsonpath='{.status.allocatable.cpu}'; echo
done
kubectl describe nodes | grep -A6 'Allocated resources' | grep cpu

What each line does

• 1 CPU per pod — Large requests fill nodes quickly.
• capacity.sh — Shows allocatable CPU and what is already requested on each node.
• Scale to 30 — More pods than the kind nodes can hold.
• Pending — The scheduler writes FailedScheduling with the reason.
• On EKS — This exact event is what CA and Karpenter react to.

Green output

$ kubectl create ns nodes
$ kubectl apply -f fill.yaml
$ bash capacity.sh
node/ch63-control-plane 8
node/ch63-worker 8
$ kubectl scale deploy sale-api -n nodes --replicas=30
$ kubectl get pods -n nodes --field-selector=status.phase=Pending | wc -l
9
$ kubectl describe pod -n nodes $(kubectl get pod -n nodes --field-selector=status.phase=Pending -o name | head -1 | cut -d/ -f2) | tail -2
  Warning  FailedScheduling  0/4 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3 Insufficient cpu.

Expected result. About 22 pods fit on the three workers; the rest are Pending with Insufficient cpu. On kind they wait forever. On EKS with CA or Karpenter, new nodes would appear within a few minutes.

Example 2: Cluster Autoscaler on EKS with an eksctl node group

Goal. Cluster Autoscaler on EKS with an eksctl node group.

cluster.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: shop-prod
  region: ap-south-1
iam:
  withOIDC: true
managedNodeGroups:
  - name: general
    instanceType: m6i.large
    minSize: 2
    maxSize: 10
    desiredCapacity: 2
    propagateASGTags: true
    tags:
      k8s.io/cluster-autoscaler/enabled: "true"
      k8s.io/cluster-autoscaler/shop-prod: "owned"

ca-args.yaml

containers:
  - name: cluster-autoscaler
    image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.30.0
    command:
      - ./cluster-autoscaler
      - --cloud-provider=aws
      - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/shop-prod
      - --balance-similar-node-groups
      - --skip-nodes-with-system-pods=false
      - --scale-down-utilization-threshold=0.5
      - --scale-down-unneeded-time=10m
      - --expander=least-waste

What each line does

• ASG tags — CA discovers node groups by these two tags.
• minSize / maxSize — CA never goes outside this range.
• IRSA — CA's service account needs an IAM role allowed to call autoscaling:SetDesiredCapacity and describe calls; withOIDC enables that.
• Image version — Match the CA minor version to your Kubernetes minor version.
• least-waste — When several groups fit, choose the one that leaves the least idle CPU and memory.

Green output

$ eksctl create cluster -f cluster.yaml
$ kubectl scale deploy sale-api -n nodes --replicas=30
$ kubectl logs -n kube-system deploy/cluster-autoscaler | grep -i scale-up
I1025 scale_up.go: Scale-up: setting group eks-general-5ac8 size to 6
$ kubectl get nodes
ip-10-0-1-21.ap-south-1.compute.internal   Ready   3m
ip-10-0-2-48.ap-south-1.compute.internal   Ready   1m
...
$ kubectl scale deploy sale-api -n nodes --replicas=3
$ kubectl logs -n kube-system deploy/cluster-autoscaler | grep -i 'scale_down'
I1025 scale_down.go: ip-10-0-2-48... was unneeded for 10m0s, removing

Expected result. CA raised the ASG from 2 to 6 nodes for the Pending pods, then, after the load dropped and nodes were unneeded for 10 minutes, drained and removed them. New nodes take a few minutes because EC2 must boot and join.

Example 3: Karpenter NodePool with Spot, consolidation and limits

Goal. Karpenter NodePool with Spot, consolidation and limits.

nodepool.yaml

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["ap-south-1a", "ap-south-1b", "ap-south-1c"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  limits:
    cpu: "200"
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
    budgets:
      - nodes: "10%"
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  role: KarpenterNodeRole-shop-prod
  amiSelectorTerms:
    - alias: al2023@latest
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: shop-prod
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: shop-prod

What each line does

• requirements — Any c, m or r family newer than generation 5, Spot or On-Demand, in three AZs.
• Cheapest fit — Karpenter picks the cheapest instance that fits the Pending pods, preferring Spot.
• limits cpu 200 — Hard cap on total CPU this pool may launch: a cost guardrail.
• consolidation — Removes empty nodes and replaces underused ones with smaller or cheaper ones.
• budgets 10% — At most 10% of nodes disrupted at once; PDBs are also respected.

Green output

$ kubectl apply -f nodepool.yaml
$ kubectl scale deploy sale-api -n nodes --replicas=30
$ kubectl get nodeclaims
NAME            TYPE         CAPACITY   ZONE          READY   AGE
general-7xk2p   c6a.4xlarge  spot       ap-south-1b   True    52s
general-q9w3n   m6i.2xlarge  spot       ap-south-1a   True    50s
$ kubectl scale deploy sale-api -n nodes --replicas=3
$ kubectl logs -n kube-system deploy/karpenter | grep disrupt
disrupting nodeclaim(s) via delete, terminating 1 nodes (2 pods) general-q9w3n/m6i.2xlarge/spot

Expected result. Karpenter launched two right-sized Spot instances in under a minute, with no node group, and after scale-down it consolidated by removing a node it no longer needed.

Every command, step by step (kind lab with kubectl)

1. Namespace

   kubectl create ns nodes
   

   What it does: Lab space.

   namespace/nodes created
   

2. Fill

   kubectl apply -f fill.yaml
   kubectl scale deploy sale-api -n nodes --replicas=30
   

   What it does: Too many pods.

   deployment.apps/sale-api scaled
   

3. Capacity

   bash capacity.sh
   

   What it does: What fits.

   cpu   7950m (99%)
   

4. Pending

   kubectl get pods -n nodes --field-selector=status.phase=Pending
   

   What it does: Unschedulable.

   sale-api-6d8f9c7b5-x2k9m   0/1   Pending
   

5. Reason

   kubectl get events -n nodes --field-selector reason=FailedScheduling | tail -1
   

   What it does: Insufficient cpu.

   3 Insufficient cpu
   

6. Priority class

   kubectl create priorityclass overprovision --value=-10 --description='placeholder pods'
   

   What it does: For headroom pods.

   priorityclass.scheduling.k8s.io/overprovision created
   

7. Headroom pods

   kubectl create deploy headroom -n nodes --image=registry.k8s.io/pause:3.9 --replicas=2
   kubectl set resources deploy headroom -n nodes --requests=cpu=1
   kubectl patch deploy headroom -n nodes -p '{"spec":{"template":{"spec":{"priorityClassName":"overprovision"}}}}'
   

   What it does: Preemptible placeholders.

   deployment.apps/headroom patched
   

8. Preemption

   kubectl get events -n nodes | grep -i preempt
   

   What it does: Real pods evict placeholders.

   Preempted by nodes/sale-api-6d8f9c7b5-x2k9m
   

9. EKS CA

   kubectl logs -n kube-system deploy/cluster-autoscaler | grep -i scale-up
   

   What it does: On a real cluster.

   Scale-up: setting group eks-general-5ac8 size to 6
   

10. EKS Karpenter

    kubectl get nodepools,nodeclaims
    

    What it does: On a real cluster.

    nodepool.karpenter.sh/general   True
    

11. Clean up

    kubectl delete ns nodes
    kubectl delete priorityclass overprovision
    

    What it does: Ready for 63.8.

    namespace "nodes" deleted
    

When it is red 1

Pods Pending but CA does nothing

Cause. No node group fits the pod (too big, wrong selector or taint), or the group is at maxSize.

Fix, step by step

1. Read CA logs for NotTriggerScaleUp.
2. Add a group or raise maxSize.

When it is red 2

CA never scales down

Cause. Pods with local storage, no controller, kube-system pods or strict PDBs block it.

Fix, step by step

1. Use safe-to-evict annotations and sane PDBs.

When it is red 3

Karpenter launches nothing

Cause. NodePool requirements or limits exclude every instance type, or EC2NodeClass selectors find no subnets.

Fix, step by step

1. kubectl describe nodepool and check karpenter logs.

When it is red 4

Too many node replacements

Cause. Aggressive consolidation.

Fix, step by step

1. Set consolidateAfter higher and disruption budgets.

When it is red 5

Spot nodes vanish with pods

Cause. Spot interruption.

Fix, step by step

1. Run several replicas spread over zones and use interruption handling in Karpenter.

Practice tasks

kubectl get pods -n nodes

kubectl get events -n nodes

bash capacity.sh

kubectl get pods -n nodes -o wide

kubectl get events -n nodes | grep -i preempt

cat ca-args.yaml

kubectl explain nodepool.spec.template.spec.requirements

kubectl annotate pod NAME -n nodes karpenter.sh/do-not-disrupt=true

kubectl apply --dry-run=client -f nodepool.yaml

kubectl get nodes -L karpenter.sh/capacity-type

Recap

• Pending pods trigger node scale-up; idle nodes trigger scale-down.
• Cluster Autoscaler resizes predefined node groups.
• Karpenter launches right-sized instances directly and consolidates to save cost.
• Both decide on requests, respect PDBs and need guardrails like max size or limits.
• Next lesson: a full load-test lab that ties probes, HPA and node scaling together.

FAQ

Q. CA or Karpenter for EKS?

Karpenter is the common choice for new EKS clusters because it is faster and cheaper with mixed instance types. CA is still common on older clusters and other clouds.

Q. Does GKE or AKS need these?

GKE and AKS have built-in cluster autoscalers, and GKE has node auto-provisioning similar to Karpenter.

Q. How fast is scale-up?

CA usually takes 2 to 5 minutes; Karpenter often under a minute, plus image pull time.

Q. How do I keep headroom?

Low-priority pause pods that real pods preempt, so a new node starts while users are already served.

Q. Is Spot safe for production?

For stateless, replicated services, yes, with spreading, PDBs and interruption handling.

Q. Can CA and Karpenter run together?

Avoid managing the same nodes with both.

Q. What stops a runaway bill?

maxSize on node groups, limits on NodePools, and budgets and alerts in AWS.

Interview questions

1. How does Cluster Autoscaler decide to add a node?

   Model answer: It simulates scheduling Pending pods on a new node from each node group and scales up the group that fits, using the configured expander.

2. When does CA remove a node?

   Model answer: When its requested utilisation is below the threshold for the unneeded time and all its pods can be rescheduled elsewhere without violating PDBs or other rules.

3. Karpenter vs Cluster Autoscaler?

   Model answer: Karpenter provisions instances directly per pod requirements without node groups, picks cheapest fits including Spot, and consolidates; CA resizes predefined ASGs.

4. HPA scaled to 30 but pods are Pending. What is wrong?

   Model answer: Node capacity; check FailedScheduling events, node autoscaler logs, max sizes or NodePool limits, and quotas in the AWS account.

5. How do you keep node scale-down from hurting availability?

   Model answer: Run several replicas, set PDBs, spread across zones, and use disruption budgets in Karpenter.

Samajla ka?