Ravindra BagaleCourses & study guides Track your progress

Guides

Kubernetes Pod Stuck in Pending: How to Find and Fix the Reason

A pod stuck in Pending has no node yet, so nothing is running. Run kubectl describe pod <name> and read the FailedScheduling event. It tells you exactly why every node was rejected, for example Insufficient cpu, untolerated taint, didn't match node selector or unbound PersistentVolumeClaims. Fix that one rule, or add a node.

Quick answer

kubectl describe pod report-job-8xk2 | grep -A5 Events   # read FailedScheduling
kubectl describe nodes | grep -A8 "Allocated resources"  # how full is each node?
kubectl get pvc                                          # all claims Bound?

A live example first

On a small 2-node EKS lab (t3.medium nodes, 2 vCPU each) we deploy an app that asks for 3 CPUs:

resources:
  requests:
    cpu: "3"
    memory: "512Mi"
kubectl apply -f report.yaml
kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
report-5d9c7b8f4-hk2lp    0/1     Pending   0          3m
kubectl describe pod report-5d9c7b8f4-hk2lp
Events:
  Warning  FailedScheduling  default-scheduler  0/2 nodes are available:
           2 Insufficient cpu. preemption: 0/2 nodes are available:
           2 No preemption victims found for incoming pod.

A t3.medium has 2 vCPU, and part of that is reserved for the system. No node can ever give 3 CPUs. We change the request to 500m and apply again, and the pod starts in seconds. Friends, the scheduler always tells you the reason. You only have to read it.

Why does a pod stay Pending?

Pod Pending: why no node fits The scheduler checks each node for free requests, taints, labels and volumes. If none fits, the pod waits in Pending. Pod Pending: why no node fits Pendingno node yetSchedulerchecks nodesReasoncpu/taint/pvcFixrequest/labelRunningnode chosen you FailedScheduling explains which rule ruled out every node

The scheduler explains itself in FailedScheduling: CPU, memory, taints, selectors or volumes.

The scheduler looks at every node and filters out the ones that cannot run the pod. A node is filtered out when:

  • the free requests (not real usage) are smaller than the pod's requests,
  • the node has a taint the pod does not tolerate,
  • the node labels do not match the pod's nodeSelector or affinity,
  • a required volume is not ready or is in another Availability Zone.

If every node is filtered out, the pod waits in Pending and the scheduler tries again later.

How do I fix a Pending pod step by step?

  1. Run kubectl describe pod <name>.
  2. Read the FailedScheduling message word by word.
  3. Find the reason in the table below.
  4. Fix the pod spec or the cluster.
  5. Run kubectl get pods -w until the pod leaves Pending.

Reason 1: Insufficient cpu or memory

Check how much is already promised on each node:

kubectl describe node ip-10-0-1-25.ap-south-1.compute.internal
Allocatable:
  cpu:     1930m
  memory:  3388Mi
Allocated resources:
  Resource  Requests      Limits
  cpu       1750m (90%)   2 (103%)
  memory    2100Mi (61%)  3Gi (92%)

Only 180m of CPU is free. The fixes are:

  1. Lower the requests to what the app really needs (measure with kubectl top pod).
  2. Remove old Deployments you no longer use.
  3. Add nodes, or turn on Cluster Autoscaler or Karpenter on EKS.

Reason 2: Taints and tolerations

0/3 nodes are available: 3 node(s) had untolerated taint {dedicated: gpu}.

The nodes are reserved. Either run on other nodes, or add a toleration if this pod really belongs there:

tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"

Reason 3: nodeSelector or affinity does not match

0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector.

Compare the pod spec with the real labels:

kubectl get nodes --show-labels | tr ',' '\n' | grep -i zone

A common mistake is disktype: SSD in the pod while the node label is disktype=ssd. Labels are case-sensitive.

Reason 4: The PersistentVolumeClaim is not Bound

0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims.
kubectl get pvc
kubectl describe pvc data-mysql-0

On EKS, check three things:

  1. The EBS CSI driver add-on is installed.
  2. A StorageClass exists (kubectl get storageclass) and is the default, or is named in the PVC.
  3. The EBS volume and the node are in the same Availability Zone. An EBS volume in ap-south-1a cannot attach to a node in ap-south-1b.

Reason 5: ResourceQuota or LimitRange

If a namespace has a quota, the pod may be refused before scheduling. Look at the ReplicaSet events:

kubectl describe rs -l app=report
kubectl describe quota -n reports

Same idea on every cloud

Cloud How to add capacity
AWS EKS Increase node group desired size, or Karpenter / Cluster Autoscaler
Google GKE Turn on node auto-provisioning or the cluster autoscaler
Azure AKS az aks nodepool scale or the cluster autoscaler

Real example: planning capacity

When Pokémon GO launched in 2016, Google Cloud's public blog post explained that traffic reached about 50 times the original estimate. The game ran on Google Kubernetes Engine, and the team had to add capacity and upgrade the cluster live. On a smaller scale, every Pending pod with Insufficient cpu is the same lesson: set honest requests, and have a plan for adding nodes.

Mistakes students make (and how to avoid them)

  1. Copying requests from a blog. A request of 2 CPUs "to be safe" on a 2-vCPU node means the pod can never fit. Start from measured use.
  2. Thinking idle CPU is free CPU. The scheduler adds up requests. Ten idle pods that each request 200m still block 2 full CPUs.
  3. Forgetting the DaemonSets. Logging, monitoring and the CNI run one pod per node and use part of every node before your app arrives.
  4. Using one AZ for storage and another for nodes. With EBS, use a StorageClass with volumeBindingMode: WaitForFirstConsumer, so the volume is created in the zone where the pod is scheduled.
  5. Running the autoscaler with max size equal to min size. Cluster Autoscaler cannot add a node if the node group maximum is already reached.

Practice lab (15 minutes)

  1. Deploy a pod that requests 3 CPUs on a small node and read the event.
  2. Taint a node with kubectl taint nodes NODE dedicated=gpu:NoSchedule and watch new pods avoid it.
  3. Remove the taint with the same command ending in -.

Summary table

FailedScheduling says Cause Fix
Insufficient cpu / memory Requests bigger than free capacity Lower requests or add nodes
untolerated taint Node is reserved Add toleration or use other nodes
didn't match ... selector Labels do not match Fix the label or the selector
unbound ... PersistentVolumeClaims Volume not ready StorageClass, CSI driver, same AZ
No events Quota or scheduler name describe rs, describe quota

Quick quiz: check what you learned

  1. What does the scheduler use to decide if a pod fits?

    1. The pod CPU and memory requests
    2. The live CPU usage on the node
    3. The size of the container image
    4. The number of Services defined
    Show answer

    A. Scheduling is based on requests, not on what the node is using right now.

  2. "untolerated taint" in FailedScheduling means what?

    1. The pod lacks a toleration
    2. The image tag is missing here
    3. The PVC is already bound
    4. The probe path is incorrect
    Show answer

    A. A tainted node accepts only pods that tolerate that taint.

  3. A pod waits because its PVC is not Bound. What do you check?

    1. The StorageClass and CSI driver
    2. The liveness probe timeout
    3. The Docker Hub rate limit
    4. The container exit code
    Show answer

    A. The volume must be provisioned and Bound before the pod can start.

  4. Which command shows how much of a node is already requested?

    1. kubectl describe node NAME
    2. kubectl logs NAME --previous
    3. kubectl get secrets -A
    4. kubectl rollout undo NAME
    Show answer

    A. describe node shows Allocatable and the Allocated resources table.

Learn it properly

This guide is the short path. The free DevOps course covers Kubernetes step by step, with labs:

Frequently asked questions

What does Pending mean in Kubernetes?

The pod is accepted but the scheduler has not found a node for it yet, or the node has not started its containers. No container is running.

What is "0/3 nodes are available: 3 Insufficient cpu"?

None of the 3 nodes has enough free CPU requests for this pod. Lower the request or add nodes.

Why is my pod Pending when the node has free CPU?

Scheduling uses requests, not real usage. If other pods requested the CPU, it is counted as used even when idle.

What does "untolerated taint" mean?

The node has a taint, such as a control-plane or GPU taint, and your pod has no matching toleration.

Can a PVC keep a pod Pending?

Yes. If the PersistentVolumeClaim is not Bound, the pod waits. Check the StorageClass and the EBS CSI driver on EKS.

Pending with no events at all?

Check that the scheduler is running and that schedulerName in the pod spec is correct. On managed clusters, check the namespace ResourceQuota.