Ravindra BagaleCourses & study guides Track your progress

Guides

How to Fix ImagePullBackOff and ErrImagePull in Kubernetes

ImagePullBackOff means the node could not download your container image, so the container never started. Run kubectl describe pod <name> and read the Failed event at the bottom. "not found" means a wrong image name or tag, "unauthorized" or "no basic auth credentials" means a missing registry login, and "toomanyrequests" means a Docker Hub rate limit.

Quick answer

kubectl describe pod web-5f7c9-abcde | tail -20     # read the Failed event
docker pull nginx:1.27.0                            # test the same image yourself
kubectl create secret docker-registry regcred \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=ravi --docker-password='<token>'    # private registry login

A live example first

We make a small typing mistake, nginx:1.277 instead of nginx:1.27:

kubectl create deployment web --image=nginx:1.277
kubectl get pods
NAME                   READY   STATUS             RESTARTS   AGE
web-7b6c9d8f6c-4tq9p   0/1     ImagePullBackOff   0          48s

Note RESTARTS is 0. The container never ran, so kubectl logs shows nothing useful. The answer is in the events:

kubectl describe pod web-7b6c9d8f6c-4tq9p
Events:
  Normal   Pulling  45s  kubelet  Pulling image "nginx:1.277"
  Warning  Failed   43s  kubelet  Failed to pull image "nginx:1.277": rpc error:
           ... docker.io/library/nginx:1.277: not found
  Warning  Failed   43s  kubelet  Error: ErrImagePull
  Normal   BackOff  20s  kubelet  Back-off pulling image "nginx:1.277"
  Warning  Failed   20s  kubelet  Error: ImagePullBackOff

"not found" is clear. Fix the tag:

kubectl set image deployment/web nginx=nginx:1.27
kubectl get pods -w

Within seconds the pod is 1/1 Running. Friends, most ImagePullBackOff cases in class are exactly this: a typo.

Why does the image pull fail?

ImagePullBackOff: why the pull fails The kubelet asks the registry for the image. A wrong tag, missing login or rate limit stops the pull. Fix it and the pod runs. ImagePullBackOff: why the pull fails Pod specimage: tagkubeletpull requestRegistry401/404/429Fixtag / secret / IAMRunningimage pulled you Read the Events line: not found = name/tag, unauthorized = login, toomanyrequests = rate limit

The Events line tells you which one it is: not found, unauthorized or toomanyrequests.

The kubelet on each node pulls the image through the container runtime (containerd). The registry answers with an HTTP status: 404 for not found, 401 for login problems, 429 for too many requests. Kubernetes shows this as ErrImagePull on the first failure and ImagePullBackOff while it waits to retry (the same 10s to 5 min back-off used for crashes).

How do I fix ImagePullBackOff step by step?

  1. Run kubectl describe pod <name>.
  2. Read the last Failed event line.
  3. If it says not found or manifest unknown, check the name and tag.
  4. If it says unauthorized, denied or no basic auth credentials, fix the login.
  5. If it says toomanyrequests, fix the rate limit.
  6. If it says i/o timeout or dial tcp, fix node internet access (NAT gateway or VPC endpoints).
  7. Apply the fix and run kubectl get pods -w.

Fixes by registry

Docker Hub (public images)

Check that the tag really exists on hub.docker.com. Official images live under library/, so nginx means docker.io/library/nginx.

Since November 2020, Docker Hub limits anonymous pulls per IP address. On a cloud cluster, all nodes behind one NAT gateway share that single IP, so a busy cluster hits the limit fast:

toomanyrequests: You have reached your pull rate limit.

Fix it with an authenticated pull secret, or copy the image into your own registry once.

Docker Hub (private repo)

kubectl create secret docker-registry regcred \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=ravi \
  --docker-password='<access-token>'
spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: app
      image: ravi/orders-api:2.4.1

Use an access token, not your account password, and keep the secret in the same namespace as the pod.

Amazon ECR on EKS

The image name must be the full ECR URI:

123456789012.dkr.ecr.ap-south-1.amazonaws.com/orders-api:2.4.1
  1. Open IAM and find the EKS node group role.
  2. Attach the AWS managed policy AmazonEC2ContainerRegistryReadOnly.
  3. Check that the region in the URI is the region where you pushed.
  4. Check that private subnets have a NAT gateway or ECR VPC endpoints (ecr.api, ecr.dkr and the S3 gateway).

With these in place, no imagePullSecret is needed.

Google Artifact Registry (GKE) and Azure ACR (AKS)

  • GKE: give the node service account the Artifact Registry Reader role.
  • AKS: run az aks update -n mycluster -g myrg --attach-acr myregistry.

The idea is the same on every cloud: the node identity needs read access to the registry.

Image architecture: the Mac M1 trap

If you build on an Apple Silicon Mac, the image is arm64. On an amd64 node, the pull works but the container fails with exec format error. Build for the node:

docker buildx build --platform linux/amd64 -t orders-api:2.4.1 --push .

Real examples

  • Docker Hub rate limits (2020): when Docker announced the anonymous pull limits, many CI pipelines and clusters started failing with toomanyrequests. AWS answered by launching the ECR Public Gallery, which mirrors popular images such as nginx and python. Today many teams pull base images from public.ecr.aws for this reason.
  • The latest tag: Kubernetes documentation warns against :latest in production because different nodes may pull different images. A fixed tag like 2.4.1, or a digest, makes pulls repeatable and rollbacks easy.

Mistakes students make (and how to avoid them)

  1. Pushing to one region and pulling from another. An ECR repository in us-east-1 is not visible from an image URI that says ap-south-1. Copy the URI straight from the ECR console.
  2. Creating the pull secret in the wrong namespace. Secrets are per namespace. A secret in default does nothing for a pod in shop.
  3. Using your Docker Hub password. Create an access token with read-only scope instead, so a leaked secret cannot push or delete images.
  4. Forgetting the tag after a rebuild. If CI built 2.4.2 but the YAML still says 2.4.1-rc, the old or missing tag is pulled. Let the pipeline write the tag into the manifest.
  5. Private subnets with no way out. EKS nodes in private subnets need either a NAT gateway or the ECR VPC endpoints. Without them, the event says i/o timeout, not "unauthorized".

Practice lab (15 minutes)

  1. Deploy nginx:1.277 and read the not found event.
  2. Fix it with kubectl set image.
  3. Push a small image to a private ECR repository.
  4. Remove the ECR read policy from a test node role and watch the unauthorized event.
  5. Put the policy back and confirm the pod starts.

Summary table

Event message Cause Fix
not found / manifest unknown Wrong name or tag Correct the image and tag
unauthorized / no basic auth credentials No registry login imagePullSecret or node IAM role
toomanyrequests Docker Hub rate limit Authenticate, or mirror to ECR
i/o timeout Node has no route to the registry NAT gateway or VPC endpoints
exec format error (after pull) Wrong CPU architecture buildx --​platform linux/​amd64

Quick quiz: check what you learned

  1. Where do you read the real reason for an image pull failure?

    1. Events in kubectl describe pod
    2. The output of kubectl top nodes
    3. The container logs --previous
    4. The Service endpoints list
    Show answer

    A. The container never started, so there are no logs; the Failed event holds the reason.

  2. A pull fails with "not found". What do you check first?

    1. The image name and its tag
    2. The pod memory limit
    3. The node disk encryption
    4. The liveness probe path
    Show answer

    A. Not found means the registry has no image with that exact name and tag.

  3. How does an EKS node pull from private ECR without a secret?

    1. Its IAM role has ECR read
    2. It uses Docker Hub login
    3. ECR images are all public
    4. kubectl sends your password
    Show answer

    A. With AmazonEC2ContainerRegistryReadOnly on the node role, the kubelet gets ECR tokens itself.

  4. Docker Hub returns "toomanyrequests". What fixes it best?

    1. Log in or mirror to ECR
    2. Increase the CPU request
    3. Delete the namespace
    4. Change the restart policy
    Show answer

    A. The limit is per IP for anonymous pulls; authenticated pulls or your own registry avoid it.

Learn it properly

This guide is the short path. The free DevOps course covers Kubernetes step by step, with labs:

Frequently asked questions

What is the difference between ErrImagePull and ImagePullBackOff?

ErrImagePull is the first failed pull. ImagePullBackOff means Kubernetes failed again and is now waiting longer between tries. The fix is the same.

How do I see why the image pull failed?

Run kubectl describe pod POD and read the Events. The Failed line has the real message, such as "not found" or "unauthorized".

How do I pull from a private Docker Hub repo?

Create a secret with kubectl create secret docker-registry and add it under imagePullSecrets in the pod spec.

Does EKS need an imagePullSecret for ECR?

No, when the node IAM role has the AmazonEC2ContainerRegistryReadOnly policy. The kubelet logs in to ECR automatically.

What is "toomanyrequests" from Docker Hub?

Docker Hub limits anonymous pulls per IP. Nodes behind one NAT share that limit. Log in, or copy the image to ECR.

Should I use the latest tag?

No. Use a fixed version tag or digest so every node pulls the same image and pulls are predictable.