Chapter 59: Kubernetes Pods in Depth: Architecture, YAML Anatomy and Pod Types

59.7 Static pods: kubelet manifests, mirror pods and the control plane

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Here is a puzzle. The API server is itself a pod. But pods are created through the API server. So who starts the API server? The answer is the kubelet. On every node, the kubelet watches one folder, usually /etc/kubernetes/manifests. Any pod YAML placed there is started directly by the kubelet, with no API server, no scheduler and no controller. These are static pods. This is how kubeadm, kind and many on-prem clusters run etcd, kube-apiserver, kube-controller-manager and kube-scheduler. If the API server crashes, the kubelet restarts it from the file. In this lesson we go inside a kind node, read the control plane manifests, create our own static pod, see its mirror pod in kubectl, and learn why kubectl delete does not remove it. Then we break things: a bad YAML in the folder, and an edit that takes down the API server.

What: every term in plain English

• Static pod — A pod started by the kubelet from a file on the node, not through the API server.
• staticPodPath — The kubelet setting that names the folder to watch. kubeadm uses /etc/kubernetes/manifests.
• Mirror pod — A read-only copy the kubelet creates in the API server so kubectl can show the static pod.
• Node name suffix — Mirror pods are named podname-nodename, like kube-apiserver-ch59-control-plane.
• kubeadm — The tool that installs clusters and writes the control plane static pod files. kind uses it inside.
• crictl — A command-line tool on the node to talk to containerd directly, without the API server.
• kubelet config — /var/lib/kubelet/config.yaml holds staticPodPath and other kubelet settings.
• Managed control plane — On EKS, GKE and AKS the cloud runs the control plane. You cannot see its static pods.

How: step by step

1. Open a shell on the node. For kind: docker exec -it ch59-control-plane bash.
2. Find the folder. grep staticPodPath /var/lib/kubelet/config.yaml.
3. Read the files. ls /etc/kubernetes/manifests shows etcd, apiserver, controller-manager and scheduler.
4. Add your own YAML. Copy a pod file into the folder. Never edit files with an editor that leaves swap files there.
5. Check from kubectl. kubectl get pods -A shows the mirror pod with the node name suffix.
6. Remove by deleting the file. kubectl delete only removes the mirror; the kubelet brings it back.
7. Use crictl when the API is down. crictl ps and crictl logs work even without kube-apiserver.

Example 1: Read the control plane static pods in kind

Goal. Read the control plane static pods in kind.

commands.sh

docker exec -it ch59-control-plane bash
grep staticPodPath /var/lib/kubelet/config.yaml
ls -l /etc/kubernetes/manifests
grep -E 'image:|- --etcd-servers|- --secure-port' /etc/kubernetes/manifests/kube-apiserver.yaml
exit
kubectl get pods -n kube-system -o wide | grep control-plane

What each line does

• docker exec ch59-control-plane — A kind node is a Docker container, so this is like SSH to a real node.
• staticPodPath — Shows the folder the kubelet watches.
• ls manifests — Four files: the whole control plane.
• --etcd-servers — The API server's link to etcd, which is also a static pod.
• grep control-plane — Mirror pods carry the node name at the end.

Green output

staticPodPath: /etc/kubernetes/manifests
-rw------- 1 root root 2406 etcd.yaml
-rw------- 1 root root 3896 kube-apiserver.yaml
-rw------- 1 root root 3428 kube-controller-manager.yaml
-rw------- 1 root root 1463 kube-scheduler.yaml
    image: registry.k8s.io/kube-apiserver:v1.31.2
    - --etcd-servers=https://127.0.0.1:2379
    - --secure-port=6443
etcd-ch59-control-plane                      1/1   Running   0   3h   172.18.0.3   ch59-control-plane
kube-apiserver-ch59-control-plane            1/1   Running   0   3h   172.18.0.3   ch59-control-plane
kube-controller-manager-ch59-control-plane   1/1   Running   0   3h   172.18.0.3   ch59-control-plane
kube-scheduler-ch59-control-plane            1/1   Running   0   3h   172.18.0.3   ch59-control-plane

Expected result. The heart of Kubernetes is four YAML files read by the kubelet. Files are rw------- (600): only root can read them, which is correct.

Example 2: Create your own static pod on a worker

Goal. Create your own static pod on a worker.

node-health.yaml

apiVersion: v1
kind: Pod
metadata:
  name: node-health
  namespace: default
  labels:
    app: node-health
spec:
  hostNetwork: true
  containers:
    - name: web
      image: nginx:1.27
      ports:
        - containerPort: 80
      resources:
        requests: {cpu: 10m, memory: 16Mi}
        limits: {memory: 32Mi}

copy.sh

docker cp node-health.yaml ch59-worker:/etc/kubernetes/manifests/node-health.yaml
kubectl get pods -o wide | grep node-health
kubectl delete pod node-health-ch59-worker
kubectl get pods | grep node-health
docker exec ch59-worker rm /etc/kubernetes/manifests/node-health.yaml
kubectl get pods | grep node-health || echo "gone"

What each line does

• hostNetwork: true — The pod uses the node's network, common for node agents and the control plane.
• docker cp into manifests — Placing the file is the only step. There is no kubectl apply.
• kubectl delete pod — Deletes only the mirror. The kubelet recreates it in seconds.
• rm the file — The real way to remove a static pod.

Green output

$ kubectl get pods -o wide | grep node-health
node-health-ch59-worker   1/1   Running   0   6s   172.18.0.4   ch59-worker
$ kubectl delete pod node-health-ch59-worker
pod "node-health-ch59-worker" deleted
$ kubectl get pods | grep node-health
node-health-ch59-worker   1/1   Running   0   2s
$ docker exec ch59-worker rm /etc/kubernetes/manifests/node-health.yaml
$ kubectl get pods | grep node-health || echo "gone"
gone

Expected result. The static pod came back after kubectl delete. It only left when the file was removed.

Example 3: Self-healing control plane: kill the scheduler and watch it return

Goal. Self-healing control plane: kill the scheduler and watch it return.

heal.sh

docker exec ch59-control-plane crictl ps --name kube-scheduler
docker exec ch59-control-plane sh -c 'crictl stop $(crictl ps -q --name kube-scheduler)'
docker exec ch59-control-plane crictl ps -a --name kube-scheduler
kubectl get pod -n kube-system kube-scheduler-ch59-control-plane

What each line does

• crictl ps --name — Lists the container through containerd, not through the API.
• crictl stop — Simulates a crash of the scheduler process.
• crictl ps -a — Shows the old exited container and the new running one.
• RESTARTS 1 — The kubelet restarted it because the file is still in manifests.

Green output

CONTAINER      IMAGE          STATE     NAME             ATTEMPT
5f1c2e9b0a11   9aa1fad94157   Running   kube-scheduler   0
CONTAINER      IMAGE          STATE     NAME             ATTEMPT
a8d03be62c47   9aa1fad94157   Running   kube-scheduler   1
5f1c2e9b0a11   9aa1fad94157   Exited    kube-scheduler   0
NAME                                READY   STATUS    RESTARTS     AGE
kube-scheduler-ch59-control-plane   1/1     Running   1 (8s ago)   3h

Expected result. The kubelet keeps the control plane alive by itself. That is self-healing without any controller.

Every command, step by step (kind lab with kubectl)

1. Open the node

   docker ps --format '{{.Names}}'
   

   What it does: kind nodes are containers.

   ch59-control-plane
   ch59-worker
   ch59-worker2
   

2. Find staticPodPath

   docker exec ch59-worker grep staticPodPath /var/lib/kubelet/config.yaml
   

   What it does: Same on every node.

   staticPodPath: /etc/kubernetes/manifests
   

3. Check the worker folder

   docker exec ch59-worker ls /etc/kubernetes/manifests
   

   What it does: Empty on workers: they run no control plane.

   (no output)
   

4. Place a static pod

   docker cp node-health.yaml ch59-worker:/etc/kubernetes/manifests/
   

   What it does: The kubelet checks the folder every few seconds.

   Successfully copied 2.05kB
   

5. See the mirror pod

   kubectl get pod node-health-ch59-worker -o jsonpath='{.metadata.annotations.kubernetes\.io/config\.mirror}{"\n"}{.metadata.ownerReferences[0].kind}{"\n"}'
   

   What it does: A mirror pod has a config.mirror annotation and is owned by the Node.

   3e1a9c0f5b2d...
   Node
   

6. Try to edit the mirror

   kubectl label pod node-health-ch59-worker team=sre
   

   What it does: Changes on the mirror are not saved to the file.

   pod/node-health-ch59-worker labeled
   

7. Edit the real file

   docker exec ch59-worker sed -i 's/nginx:1.27/nginx:1.27-alpine/' /etc/kubernetes/manifests/node-health.yaml
   kubectl get pod node-health-ch59-worker -o jsonpath='{.spec.containers[0].image}{"\n"}'
   

   What it does: The kubelet recreates the pod with the new image.

   nginx:1.27-alpine
   

8. Logs without the API

   docker exec ch59-worker crictl logs $(docker exec ch59-worker crictl ps -q --name web) 2>&1 | tail -2
   

   What it does: Useful when the API server is down.

   2026/10/25 06:40:11 [notice] 1#1: start worker processes
   

9. Remove it

   docker exec ch59-worker rm /etc/kubernetes/manifests/node-health.yaml
   

   What it does: Gone within seconds.

   (no output)
   

When it is red 1

The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?

Cause. Someone edited kube-apiserver.yaml with a mistake, so the API server will not start.

Fix, step by step

1. docker exec or SSH to the control plane node.
2. crictl ps -a --name kube-apiserver and crictl logs the latest one.
3. Fix the YAML or restore the backup, for example cp /root/kube-apiserver.yaml.bak back.
4. Always back up the file outside the manifests folder before editing.

When it is red 2

kubelet: Could not process manifest file "/etc/kubernetes/manifests/node-health.yaml": invalid pod: spec.containers: Required value

Cause. Bad YAML in the folder. kubectl shows nothing because no mirror is made.

Fix, step by step

1. journalctl -u kubelet | grep manifest on the node.
2. Fix indentation, then save.

When it is red 3

two kube-apiserver pods running, one crashing

Cause. A backup file like kube-apiserver.yaml.bak was left inside manifests, so the kubelet runs it too.

Fix, step by step

1. Keep backups in another folder, like /root/backup.
2. Remove the extra file.

When it is red 4

Error from server (Forbidden): pods is forbidden: a mirror pod may not reference service accounts

Cause. Static pods cannot use ServiceAccounts, ConfigMaps or Secrets from the API.

Fix, step by step

1. Use hostPath files instead.
2. Or run the workload as a DaemonSet (lesson 60.5).

Practice tasks

docker exec ch59-control-plane grep staticPodPath /var/lib/kubelet/config.yaml

docker exec ch59-control-plane ls /etc/kubernetes/manifests

kubectl get pods -A | grep control-plane

docker cp node-health.yaml ch59-worker2:/etc/kubernetes/manifests/

kubectl delete pod node-health-ch59-worker2

sed -i 's/1.27/1.27-alpine/'

docker exec ch59-control-plane crictl ps

kubectl get pod etcd-ch59-control-plane -n kube-system -o jsonpath='{.metadata.ownerReferences[0].kind}'

crictl stop $(crictl ps -q --name kube-scheduler)

cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/

Recap

• Static pods are started by the kubelet from staticPodPath, without the API server.
• kubeadm and kind run the whole control plane as static pods.
• kubectl shows mirror pods named podname-nodename; deleting them does nothing lasting.
• Create, change and remove static pods by changing files.
• Keep backups outside the manifests folder.
• Use crictl and journalctl when the API server is down.

FAQ

Q. Can I see static pods on EKS?

No. AWS runs the control plane for you. On worker nodes you can still add static pods, but DaemonSets are better.

Q. Static pod or DaemonSet?

DaemonSet for almost everything. Static pods are for the control plane and bootstrapping.

Q. Does the scheduler place static pods?

No. They run on the node that has the file.

Q. Can a Deployment manage static pods?

No. Only the kubelet manages them.

Q. Where is staticPodPath set?

In the kubelet config file, usually /var/lib/kubelet/config.yaml, or the --pod-manifest-path flag.

Q. Why are the manifest files 600?

They contain paths to keys and certificates, so only root should read them.

Interview questions

1. What is a static pod?

   Model answer: A pod managed directly by the kubelet on one node from a file in staticPodPath, with a mirror pod in the API server for visibility.

2. How does the control plane start if the API server is a pod?

   Model answer: The kubelet reads static pod manifests for etcd and kube-apiserver and starts them without needing the API.

3. How do you delete a static pod?

   Model answer: Remove its manifest file from the node; kubectl delete only removes the mirror, which comes back.

4. The API server is down after an edit. How do you recover?

   Model answer: SSH to the control plane, check crictl logs, fix or restore the manifest from a backup kept outside the folder.

5. Static pod versus DaemonSet?

   Model answer: Static pods are per-node files with no controller; DaemonSets are API objects that put one pod on every matching node and support rolling updates.

Samajla ka?