Chapter 59: Kubernetes Pods in Depth: Architecture, YAML Anatomy and Pod Types

59.1 Cluster and pod architecture: what really happens on kubectl apply

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose we are at Swiggy on a Friday night. An engineer types kubectl apply -f order-api.yaml. Ten seconds later, three pods are running on three machines. Who picked the machines? Who pulled the image? Who gave each pod an IP address? If you cannot answer, you cannot debug a pod stuck in Pending at 9 pm. In chapter 15 and chapter 18 we used kubectl. In this lesson we open the box. We follow one pod from your terminal to a running container, step by step, on a real kind cluster. You will see every component with your own eyes: the API server, etcd, the scheduler, the controller manager, the kubelet, the container runtime and the tiny pause container inside each pod.

What: every term in plain English

• Cluster — A group of machines that Kubernetes treats as one big computer.
• Control plane — The brain of the cluster. It stores the wanted state and makes decisions. It does not run your app.
• Worker node — A machine that runs your pods. In kind, each node is a Docker container.
• kube-apiserver — The front door. Every request (kubectl, kubelet, controllers) goes through it. It checks who you are and saves objects.
• etcd — A small, safe key-value database. It is the memory of the cluster. All objects live here.
• kube-scheduler — Picks a node for each new pod. It looks at free CPU, memory, labels and rules.
• kube-controller-manager — Runs many control loops. Each loop compares wanted state with real state and fixes the difference.
• kubelet — An agent on every node. It asks the API server which pods belong to its node and starts them.
• Container runtime — The program that actually runs containers. Today it is usually containerd.
• kube-proxy — An agent on every node that sets network rules so Service IPs reach pods.
• Pause container — A tiny container that holds the pod's network namespace. App containers join it, so they share one IP.
• Watch — A long-lived connection. Components watch the API server and react the moment something changes.

How: step by step

1. You run kubectl apply. kubectl reads the YAML and sends it to kube-apiserver over HTTPS.
2. The API server checks and saves. It checks your identity, your permissions (RBAC) and the YAML. Then it writes the object to etcd.
3. Controllers create pods. The Deployment controller creates a ReplicaSet. The ReplicaSet controller creates Pod objects with no node yet.
4. The scheduler picks a node. It sees pods with an empty nodeName, scores the nodes, and writes the chosen node into the pod.
5. The kubelet starts the pod. The kubelet on that node sees the new pod, asks containerd to start the pause container, then the app containers.
6. Network and status. The CNI plugin gives the pod an IP. The kubelet reports Running back to the API server.
7. Self-healing loops. If a pod dies, the ReplicaSet controller sees 2 instead of 3 and creates a new pod. The loop never stops.

Example 1: Create a kind cluster and see every control plane part

Goal. Create a kind cluster and see every control plane part.

kind-3node.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: ch59
nodes:
  - role: control-plane
  - role: worker
  - role: worker

look.sh

kind create cluster --config kind-3node.yaml
kubectl cluster-info --context kind-ch59
kubectl get nodes -o wide
kubectl get pods -n kube-system -o wide
docker ps --format 'table {{.Names}}\t{{.Image}}'

What each line does

• kind: Cluster — This file is for kind, not for Kubernetes. It describes the local cluster shape.
• apiVersion: kind.x-k8s.io/v1alpha4 — The version of the kind config format.
• name: ch59 — The cluster name. The kubectl context becomes kind-ch59.
• role: control-plane — One node runs the brain: API server, etcd, scheduler, controller manager.
• role: worker (two times) — Two nodes for your pods. Now the scheduler has a real choice.
• kubectl get pods -n kube-system — On kind and kubeadm, the control plane parts run as pods in the kube-system namespace.
• docker ps — Each kind node is just a Docker container. That is why kind starts in about a minute.

Green output

$ kind create cluster --config kind-3node.yaml
Creating cluster "ch59" ...
 ✓ Ensuring node image (kindest/node:v1.31.0)
 ✓ Preparing nodes
 ✓ Writing configuration
 ✓ Starting control-plane
 ✓ Installing CNI
 ✓ Installing StorageClass
 ✓ Joining worker nodes
Set kubectl context to "kind-ch59"
$ kubectl get nodes
NAME                 STATUS   ROLES           AGE   VERSION
ch59-control-plane   Ready    control-plane   62s   v1.31.0
ch59-worker          Ready    worker          41s   v1.31.0
ch59-worker2         Ready    worker          41s   v1.31.0
$ kubectl get pods -n kube-system
NAME                                         READY   STATUS    RESTARTS   AGE
coredns-7db6d8ff4d-6x2mn                     1/1     Running   0          55s
coredns-7db6d8ff4d-v8k4p                     1/1     Running   0          55s
etcd-ch59-control-plane                      1/1     Running   0          70s
kindnet-4dq8r                                1/1     Running   0          41s
kube-apiserver-ch59-control-plane            1/1     Running   0          70s
kube-controller-manager-ch59-control-plane   1/1     Running   0          70s
kube-proxy-9kq2l                             1/1     Running   0          41s
kube-scheduler-ch59-control-plane            1/1     Running   0          70s
$ docker ps --format 'table {{.Names}}\t{{.Image}}'
NAMES                IMAGE
ch59-worker          kindest/node:v1.31.0
ch59-worker2         kindest/node:v1.31.0
ch59-control-plane   kindest/node:v1.31.0

Expected result. You can point at each part of the brain. kindnet is the CNI plugin; kube-proxy runs once per node.

Example 2: Follow one Deployment from apply to running pods using events

Goal. Follow one Deployment from apply to running pods using events.

order-api.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
  labels:
    app: order-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-api
  template:
    metadata:
      labels:
        app: order-api
    spec:
      containers:
        - name: web
          image: nginx:1.27
          ports:
            - containerPort: 80

follow.sh

kubectl apply -f order-api.yaml
kubectl get events --sort-by=.metadata.creationTimestamp | tail -15
kubectl get deploy,rs,pods -l app=order-api -o wide

What each line does

• apiVersion: apps/v1 — Deployments live in the apps API group, version v1.
• kind: Deployment — We ask for a Deployment. It will create a ReplicaSet, which creates pods.
• replicas: 3 — Wanted state: three pods.
• selector.matchLabels — The Deployment owns pods that carry app: order-api.
• template — The pod blueprint. Every pod is a copy of this.
• image: nginx:1.27 — containerd pulls this image on the chosen node.
• get events --sort-by — Events are the cluster diary. Each line shows which component did what.

Green output

$ kubectl get events --sort-by=.metadata.creationTimestamp | tail -9
LAST SEEN   TYPE     REASON              OBJECT                            MESSAGE
12s         Normal   ScalingReplicaSet   deployment/order-api              Scaled up replica set order-api-5d8c9b7f6 to 3
12s         Normal   SuccessfulCreate    replicaset/order-api-5d8c9b7f6    Created pod: order-api-5d8c9b7f6-k2x7p
12s         Normal   Scheduled           pod/order-api-5d8c9b7f6-k2x7p     Successfully assigned default/order-api-5d8c9b7f6-k2x7p to ch59-worker2
11s         Normal   Pulling             pod/order-api-5d8c9b7f6-k2x7p     Pulling image "nginx:1.27"
6s          Normal   Pulled              pod/order-api-5d8c9b7f6-k2x7p     Successfully pulled image "nginx:1.27" in 4.9s
6s          Normal   Created             pod/order-api-5d8c9b7f6-k2x7p     Created container web
6s          Normal   Started             pod/order-api-5d8c9b7f6-k2x7p     Started container web
$ kubectl get pods -l app=order-api -o wide
NAME                        READY   STATUS    IP           NODE
order-api-5d8c9b7f6-k2x7p   1/1     Running   10.244.2.3   ch59-worker2
order-api-5d8c9b7f6-m9qzt   1/1     Running   10.244.1.4   ch59-worker
order-api-5d8c9b7f6-r4w8d   1/1     Running   10.244.2.4   ch59-worker2

Expected result. The events match the 7 steps: deployment controller, replicaset controller, scheduler, then kubelet with containerd.

Example 3: Look inside a node and find the pause container

Goal. Look inside a node and find the pause container.

inside-node.sh

POD=$(kubectl get pod -l app=order-api -o jsonpath='{.items[0].metadata.name}')
NODE=$(kubectl get pod $POD -o jsonpath='{.spec.nodeName}')
echo "pod $POD is on $NODE"
docker exec $NODE crictl pods --name $POD
docker exec $NODE crictl ps --pod $(docker exec $NODE crictl pods --name $POD -q)
docker exec $NODE crictl images | grep pause
docker exec $NODE ls /etc/kubernetes/manifests
docker exec ch59-control-plane ls /etc/kubernetes/manifests

What each line does

• jsonpath — Picks one field from the JSON answer. Here, the first pod name and its node.
• docker exec $NODE — Runs a command inside the kind node container, like SSH into a real node.
• crictl pods — Talks to containerd directly. It shows the pod sandbox (the pause container).
• crictl ps --pod — Lists the app containers inside that sandbox.
• crictl images | grep pause — The pause image is tiny, under 1 MB. It only holds the namespaces.
• /etc/kubernetes/manifests — On the control plane this folder holds the static pod files for the API server, etcd, scheduler and controller manager. Lesson 59.7 covers static pods.

Green output

pod order-api-5d8c9b7f6-k2x7p is on ch59-worker2
POD ID          CREATED          STATE   NAME                        NAMESPACE
3f1c2a9b8d7e6   2 minutes ago    Ready   order-api-5d8c9b7f6-k2x7p   default
CONTAINER       IMAGE           STATE     NAME   POD ID
a1b2c3d4e5f60   39286ab8a5e14   Running   web    3f1c2a9b8d7e6
registry.k8s.io/pause    3.10    873ed75102791    320kB
$ docker exec ch59-control-plane ls /etc/kubernetes/manifests
etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml

Expected result. You saw the real layers: pod sandbox (pause), app container, and the control plane running as static pods.

Every command, step by step (kind lab with kubectl)

1. Check the cluster is healthy

   kubectl get --raw='/readyz?verbose' | tail -4
   

   What it does: Asks the API server to run its own health checks.

   [+]poststarthook/apiservice-openapiv3-controller ok
   [+]shutdown ok
   readyz check passed
   

2. Read objects straight from etcd keys

   docker exec ch59-control-plane sh -c 'ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get /registry/deployments/default --prefix --keys-only'
   

   What it does: Proves that the Deployment is stored in etcd under /registry. Only for learning; never edit etcd by hand.

   /registry/deployments/default/order-api
   

3. Stop the scheduler and watch Pending

   docker exec ch59-control-plane mv /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/
   kubectl scale deploy order-api --replicas=5
   kubectl get pods -l app=order-api
   

   What it does: Without a scheduler, new pods have no node, so they stay Pending.

   order-api-5d8c9b7f6-x8v2n   0/1     Pending   0          8s
   order-api-5d8c9b7f6-z3p9q   0/1     Pending   0          8s
   

4. Bring the scheduler back

   docker exec ch59-control-plane mv /tmp/kube-scheduler.yaml /etc/kubernetes/manifests/
   sleep 20; kubectl get pods -l app=order-api
   

   What it does: The kubelet starts the scheduler again from the file. Pending pods get nodes.

   order-api-5d8c9b7f6-x8v2n   1/1     Running   0          45s
   order-api-5d8c9b7f6-z3p9q   1/1     Running   0          45s
   

5. Kill a pod and see self-healing

   kubectl delete pod -l app=order-api --wait=false
   kubectl get pods -l app=order-api
   

   What it does: The ReplicaSet controller sees fewer pods than wanted and makes new ones.

   order-api-5d8c9b7f6-b7k1c   0/1     ContainerCreating   0   1s
   order-api-5d8c9b7f6-h4n6t   1/1     Running             0   2s
   

6. See which pods each kubelet runs

   kubectl get pods -A -o wide --field-selector spec.nodeName=ch59-worker
   

   What it does: Every pod with nodeName ch59-worker is the kubelet's job on that node.

   default       order-api-5d8c9b7f6-m9qzt   1/1   Running
   kube-system   kindnet-7wqz5               1/1   Running
   kube-system   kube-proxy-2xg8h            1/1   Running
   

7. Clean up

   kubectl delete -f order-api.yaml
   

   What it does: Removes the Deployment, its ReplicaSet and pods. Keep the cluster for the next lessons.

   deployment.apps "order-api" deleted
   

When it is red 1

The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?

Cause. The API server is down or kubectl points to the wrong cluster.

Fix, step by step

1. Run kubectl config current-context and check it says kind-ch59.
2. Run docker ps and check the ch59-control-plane container is Up.
3. If it stopped, run docker start ch59-control-plane and wait 30 seconds.
4. Run kubectl get nodes again.

When it is red 2

NAME                        READY   STATUS    RESTARTS   AGE
order-api-5d8c9b7f6-x8v2n   0/1     Pending   0          3m

Cause. No node was chosen. Either the scheduler is down, or no node has enough CPU or memory.

Fix, step by step

1. Run kubectl describe pod order-api-5d8c9b7f6-x8v2n and read Events.
2. If there are no events at all, check the scheduler pod in kube-system.
3. If you see Insufficient cpu, lower the requests or add a node.
4. Run kubectl get pods again after the fix.

When it is red 3

Warning  FailedCreatePodSandBox  kubelet  Failed to create pod sandbox: plugin type="ptp" failed: failed to allocate IP

Cause. The CNI plugin could not give the pod an IP address.

Fix, step by step

1. Check the CNI pods: kubectl get pods -n kube-system -l app=kindnet.
2. Restart a broken one with kubectl delete pod on it; the DaemonSet recreates it.
3. For EKS, check free IPs in the subnet for the AWS VPC CNI.
4. Delete the stuck pod so it is recreated.

Practice tasks

kind create cluster --config kind-3node.yaml
kubectl get nodes

kubectl get pods -n kube-system | grep control-plane

kubectl get events --sort-by=.metadata.creationTimestamp

kubectl get pods -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName

docker exec ch59-worker crictl images | grep pause

docker exec ch59-control-plane mv /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/
kubectl scale deploy order-api --replicas=5

docker exec ch59-control-plane mv /tmp/kube-scheduler.yaml /etc/kubernetes/manifests/

kubectl delete pod -l app=order-api --wait=false
kubectl get pods -l app=order-api

etcdctl get /registry/pods/default --prefix --keys-only

docker stop ch59-worker2
kubectl get nodes -w

Recap

• The control plane decides; worker nodes run pods.
• Everything goes through the API server and is stored in etcd.
• Controllers create pods, the scheduler picks nodes, the kubelet starts containers.
• A pod is a pause container plus your app containers sharing one IP.
• Events show every step; read them first when debugging.
• Basics are in chapter 15 and chapter 18; this lesson shows the inside.

FAQ

Q. Is kind the same as a real cluster?

The components are the same real Kubernetes binaries. Only the nodes are Docker containers instead of VMs.

Q. On EKS, can I see etcd and the scheduler?

No. AWS runs the control plane for you. You only see worker nodes and add-ons.

Q. Can I use minikube instead of kind?

Yes. Run minikube start --nodes 3. Use minikube ssh in place of docker exec.

Q. Does the kubelet run as a pod?

No. It is a normal service (systemd) on each node, because it must start the pods.

Q. Why is the scheduler separate from the API server?

Each part does one job. You can even replace the scheduler with your own.

Q. What happens if etcd is lost?

The cluster forgets everything. That is why production backs up etcd and runs 3 or 5 copies.

Interview questions

1. Explain what happens when you run kubectl apply for a Deployment.

   Model answer: kubectl sends it to the API server, which authenticates, authorizes, validates and stores it in etcd. The Deployment controller creates a ReplicaSet, which creates pods. The scheduler assigns nodes. The kubelet on each node starts the containers through containerd, CNI gives IPs, and status goes back to the API server.

2. What are the control plane components?

   Model answer: kube-apiserver, etcd, kube-scheduler, kube-controller-manager and, in clouds, cloud-controller-manager.

3. What runs on every worker node?

   Model answer: kubelet, kube-proxy (or a CNI that replaces it), and a container runtime like containerd.

4. A pod is stuck in Pending. Where do you look?

   Model answer: kubectl describe pod events: scheduler messages such as insufficient CPU, taints, node selectors, or unbound PVCs.

5. What is the role of the pause container?

   Model answer: It creates and holds the Linux namespaces of the pod, so app containers can restart without losing the pod IP.

6. Why is etcd critical?

   Model answer: It is the only source of truth. Losing it loses the cluster state, so it needs backups and an odd number of members for quorum.

Samajla ka?