Chapter 62: Kubernetes Services, Networking and Load Balancing: CNI, ClusterIP, NodePort, LoadBalancer, DNS and NetworkPolicy

62.1 Pod networking and CNI: how every pod gets an IP (Calico, Cilium, AWS VPC CNI)

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose Zepto runs its order API as 20 pods spread across 5 nodes. Any pod must be able to call any other pod directly by IP, without NAT, whichever node it is on. That is the Kubernetes network model, and Kubernetes itself does not implement it. It hands the job to a CNI (Container Network Interface) plugin. When kubelet starts a pod, the container runtime creates a network namespace and calls the CNI plugin. The plugin gives the pod an IP from the node's pod range, creates a virtual ethernet pair (one end eth0 in the pod, the other on the node), and sets up routes so other nodes can reach that IP. Different plugins do the cross-node part differently: Calico uses plain routing (BGP) or VXLAN, Cilium uses eBPF programs in the kernel, and the AWS VPC CNI gives pods real VPC IP addresses from the node's network interfaces. Without a CNI, nodes stay NotReady and pods stay Pending or ContainerCreating. The CNI choice also decides whether NetworkPolicy works at all; kind's default kindnet supports basic policy only in newer versions, so this chapter installs Calico. In this lesson we build the ch62 cluster with no CNI, watch it fail, install Calico, trace a pod's IP and veth, ping across nodes, and compare how EKS gives pods VPC IPs.

What: every term in plain English

• Kubernetes network model — Every pod gets its own IP and can reach every other pod without NAT.
• CNI — The plugin standard the runtime calls to set up and tear down pod networking.
• Pod CIDR — The IP range for pods, split into one block per node.
• Service CIDR — A separate range for Service virtual IPs (62.2).
• veth pair — A virtual cable: eth0 inside the pod, a calixxx or vethxxx interface on the node.
• Network namespace — The isolated network stack a pod's containers share.
• Calico — CNI using routing (BGP) or VXLAN, with strong NetworkPolicy support.
• Cilium — CNI built on eBPF with policy, observability (Hubble) and kube-proxy replacement.
• AWS VPC CNI — EKS default; pods get real VPC IPs from ENIs on the node.
• Overlay vs routed — Overlay wraps packets (VXLAN); routed sends pod IPs natively.

How: step by step

1. Create the ch62 cluster with disableDefaultCNI. Nodes stay NotReady.
2. Install Calico with the operator. Nodes become Ready.
3. Run two pods on different nodes. Read their IPs.
4. Ping and curl pod to pod. No NAT, direct IPs.
5. Trace the veth and routes on the node. See what the CNI built.
6. Compare with EKS and the VPC CNI. Pod IPs from the VPC subnet.

Example 1: Build a cluster with no CNI and see why it is needed

Goal. Build a cluster with no CNI and see why it is needed.

kind-ch62.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: ch62
networking:
  disableDefaultCNI: true
  podSubnet: 192.168.0.0/16
  serviceSubnet: 10.96.0.0/12
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 30080
        hostPort: 30080
  - role: worker
  - role: worker

What each line does

• disableDefaultCNI — kind will not install kindnet; we add Calico ourselves.
• podSubnet 192.168.0.0/16 — Calico's default pool, so no extra config.
• serviceSubnet — The range for ClusterIPs.
• extraPortMappings 30080 — Used in 62.3 to reach a NodePort from the laptop.
• Two workers — Needed to test cross-node traffic.

Green output

$ kind create cluster --config kind-ch62.yaml
Creating cluster "ch62" ...
$ kubectl get nodes
NAME                 STATUS     ROLES           AGE
ch62-control-plane   NotReady   control-plane   60s
ch62-worker          NotReady   <none>          40s
ch62-worker2         NotReady   <none>          40s
$ kubectl describe node ch62-worker | grep -m1 NetworkReady
  Ready  False  ...  KubeletNotReady  container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
$ kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS
coredns-7db6d8ff4d-5kq2x   0/1     Pending

Expected result. No CNI means no pod networking, so the nodes report NotReady and even CoreDNS cannot start.

Example 2: Install Calico and test pod-to-pod traffic across nodes

Goal. Install Calico and test pod-to-pod traffic across nodes.

calico-install.sh

kubectl create -f ./calico/tigera-operator.yaml
kubectl create -f ./calico/custom-resources.yaml
kubectl wait --for=condition=Available tigerastatus/calico --timeout=300s
kubectl get nodes

two-pods.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: net
---
apiVersion: v1
kind: Pod
metadata:
  name: a
  namespace: net
spec:
  nodeName: ch62-worker
  containers:
    - name: c
      image: nicolaka/netshoot:v0.13
      command: ["sleep", "3600"]
---
apiVersion: v1
kind: Pod
metadata:
  name: b
  namespace: net
spec:
  nodeName: ch62-worker2
  containers:
    - name: c
      image: nginx:1.27

What each line does

• tigera-operator.yaml / custom-resources.yaml — Downloaded from the Calico docs release page (v3.28) into ./calico.
• tigerastatus — The operator's health object; Available means Calico is up.
• nodeName — Forces a and b onto different workers.
• netshoot — A debug image with ping, curl, ip, dig and tcpdump.
• Direct IP — a reaches b's pod IP without any Service.

Green output

$ bash calico-install.sh
tigerastatus.operator.tigera.io/calico condition met
NAME                 STATUS   ROLES
ch62-control-plane   Ready    control-plane
ch62-worker          Ready    <none>
ch62-worker2         Ready    <none>
$ kubectl apply -f two-pods.yaml
$ kubectl get pods -n net -o wide
NAME   READY   STATUS    IP                NODE
a      1/1     Running   192.168.73.130    ch62-worker
b      1/1     Running   192.168.104.2     ch62-worker2
$ kubectl exec -n net a -- ping -c2 192.168.104.2
64 bytes from 192.168.104.2: icmp_seq=1 ttl=62 time=0.21 ms
$ kubectl exec -n net a -- curl -s 192.168.104.2 | grep title
<title>Welcome to nginx!</title>

Expected result. Each node got its own block from 192.168.0.0/16, and pod a reached pod b on another node by its real IP. That is the network model working.

Example 3: Trace what the CNI built on the node, and compare with EKS

Goal. Trace what the CNI built on the node, and compare with EKS.

trace.sh

kubectl exec -n net a -- ip -4 addr show eth0
kubectl exec -n net a -- ip route
docker exec ch62-worker ip route | grep 192.168
docker exec ch62-worker ip link | grep cali

What each line does

• eth0 in the pod — The pod end of the veth pair with the pod IP /32.
• Default route 169.254.1.1 — Calico's trick: a fake gateway answered by the node.
• Routes per node block — 192.168.104.0/26 via the other worker (tunnel or BGP).
• caliXXXX — The node end of each pod's veth.
• EKS — With the VPC CNI the pod IP is a secondary IP on an ENI, from the subnet itself.

Green output

$ bash trace.sh
3: eth0@if9: <BROADCAST,MULTICAST,UP> mtu 1450
    inet 192.168.73.130/32 scope global eth0
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
192.168.73.128/26 via ... blackhole
192.168.104.0/26 via 172.18.0.4 dev vxlan.calico onlink
9: cali7a3f1e2c4b0@if3: <BROADCAST,MULTICAST,UP>
$ kubectl get pods -n kube-system -l k8s-app=aws-node -o wide   # on EKS
aws-node-8x2kd   2/2   Running   10.0.12.41   ip-10-0-12-41.ap-south-1.compute.internal
$ kubectl get pod -n shop api-6c9d -o wide                       # on EKS
api-6c9d   1/1   Running   10.0.12.87   ip-10-0-12-41.ap-south-1.compute.internal

Expected result. On kind, Calico routes the other node's /26 through a VXLAN tunnel. On EKS the pod IP 10.0.12.87 comes from the same VPC subnet as the node, so security groups and VPC flow logs see real pod IPs.

Every command, step by step (kind lab with kubectl)

1. Create the cluster

   kind create cluster --config kind-ch62.yaml
   

   What it does: No CNI yet.

   Creating cluster "ch62" ...
   

2. See NotReady

   kubectl get nodes
   

   What it does: Expected.

   ch62-worker   NotReady   <none>
   

3. Install Calico

   bash calico-install.sh
   

   What it does: Takes 1-3 minutes.

   tigerastatus.operator.tigera.io/calico condition met
   

4. Check Calico pods

   kubectl get pods -n calico-system
   

   What it does: One calico-node per node.

   calico-kube-controllers-6b9c...   1/1   Running
   calico-node-4k7xd                 1/1   Running
   calico-node-9q2lm                 1/1   Running
   calico-node-t5zrw                 1/1   Running
   

5. See IP blocks per node

   kubectl get blockaffinities.crd.projectcalico.org -o custom-columns=NODE:.spec.node,CIDR:.spec.cidr
   

   What it does: Each node owns a /26.

   NODE                 CIDR
   ch62-worker          192.168.73.128/26
   ch62-worker2         192.168.104.0/26
   

6. Run two pods

   kubectl apply -f two-pods.yaml
   kubectl get pods -n net -o wide
   

   What it does: Different nodes.

   a   Running   192.168.73.130   ch62-worker
   b   Running   192.168.104.2    ch62-worker2
   

7. Ping across nodes

   kubectl exec -n net a -- ping -c2 192.168.104.2
   

   What it does: Works.

   2 packets transmitted, 2 received, 0% packet loss
   

8. Check source IP seen by b

   kubectl exec -n net a -- curl -s 192.168.104.2 >/dev/null
   kubectl logs -n net b | tail -1
   

   What it does: b sees a's pod IP, so no NAT.

   192.168.73.130 - - [25/Oct/2026:02:40:11 +0000] "GET / HTTP/1.1" 200 615
   

9. Look at the pod's interfaces

   kubectl exec -n net a -- ip -4 addr show eth0
   

   What it does: A /32 on eth0.

   inet 192.168.73.130/32 scope global eth0
   

10. Find the veth on the node

    docker exec ch62-worker ip link | grep cali
    

    What it does: One cali interface per pod on that node.

    9: cali7a3f1e2c4b0@if3: <BROADCAST,MULTICAST,UP>
    

11. Read the CNI config

    docker exec ch62-worker ls /etc/cni/net.d
    

    What it does: kubelet and containerd read this folder.

    10-calico.conflist
    calico-kubeconfig
    

12. Keep the cluster

    kubectl delete pod -n net a b
    

    What it does: 62.2 reuses ch62 and namespace net.

    pod "a" deleted
    pod "b" deleted
    

When it is red 1

container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady

Cause. No CNI is installed or its pods are failing.

Fix, step by step

1. Install a CNI.
2. kubectl get pods -n calico-system and read the logs.

When it is red 2

Failed to create pod sandbox: plugin type="calico" failed (add): error getting ClusterInformation

Cause. calico-node is not ready yet on that node.

Fix, step by step

1. Wait for tigerastatus Available.
2. Check calico-node logs.

When it is red 3

ping: 192.168.104.2: 100% packet loss

Cause. Cross-node routing or a firewall/MTU issue.

Fix, step by step

1. Check the VXLAN routes.
2. On cloud VMs, allow the encapsulation port or protocol between nodes.

When it is red 4

failed to assign an IP address to container (aws-node)

Cause. EKS node ran out of ENI IPs.

Fix, step by step

1. Enable prefix delegation (ENABLE_PREFIX_DELEGATION=true).
2. Use larger instance types or more subnets.

Practice tasks

kubectl get nodes

kubectl get tigerastatus

kubectl get pods -n net -o wide

kubectl exec -n net a -- ping -c2 $B_IP

kubectl logs -n net b

kubectl get blockaffinities.crd.projectcalico.org

docker exec ch62-worker ip link

docker exec ch62-worker ip route

kubectl exec -n net a -- ip route

kubectl get pods -A -o wide

Recap

• Every pod gets its own IP and can reach every other pod without NAT.
• A CNI plugin builds this: IPs, veth pairs and routes.
• No CNI means NotReady nodes.
• Calico routes or tunnels, Cilium uses eBPF, EKS gives VPC IPs.
• Next: ClusterIP Services and EndpointSlices.

FAQ

Q. Why do pods on EKS sometimes stay Pending with IP errors?

The VPC CNI ran out of free IPs in the subnet or hit the per-instance ENI limit. Use bigger subnets, prefix delegation, or custom networking.

Q. Can I change the CNI on a running cluster?

Possible but risky; every pod must be recreated. Most teams build a new cluster and move workloads.

Q. Does the pod CIDR overlap matter?

Yes. If the pod CIDR overlaps your office VPN or VPC ranges, traffic gets routed wrongly. Plan CIDRs early.

Q. Which CNI should I pick on EKS?

The AWS VPC CNI is the default and well supported; Cilium is popular when teams want eBPF features.

Q. What is MTU and why is it 1450 here?

VXLAN adds 50 bytes of header, so the pod MTU is smaller than the node's 1500.

Q. Do pods keep their IP after restart?

No. A new pod gets a new IP. That is why Services exist.

Q. Can I change the CNI on a live cluster?

It is risky and usually means a new cluster or a careful migration.

Q. Does Flannel support NetworkPolicy?

No, Flannel alone does not; Calico or Cilium do.

Q. What is Multus?

A meta-plugin that gives pods more than one network interface, used in telecom.

Interview questions

1. What is the Kubernetes network model?

   Model answer: Every pod has a unique IP; all pods can talk to all pods and nodes without NAT.

2. What does a CNI plugin do?

   Model answer: Allocates the pod IP, creates the interface in the pod's namespace, and sets routes.

3. Why are nodes NotReady after kubeadm init?

   Model answer: No CNI is installed yet.

4. Overlay vs native routing?

   Model answer: Overlay wraps packets in VXLAN or IPIP; native routing advertises pod CIDRs (BGP or VPC).

5. How do EKS pods get IPs?

   Model answer: The VPC CNI attaches ENIs to nodes and assigns secondary IPs or prefixes to pods.

Samajla ka?