Chapter 62: Kubernetes Services, Networking and Load Balancing: CNI, ClusterIP, NodePort, LoadBalancer, DNS and NetworkPolicy

62.7 kube-proxy: how Service IPs become pod IPs with iptables and IPVS

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

A ClusterIP like 10.96.77.140 is not on any network card. No machine owns it, and ping usually gets no reply. Yet curl to it reaches a pod. The piece that makes this happen on every node is kube-proxy. kube-proxy runs as a DaemonSet. It watches Services and EndpointSlices and writes packet-rewriting rules into the node's kernel. In the default iptables mode, a packet sent to a ClusterIP hits the KUBE-SERVICES chain, jumps to a per-Service chain (KUBE-SVC-...), which picks one per-endpoint chain (KUBE-SEP-...) at random using probability rules, and that chain rewrites the destination to the pod IP and port (DNAT). conntrack remembers the choice, so the reply goes back correctly. iptables works well for most clusters, but rules are checked one by one and big rule sets take time to update. Clusters with thousands of Services, like a large Flipkart sale cluster, used IPVS mode, which uses kernel hash tables and offers schedulers like round robin and least connections. Newer options are nftables mode, and CNIs like Cilium that replace kube-proxy entirely with eBPF. In this lesson we follow one Service through the iptables chains, watch the rules change when we scale, check conntrack entries, and look at how to switch kube-proxy to IPVS on kind and how to verify which mode a cluster uses.

What: every term in plain English

• kube-proxy — DaemonSet that programs Service rules on every node.
• iptables mode — Default; uses NAT chains and random probabilities.
• KUBE-SERVICES — Entry chain matching all ClusterIPs.
• KUBE-SVC-* — One chain per Service port, choosing an endpoint.
• KUBE-SEP-* — One chain per endpoint, doing DNAT to the pod.
• DNAT — Rewriting a packet's destination address.
• conntrack — Kernel table that remembers connections and their NAT.
• IPVS mode — Kernel load balancer with hash tables and schedulers.
• nftables mode — Newer backend using nftables instead of iptables.
• kube-proxy replacement — CNIs like Cilium doing Services in eBPF.

How: step by step

1. Find kube-proxy and its mode. ConfigMap and logs.
2. Follow the wallet Service through iptables. SERVICES, SVC, SEP.
3. Scale and watch probabilities change. 1/3, 1/2, rest.
4. Read conntrack entries. NAT recorded.
5. See why ping to a ClusterIP fails. Only the port is handled.
6. Switch a kind cluster to IPVS. kubeProxyMode.
7. Clean up. Delete the extra cluster.

Example 1: Follow the wallet Service through the iptables chains

Goal. Follow the wallet Service through the iptables chains.

trace.sh

CIP=$(kubectl get svc wallet -n net -o jsonpath='{.spec.clusterIP}')
echo "ClusterIP $CIP"
docker exec ch62-worker iptables -t nat -S KUBE-SERVICES | grep "$CIP"
SVC=$(docker exec ch62-worker iptables -t nat -S KUBE-SERVICES | grep "$CIP" | grep -o 'KUBE-SVC-[A-Z0-9]*')
docker exec ch62-worker iptables -t nat -S "$SVC" | grep KUBE-SEP
SEP=$(docker exec ch62-worker iptables -t nat -S "$SVC" | grep -o 'KUBE-SEP-[A-Z0-9]*' | head -1)
docker exec ch62-worker iptables -t nat -S "$SEP" | grep DNAT

What each line does

• docker exec ch62-worker — kind nodes are containers, so we read their iptables directly.
• KUBE-SERVICES line — Matches destination ClusterIP and port 8080.
• KUBE-SVC chain — Has one jump per endpoint with probabilities.
• KUBE-SEP chain — DNAT to pod IP:8080.
• Random choice — statistic mode random spreads connections.

Green output

$ bash trace.sh
ClusterIP 10.96.12.44
-A KUBE-SERVICES -d 10.96.12.44/32 -p tcp -m comment --comment "net/wallet:http cluster IP" -m tcp --dport 8080 -j KUBE-SVC-QH7XJ2KQ4YVZ6A3M
-A KUBE-SVC-QH7XJ2KQ4YVZ6A3M -m comment --comment "net/wallet:http -> 10.244.21.7:8080" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-3TJYR5D6K2WQ7B4N
-A KUBE-SVC-QH7XJ2KQ4YVZ6A3M -m comment --comment "net/wallet:http -> 10.244.40.66:8080" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-7XPLM2QD9C8V1R6A
-A KUBE-SVC-QH7XJ2KQ4YVZ6A3M -m comment --comment "net/wallet:http -> 10.244.40.67:8080" -j KUBE-SEP-J4WQ2ZK8N5Y6T1BC
-A KUBE-SEP-3TJYR5D6K2WQ7B4N -p tcp -m comment --comment "net/wallet:http" -m tcp -j DNAT --to-destination 10.244.21.7:8080

Expected result. 1/3 picks the first pod, half of the rest picks the second, and the remainder goes to the third. Each pod gets an equal share.

Example 2: Watch rules follow scaling, and read conntrack

Goal. Watch rules follow scaling, and read conntrack.

scale-watch.sh

kubectl scale deploy wallet -n net --replicas=5
kubectl rollout status deploy wallet -n net
docker exec ch62-worker iptables -t nat -S | grep -c 'net/wallet:http ->'
kubectl exec -n net curl -- curl -s wallet:8080
docker exec ch62-worker conntrack -L -p tcp --dport 8080 2>/dev/null | head -2
kubectl scale deploy wallet -n net --replicas=3

What each line does

• Scale to 5 — kube-proxy adds two more SEP chains within a second or two.
• grep -c — Counts endpoint jump rules.
• conntrack -L — Shows original destination (ClusterIP) and reply source (pod).
• Run on the right node — conntrack is recorded on the node where the client pod runs.
• Scale back — Rules shrink again.

Green output

$ bash scale-watch.sh
deployment.apps/wallet scaled
deployment "wallet" successfully rolled out
5
wallet-v1
tcp 6 117 TIME_WAIT src=10.244.21.5 dst=10.96.12.44 sport=48212 dport=8080 src=10.244.40.66 dst=10.244.21.5 sport=8080 dport=48212 [ASSURED]

Expected result. The conntrack line shows the request went to the ClusterIP and the reply came from pod 10.244.40.66. That pairing is how DNAT is undone on the way back.

Example 3: Create a kind cluster with kube-proxy in IPVS mode and compare

Goal. Create a kind cluster with kube-proxy in IPVS mode and compare.

kind-ipvs.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: ipvs
networking:
  kubeProxyMode: ipvs
nodes:
  - role: control-plane
  - role: worker

What each line does

• kubeProxyMode: ipvs — kind sets kube-proxy's mode at creation.
• kube-ipvs0 — A dummy interface holding every ClusterIP.
• ipvsadm -Ln — Lists virtual servers and their real servers (pods).
• rr — Round robin scheduler by default.
• Separate cluster — Keeps ch62 untouched.

Green output

$ kind create cluster --config kind-ipvs.yaml
$ kubectl --context kind-ipvs get cm kube-proxy -n kube-system -o jsonpath='{.data.config\.conf}' | grep '^mode'
mode: ipvs
$ kubectl --context kind-ipvs create deploy web --image=nginx:1.27 --replicas=2
$ kubectl --context kind-ipvs expose deploy web --port 80
$ docker exec ipvs-worker sh -c 'apt-get update -qq && apt-get install -y -qq ipvsadm >/dev/null; ipvsadm -Ln | grep -A2 ":80 rr"'
TCP  10.96.140.21:80 rr
  -> 10.244.1.2:80                Masq    1      0          0
  -> 10.244.1.3:80                Masq    1      0          0
$ kind delete cluster --name ipvs

Expected result. IPVS shows the Service as a virtual server with two real servers and the rr scheduler, which is much easier to read than iptables chains.

Every command, step by step (kind lab with kubectl)

1. kube-proxy pods

   kubectl get ds kube-proxy -n kube-system
   

   What it does: One per node.

   kube-proxy   3   3   3   3   3
   

2. Mode

   kubectl get cm kube-proxy -n kube-system -o jsonpath='{.data.config\.conf}' | grep '^mode'
   

   What it does: Empty means iptables.

   mode: iptables
   

3. Logs

   kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=3
   

   What it does: Mode confirmation.

   "Using iptables Proxier"
   

4. Trace

   bash trace.sh
   

   What it does: SERVICES to SEP.

   DNAT --to-destination 10.244.21.7:8080
   

5. Rule count

   docker exec ch62-worker iptables-save -t nat | wc -l
   

   What it does: Grows with Services.

   212
   

6. Ping a ClusterIP

   kubectl exec -n net curl -- ping -c1 -W1 10.96.12.44
   

   What it does: No ICMP rule.

   1 packets transmitted, 0 received, 100% packet loss
   

7. Curl it

   kubectl exec -n net curl -- curl -s 10.96.12.44:8080
   

   What it does: Only TCP port works.

   wallet-v1
   

8. Scale

   bash scale-watch.sh
   

   What it does: Rules follow.

   5
   

9. Delete kube-proxy pod

   kubectl delete pod -n kube-system -l k8s-app=kube-proxy --field-selector spec.nodeName=ch62-worker
   

   What it does: Recreated; rules resynced.

   pod "kube-proxy-x2k9p" deleted
   

10. IPVS cluster

    kind create cluster --config kind-ipvs.yaml
    

    What it does: Compare.

    Creating cluster "ipvs" ...
    

11. Clean up

    kind delete cluster --name ipvs
    

    What it does: Back to ch62.

    Deleted nodes: ["ipvs-control-plane" "ipvs-worker"]
    

When it is red 1

Failed to execute iptables-restore: exit status 4

Cause. Another tool holds the xtables lock or rules conflict.

Fix, step by step

1. Check for other iptables writers like firewalld; kube-proxy retries.

When it is red 2

Service works on some nodes, not others

Cause. kube-proxy broken or not running on those nodes.

Fix, step by step

1. kubectl get pods -n kube-system -o wide -l k8s-app=kube-proxy; check logs.

When it is red 3

can't set sysctl net/ipv4/vs/conn_reuse_mode

Cause. IPVS kernel modules missing.

Fix, step by step

1. Load ip_vs modules or use iptables mode.

When it is red 4

Old pod IP still receives traffic after scaling

Cause. Existing conntrack entries for long-lived connections.

Fix, step by step

1. Reconnect clients; use graceful shutdown with preStop.

When it is red 5

Slow Service updates with many Services

Cause. Large iptables rule sets.

Fix, step by step

1. Consider IPVS, nftables mode, or an eBPF CNI.

Practice tasks

kubectl get ds kube-proxy -n kube-system

kubectl get cm kube-proxy -n kube-system -o yaml

kubectl exec -n net curl -- curl -s wallet:8080

bash trace.sh

bash scale-watch.sh

docker exec ch62-worker conntrack -L -p tcp --dport 8080

docker exec ch62-worker iptables-save -t nat | wc -l

kind create cluster --config kind-ipvs.yaml

kubectl get cm kube-proxy -n kube-system -o yaml

kubectl get pods -n kube-system

Recap

• kube-proxy watches Services and EndpointSlices and writes node rules.
• iptables mode: KUBE-SERVICES, KUBE-SVC (random choice), KUBE-SEP (DNAT).
• conntrack keeps each connection on its chosen pod.
• IPVS uses hash tables and schedulers; nftables and eBPF are newer options.
• Next: session affinity and topology-aware routing.

FAQ

Q. What is nftables mode and should I use it?

A newer kube-proxy backend using nftables, the successor to iptables in Linux. It updates rules faster on big clusters. Use it when your Kubernetes version and node kernels support it well.

Q. How long does kube-proxy take to apply a change?

Usually under a second on small clusters. On very large iptables clusters a full sync can take several seconds, which delays new endpoints.

Q. Can I see kube-proxy metrics?

Yes, on port 10249 /metrics, for example sync duration histograms that show how long rule updates take.

Q. Why do some guides say ClusterIPs never answer ping while mine does?

In IPVS mode the ClusterIP sits on the kube-ipvs0 interface, so the node may reply to ping. In iptables mode it never does.

Q. Why is the iptables-save output so long?

Each Service port adds a SVC chain and each endpoint adds a SEP chain, plus NodePort, firewall and masquerade rules. A cluster with 2,000 Services and 10 pods each has tens of thousands of rules.

Q. What is masquerading in these rules?

KUBE-MARK-MASQ marks packets that need SNAT, for example traffic from outside the pod network, so replies come back through the same node.

Q. How do I check which mode an EKS cluster uses?

Read the kube-proxy ConfigMap in kube-system or the kube-proxy logs, the same as on kind. The EKS add-on configuration can switch it to IPVS.

Q. Does kube-proxy carry traffic itself?

No, in iptables and IPVS modes the kernel does; kube-proxy only writes rules.

Q. Is iptables mode slow?

Not for most clusters; it matters at many thousands of Services.

Q. Can I remove kube-proxy?

Yes, if your CNI replaces it, such as Cilium's kube-proxy replacement.

Q. Does EKS use kube-proxy?

Yes, as an EKS add-on, in iptables mode by default.

Q. Why random and not round robin?

iptables has no shared counter; probabilities give an even spread.

Q. Where do NodePort rules live?

In the KUBE-NODEPORTS chain, seen in 62.3.

Interview questions

1. How does a ClusterIP reach a pod?

   Model answer: kube-proxy rules DNAT the ClusterIP and port to a chosen pod IP; conntrack handles replies.

2. Explain the iptables chains for a Service.

   Model answer: KUBE-SERVICES matches the IP, KUBE-SVC picks an endpoint with probabilities, KUBE-SEP DNATs.

3. iptables vs IPVS?

   Model answer: IPVS uses hash tables and has schedulers; better at very large scale.

4. Why can't you ping a ClusterIP?

   Model answer: It is virtual; only the configured ports have rules.

5. What happens if kube-proxy dies on a node?

   Model answer: Existing rules stay, but new changes are not applied on that node.

6. How would you confirm traffic to a Service reaches the right pods?

   Model answer: Check EndpointSlices, trace the KUBE-SVC and KUBE-SEP chains or ipvsadm entries on the client's node, and read conntrack entries for the ClusterIP and port.

Samajla ka?