Chapter 62: Kubernetes Services, Networking and Load Balancing: CNI, ClusterIP, NodePort, LoadBalancer, DNS and NetworkPolicy

62.8 Session affinity, traffic policies and topology-aware routing

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

By default a Service spreads each new connection randomly across its pods. That is usually what you want. Two common situations need more control. The first is stickiness. An older Paytm admin app keeps a user's login session in pod memory. If the second request lands on a different pod, the user is logged out. The real fix is to store sessions in Redis, but until then sessionAffinity: ClientIP sends all connections from one client IP to the same pod for a timeout (3 hours by default). It is per client IP, so many users behind one office NAT all stick to one pod. The second is cost and speed. On EKS spread across three availability zones in ap-south-1, a call from a pod in zone a to a pod in zone b crosses zones, adds latency and costs money for inter-AZ data transfer. Topology-aware routing tells kube-proxy to prefer endpoints in the same zone when there are enough of them. Kubernetes marks EndpointSlice entries with zone hints, and kube-proxy on each node uses only the hinted endpoints. Newer versions also offer spec.trafficDistribution: PreferClose with a simpler behaviour. Finally, internalTrafficPolicy: Local keeps traffic on the same node, which fits node agents like a log collector DaemonSet. In this lesson we label kind nodes with zones, try each feature, and see where each one fails.

What: every term in plain English

• sessionAffinity: ClientIP — Same client IP goes to the same pod.
• timeoutSeconds — How long affinity lasts; default 10800 seconds.
• Sticky sessions — Keeping a user on one backend.
• topology.kubernetes.io/zone — Node label with its zone.
• Topology-aware routing — Prefer same-zone endpoints using hints.
• service.kubernetes.io/topology-mode: Auto — Annotation that turns on hints.
• hints.forZones — EndpointSlice field saying which zone may use an endpoint.
• trafficDistribution: PreferClose — Newer field to prefer close endpoints.
• internalTrafficPolicy: Local — In-cluster traffic only to pods on the same node.
• Inter-AZ cost — AWS charges for data between availability zones.

How: step by step

1. Show default spreading. Many pods answer.
2. Enable ClientIP affinity. One pod answers.
3. Label nodes with zones. Fake zones on kind.
4. Enable topology-aware routing. Read hints.
5. Try trafficDistribution. PreferClose.
6. Use internalTrafficPolicy Local for a node agent. Same node only.
7. Clean up. Remove annotations.

Example 1: Make the Paytm admin app sticky with ClientIP affinity

Goal. Make the Paytm admin app sticky with ClientIP affinity.

admin.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: admin
  namespace: net
spec:
  replicas: 3
  selector:
    matchLabels:
      app: admin
  template:
    metadata:
      labels:
        app: admin
    spec:
      containers:
        - name: whoami
          image: traefik/whoami:v1.10
          ports:
            - name: http
              containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: admin
  namespace: net
spec:
  selector:
    app: admin
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 600
  ports:
    - name: http
      port: 80
      targetPort: http

What each line does

• traefik/whoami — Prints the pod hostname, so we can see which pod answered.
• sessionAffinity: ClientIP — kube-proxy uses the iptables recent module to remember the client.
• timeoutSeconds: 600 — 10 minutes of stickiness after the last connection.
• Test — Ten requests from one curl pod.
• Expected — One hostname only.

Green output

$ kubectl apply -f admin.yaml
$ for i in $(seq 10); do kubectl exec -n net curl -- curl -s admin | grep Hostname; done | sort | uniq -c
     10 Hostname: admin-7c9d5b6f4-qx2lm
$ kubectl patch svc admin -n net -p '{"spec":{"sessionAffinity":"None"}}'
$ for i in $(seq 10); do kubectl exec -n net curl -- curl -s admin | grep Hostname; done | sort | uniq -c
      3 Hostname: admin-7c9d5b6f4-8wz4k
      4 Hostname: admin-7c9d5b6f4-qx2lm
      3 Hostname: admin-7c9d5b6f4-t5j9r

Expected result. With ClientIP all ten went to one pod. With None they spread across all three.

Example 2: Fake zones on kind and turn on topology-aware routing

Goal. Fake zones on kind and turn on topology-aware routing.

zones.sh

kubectl label node ch62-worker topology.kubernetes.io/zone=ap-south-1a --overwrite
kubectl label node ch62-worker2 topology.kubernetes.io/zone=ap-south-1b --overwrite
kubectl label node ch62-control-plane topology.kubernetes.io/zone=ap-south-1a --overwrite
kubectl scale deploy admin -n net --replicas=6
kubectl rollout status deploy admin -n net
kubectl annotate svc admin -n net service.kubernetes.io/topology-mode=Auto --overwrite
sleep 5
kubectl get endpointslices -n net -l kubernetes.io/service-name=admin -o jsonpath='{range .items[*].endpoints[*]}{.nodeName}{" "}{.hints.forZones[0].name}{"\n"}{end}'

What each line does

• Zone labels — Clouds set these automatically; we set them by hand.
• 6 replicas — Hints need enough endpoints in each zone.
• topology-mode=Auto — EndpointSlice controller adds hints when safe.
• hints.forZones — Which zone's nodes should use the endpoint.
• Empty hints — Controller decided it was not safe, for example uneven spread.

Green output

$ bash zones.sh
ch62-worker ap-south-1a
ch62-worker ap-south-1a
ch62-worker ap-south-1a
ch62-worker2 ap-south-1b
ch62-worker2 ap-south-1b
ch62-worker2 ap-south-1b
$ kubectl exec -n net curl -- sh -c 'for i in 1 2 3 4 5 6; do curl -s admin | grep Hostname; done' | sort -u | wc -l
3

Expected result. The curl pod runs in zone 1b (ch62-worker2), and only the three 1b pods answer. Cross-zone traffic is gone.

Example 3: Keep a node agent's traffic on its own node with internalTrafficPolicy: Local

Goal. Keep a node agent's traffic on its own node with internalTrafficPolicy: Local.

agent.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: agent
  namespace: net
spec:
  selector:
    matchLabels:
      app: agent
  template:
    metadata:
      labels:
        app: agent
    spec:
      containers:
        - name: whoami
          image: traefik/whoami:v1.10
          ports:
            - name: http
              containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: agent
  namespace: net
spec:
  internalTrafficPolicy: Local
  selector:
    app: agent
  ports:
    - name: http
      port: 80
      targetPort: http

What each line does

• DaemonSet — One agent pod per node, like a log or metrics collector.
• internalTrafficPolicy: Local — Pods talk only to the agent on their own node.
• Benefit — No cross-node hops, and node-specific data stays local.
• Risk — If the local agent is down, traffic is dropped, not rerouted.
• Check — Compare the answering pod's node with the curl pod's node.

Green output

$ kubectl apply -f agent.yaml
$ kubectl get pod -n net curl -o jsonpath='{.spec.nodeName}{"\n"}'
ch62-worker2
$ kubectl exec -n net curl -- curl -s agent | grep Hostname
Hostname: agent-p7v2k
$ kubectl get pod -n net agent-p7v2k -o jsonpath='{.spec.nodeName}{"\n"}'
ch62-worker2

Expected result. Every call reaches the agent on the same node. Delete that agent pod and calls fail until it returns, which is the trade-off.

Every command, step by step (kind lab with kubectl)

1. Deploy admin

   kubectl apply -f admin.yaml
   

   What it does: whoami pods.

   service/admin created
   

2. Sticky

   for i in $(seq 10); do kubectl exec -n net curl -- curl -s admin | grep Hostname; done | sort | uniq -c
   

   What it does: One pod.

   10 Hostname: admin-7c9d5b6f4-qx2lm
   

3. Read the rule

   docker exec ch62-worker2 iptables -t nat -S | grep -m1 'recent'
   

   What it does: Affinity uses the recent module.

   ... -m recent --rcheck --seconds 600 --reap --name KUBE-SEP-... 
   

4. Turn off

   kubectl patch svc admin -n net -p '{"spec":{"sessionAffinity":"None"}}'
   

   What it does: Spread again.

   service/admin patched
   

5. Zones

   bash zones.sh
   

   What it does: Hints.

   ch62-worker ap-south-1a
   

6. Events

   kubectl describe svc admin -n net | grep -i topology
   

   What it does: Controller notes.

   Normal  TopologyAwareHintsEnabled  Topology Aware Hints has been enabled
   

7. PreferClose

   kubectl annotate svc admin -n net service.kubernetes.io/topology-mode-
   kubectl patch svc admin -n net -p '{"spec":{"trafficDistribution":"PreferClose"}}'
   

   What it does: Newer field.

   service/admin patched
   

8. Agent

   kubectl apply -f agent.yaml
   

   What it does: DaemonSet plus Local.

   daemonset.apps/agent created
   

9. Local check

   kubectl exec -n net curl -- curl -s agent | grep Hostname
   

   What it does: Same node.

   Hostname: agent-p7v2k
   

10. Break local

    kubectl delete pod -n net agent-p7v2k --wait=false
    kubectl exec -n net curl -- curl -s -m 2 agent
    

    What it does: Drop while restarting.

    curl: (28) Connection timed out
    

11. Clean up

    kubectl delete -f agent.yaml -f admin.yaml
    

    What it does: Keep zone labels.

    service "admin" deleted
    

When it is red 1

Users still logged out with ClientIP affinity

Cause. Clients come through a proxy or LB with changing source IPs.

Fix, step by step

1. Store sessions in Redis, or use cookie affinity at the Ingress.

When it is red 2

TopologyAwareHintsDisabled: insufficient number of endpoints

Cause. Too few pods per zone.

Fix, step by step

1. Scale up and spread pods across zones.

When it is red 3

One zone overloaded with topology routing

Cause. Uneven traffic per zone.

Fix, step by step

1. Spread pods by zone; use HPA; check hints.

When it is red 4

Connection refused/timeout with internalTrafficPolicy Local

Cause. No ready pod on the client's node.

Fix, step by step

1. Make it a DaemonSet or remove Local.

Practice tasks

kubectl get svc admin -n net -o yaml

kubectl patch svc admin -n net -p '{"spec":{"sessionAffinity":"None"}}'

kubectl get nodes -L topology.kubernetes.io/zone

kubectl get endpointslices -n net -o yaml

kubectl describe svc admin -n net

kubectl get svc admin -n net -o yaml

kubectl get ds -n net

docker exec ch62-worker2 iptables -t nat -S

kubectl get endpointslices -n net

kubectl get pods -n net -o wide

Recap

• sessionAffinity: ClientIP keeps one client IP on one pod for a timeout.
• Topology-aware routing prefers same-zone pods and saves inter-AZ cost.
• trafficDistribution: PreferClose is the newer, simpler option.
• internalTrafficPolicy: Local keeps traffic on the same node.
• Next: NetworkPolicy with Calico.

FAQ

Q. Does session affinity survive a pod restart?

No. If the chosen pod goes away, the client is sent to a new pod and the in-memory session is lost.

Q. Can I make affinity last forever?

The maximum timeoutSeconds is 86400, one day. Long stickiness makes load uneven, so keep it short.

Q. Does topology routing help with latency too?

Yes. Same-zone calls in ap-south-1 are usually a fraction of a millisecond faster, which adds up in chatty microservices.

Q. How do I spread pods evenly across zones?

Use topologySpreadConstraints with topologyKey topology.kubernetes.io/zone and maxSkew 1, so hints can stay enabled.

Q. How do I know if hints are being used?

Read the EndpointSlice: each endpoint has hints.forZones. Also check the Service events for TopologyAwareHintsEnabled or Disabled with a reason.

Q. What happens when a zone loses all its pods?

The controller removes hints, and kube-proxy falls back to using all endpoints, so traffic still flows, just across zones.

Q. Why does PreferClose exist if topology-mode already works?

Topology mode tries to balance by CPU per zone and turns itself off in many cases. PreferClose simply prefers the same zone when any endpoint is there, which is easier to predict.

Q. Is ClientIP affinity a real session fix?

No. Shared session storage is the real fix.

Q. Does affinity work with Ingress?

The Ingress controller is the client; use cookie affinity at the Ingress.

Q. Does topology routing work on kind?

Yes, with fake zone labels as shown.

Q. Is inter-AZ traffic really charged?

Yes, AWS charges per GB in each direction between zones.

Q. Can affinity and topology be combined?

Yes, but keep it simple and measure.

Q. Does externalTrafficPolicy Local relate?

It is the same idea for traffic from outside, seen in 62.3.

Interview questions

1. What is sessionAffinity ClientIP?

   Model answer: kube-proxy sends a given client IP to the same endpoint until a timeout.

2. Why can ClientIP affinity be uneven?

   Model answer: Many users behind one NAT IP all land on one pod.

3. What is topology-aware routing?

   Model answer: EndpointSlice hints make kube-proxy prefer same-zone endpoints.

4. When is internalTrafficPolicy Local useful?

   Model answer: Node-local agents like logging or DNS caches.

5. How does topology routing save money?

   Model answer: It avoids cross-AZ data transfer charges.

6. How would you reduce inter-AZ data transfer cost for service-to-service calls on EKS?

   Model answer: Spread pods evenly across zones, enable topology-aware routing or trafficDistribution PreferClose, and confirm hints in EndpointSlices; measure with VPC flow logs or cost reports.

Samajla ka?