Chapter 62: Kubernetes Services, Networking and Load Balancing: CNI, ClusterIP, NodePort, LoadBalancer, DNS and NetworkPolicy

62.6 CoreDNS: how service names resolve, ndots, and fixing DNS problems

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Every lesson so far used names like wallet, store or pg-0.pg, and they just worked. The server behind that is CoreDNS. It runs as a Deployment in kube-system behind a Service called kube-dns, usually at 10.96.0.10, and every pod's /etc/resolv.conf points to it. When it fails, everything fails. An outage at a company like PhonePe can begin with a single wrong edit to the CoreDNS ConfigMap or with CoreDNS pods starved of CPU, and suddenly every service call shows 'could not resolve host'. DNS is also a quiet performance problem: with the default ndots:5 setting, a lookup of api.razorpay.com from a pod first tries several cluster search domains before the real name, so one call can become four or five DNS queries. CoreDNS is configured by a file called the Corefile, stored in the coredns ConfigMap. Each plugin line does one job: kubernetes answers cluster names, forward sends other names to upstream DNS, cache keeps answers, log and errors help debugging. You can add a stub domain so names like corp.internal go to your company DNS server. In this lesson we read resolv.conf and the Corefile, watch the search list in action, add a custom forward zone, turn on query logging, and debug a pod with a broken dnsPolicy.

What: every term in plain English

• CoreDNS — The cluster DNS server, running in kube-system.
• kube-dns Service — The Service in front of CoreDNS, kept under the old name.
• Corefile — CoreDNS config, inside the coredns ConfigMap.
• search list — Domains tried after a short name, like net.svc.cluster.local.
• ndots — Names with fewer dots than this go through the search list first.
• FQDN — A full name ending in a dot, which skips the search list.
• forward plugin — Sends non-cluster names to upstream DNS.
• dnsPolicy — Pod setting: ClusterFirst (default), Default, None, ClusterFirstWithHostNet.
• dnsConfig — Pod-level overrides like ndots or extra nameservers.
• NodeLocal DNSCache — A DNS cache on every node to cut latency and load.

How: step by step

1. Read a pod's resolv.conf. nameserver, search, ndots.
2. Read the Corefile. Plugins.
3. Watch search list expansion. log plugin.
4. Use an FQDN to skip the search. Trailing dot.
5. Add a forward zone for corp.internal. Stub domain.
6. Break dnsPolicy and fix it. dnsPolicy: Default.
7. Clean up. Restore the Corefile.

Example 1: Read how a pod is told to use CoreDNS

Goal. Read how a pod is told to use CoreDNS.

dns-check.sh

kubectl exec -n net curl -- cat /etc/resolv.conf
kubectl get svc -n kube-system kube-dns
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

What each line does

• nameserver 10.96.0.10 — The kube-dns ClusterIP.
• search — net.svc.cluster.local, svc.cluster.local, cluster.local.
• options ndots:5 — Short names try the search list first.
• Two CoreDNS pods — For availability, on different nodes.
• wallet — Becomes wallet.net.svc.cluster.local on the first try.

Green output

$ bash dns-check.sh
search net.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
NAME       TYPE        CLUSTER-IP   PORT(S)
kube-dns   ClusterIP   10.96.0.10   53/UDP,53/TCP,9153/TCP
NAME                       READY   STATUS    NODE
coredns-7db6d8ff4d-6hlz2   1/1     Running   ch62-control-plane
coredns-7db6d8ff4d-wq8nt   1/1     Running   ch62-control-plane

Expected result. kubelet wrote this file when the pod started. Port 9153 is CoreDNS's Prometheus metrics port.

Example 2: Turn on query logging and see ndots at work

Goal. Turn on query logging and see ndots at work.

enable-log.sh

kubectl get cm coredns -n kube-system -o yaml > coredns-backup.yaml
kubectl get cm coredns -n kube-system -o jsonpath='{.data.Corefile}' > Corefile
sed -i 's/^    errors$/    errors\n    log/' Corefile
kubectl create cm coredns -n kube-system --from-file=Corefile --dry-run=client -o yaml | kubectl apply -f -
sleep 35
kubectl exec -n net curl -- curl -s -o /dev/null -m 3 http://api.razorpay.com
kubectl logs -n kube-system -l k8s-app=kube-dns --since=1m | grep razorpay

What each line does

• Backup first — Always save the ConfigMap before editing.
• log plugin — Logs every query.
• reload — The reload plugin picks up changes in about 30 seconds.
• NXDOMAIN lines — Search domains tried first and failing.
• Last line — The real name, finally NOERROR.

Green output

$ bash enable-log.sh
configmap/coredns configured
[INFO] 10.244.21.5 "A IN api.razorpay.com.net.svc.cluster.local. udp" NXDOMAIN
[INFO] 10.244.21.5 "A IN api.razorpay.com.svc.cluster.local. udp" NXDOMAIN
[INFO] 10.244.21.5 "A IN api.razorpay.com.cluster.local. udp" NXDOMAIN
[INFO] 10.244.21.5 "A IN api.razorpay.com. udp" NOERROR

Expected result. One outside call cost four queries (eight with AAAA). Using api.razorpay.com. with a trailing dot, or ndots:2 in dnsConfig, cuts it to one.

Example 3: Forward a company zone and fix a pod with the wrong dnsPolicy

Goal. Forward a company zone and fix a pod with the wrong dnsPolicy.

corp-zone.txt

corp.internal:53 {
    errors
    cache 30
    forward . 10.20.0.2
}

bad-dns.yaml

apiVersion: v1
kind: Pod
metadata:
  name: bad-dns
  namespace: net
spec:
  dnsPolicy: Default
  containers:
    - name: c
      image: curlimages/curl:8.10.1
      command: ["sleep", "3600"]

What each line does

• corp.internal block — A separate server block; only that zone goes to 10.20.0.2.
• forward . 10.20.0.2 — Company DNS server, assumed reachable from nodes.
• dnsPolicy: Default — Uses the node's resolv.conf, not CoreDNS. Name is confusing.
• Symptom — Service names fail; internet names work.
• Fix — Remove dnsPolicy (ClusterFirst is the default).

Green output

$ cat corp-zone.txt >> Corefile
$ kubectl create cm coredns -n kube-system --from-file=Corefile --dry-run=client -o yaml | kubectl apply -f -
$ kubectl apply -f bad-dns.yaml
$ kubectl exec -n net bad-dns -- curl -s -m 3 wallet:8080
curl: (6) Could not resolve host: wallet
$ kubectl exec -n net bad-dns -- cat /etc/resolv.conf
nameserver 172.18.0.1
$ kubectl delete pod -n net bad-dns
$ sed -i '/dnsPolicy/d' bad-dns.yaml && kubectl apply -f bad-dns.yaml
$ kubectl exec -n net bad-dns -- curl -s wallet:8080
wallet-v1

Expected result. With dnsPolicy Default the pod used Docker's resolver, which knows nothing about cluster Services. Removing it restored ClusterFirst.

Every command, step by step (kind lab with kubectl)

1. resolv.conf

   kubectl exec -n net curl -- cat /etc/resolv.conf
   

   What it does: Three lines.

   nameserver 10.96.0.10
   

2. Corefile

   kubectl get cm coredns -n kube-system -o jsonpath='{.data.Corefile}'
   

   What it does: Plugins.

   .:53 {
       errors
       health
       kubernetes cluster.local in-addr.arpa ip6.arpa
       forward . /etc/resolv.conf
       cache 30
       reload
   }
   

3. Short name

   kubectl exec -n net curl -- nslookup wallet
   

   What it does: Search list.

   Name:   wallet.net.svc.cluster.local
   

4. Other namespace

   kubectl exec -n net curl -- nslookup kube-dns.kube-system
   

   What it does: Namespace in the name.

   Address: 10.96.0.10
   

5. Reverse lookup

   kubectl exec -n net curl -- nslookup 10.96.0.10
   

   What it does: PTR.

   10.0.96.10.in-addr.arpa name = kube-dns.kube-system.svc.cluster.local.
   

6. Enable log

   bash enable-log.sh
   

   What it does: Query log.

   NXDOMAIN ... NOERROR
   

7. FQDN

   kubectl exec -n net curl -- curl -s -o /dev/null -m 3 http://api.razorpay.com.
   

   What it does: Trailing dot; one query.

   [INFO] "A IN api.razorpay.com. udp" NOERROR
   

8. Lower ndots

   kubectl patch deploy wallet -n net -p '{"spec":{"template":{"spec":{"dnsConfig":{"options":[{"name":"ndots","value":"2"}]}}}}}'
   

   What it does: Fewer extra queries.

   deployment.apps/wallet patched
   

9. Metrics

   kubectl exec -n net curl -- curl -s 10.96.0.10:9153/metrics | grep -m2 coredns_dns_requests_total
   

   What it does: Prometheus counter.

   coredns_dns_requests_total{...type="A"} 214
   

10. Restore

    kubectl apply -f coredns-backup.yaml
    kubectl rollout restart deploy coredns -n kube-system
    

    What it does: Back to original.

    deployment.apps/coredns restarted
    

When it is red 1

SERVFAIL for every outside name

Cause. Upstream DNS unreachable from nodes, or forward points to a wrong address.

Fix, step by step

1. Test the upstream from a node; fix forward in the Corefile.

When it is red 2

Could not resolve host: wallet

Cause. CoreDNS down, wrong dnsPolicy, or NetworkPolicy blocks port 53.

Fix, step by step

1. Check CoreDNS pods, dnsPolicy, and allow UDP/TCP 53 to kube-system.

When it is red 3

plugin/loop: Loop (127.0.0.1:...) detected

Cause. forward points to a resolver that sends back to CoreDNS.

Fix, step by step

1. Point forward at a real upstream, not 127.0.0.53 on systemd-resolved nodes.

When it is red 4

CoreDNS CrashLoopBackOff after edit

Cause. Syntax error in the Corefile.

Fix, step by step

1. kubectl logs; restore coredns-backup.yaml.

When it is red 5

i/o timeout on DNS lookups under load

Cause. CoreDNS overloaded or conntrack races on UDP.

Fix, step by step

1. Scale CoreDNS, use NodeLocal DNSCache, lower ndots.

Practice tasks

kubectl exec -n net curl -- cat /etc/resolv.conf

kubectl get cm coredns -n kube-system -o yaml

kubectl exec -n net curl -- nslookup wallet

kubectl logs -n kube-system -l k8s-app=kube-dns

kubectl get deploy wallet -n net -o yaml

kubectl get cm coredns -n kube-system -o yaml

kubectl apply -f bad-dns.yaml

kubectl exec -n net curl -- curl -s 10.96.0.10:9153/metrics

kubectl get pods -n kube-system -l k8s-app=kube-dns

kubectl get ds -n kube-system

Recap

• CoreDNS answers all cluster names behind the kube-dns Service.
• resolv.conf has the nameserver, search list and ndots.
• ndots:5 makes outside names cost extra queries; use FQDNs or lower ndots.
• The Corefile controls zones, forwarding, cache and logging.
• Next: kube-proxy, which turns Service IPs into pod IPs.

FAQ

Q. Why did my edit to the Corefile disappear on EKS?

The CoreDNS add-on can overwrite manual changes on update. Put custom settings in the add-on configuration so they survive upgrades.

Q. Can CoreDNS answer custom names like db.local?

Yes, with the hosts plugin or a file zone in the Corefile, but prefer real Services so names follow pods.

Q. Why do Alpine-based images sometimes resolve names differently?

Alpine uses musl libc, whose resolver handles search lists and TCP fallback differently from glibc. If odd DNS failures appear only in Alpine pods, test with a Debian-based image.

Q. How many queries per second can CoreDNS handle?

Thousands per pod, depending on CPU and cache hits. Watch its CPU and scale replicas before big events.

Q. What should I watch on a DNS dashboard?

Request rate, error rate (SERVFAIL and NXDOMAIN), response time, cache hit ratio, and CoreDNS CPU and memory. A rising NXDOMAIN rate often means search-list waste or a typo in a service name after a release.

Q. Why does the first lookup after a restart feel slow?

The cache is empty, so every name goes upstream once. After that the cache plugin answers repeated names for up to 30 seconds.

Q. Does every outside call really cost several queries?

With ndots:5 and a name like api.razorpay.com, yes: each search domain is tried, for both A and AAAA. Busy services can multiply DNS load several times this way.

Q. How do I test DNS from a node without a pod?

Use dig or nslookup against the kube-dns ClusterIP from the node, for example nslookup wallet.net.svc.cluster.local 10.96.0.10.

Q. Why is the Service called kube-dns?

For compatibility with the older kube-dns server.

Q. Should I edit the Corefile on EKS?

It is managed by the CoreDNS add-on; use its configuration values or careful edits.

Q. How many CoreDNS replicas?

At least two, scaled with cluster size, for example with the cluster proportional autoscaler.

Q. Can a pod use a custom nameserver?

Yes, dnsPolicy None with dnsConfig.

Q. What does hostNetwork do to DNS?

It uses the node's DNS unless dnsPolicy is ClusterFirstWithHostNet.

Q. Is DNS UDP or TCP?

Mostly UDP, falling back to TCP for large answers. Allow both.

Interview questions

1. How does a pod resolve a Service name?

   Model answer: resolv.conf points to CoreDNS; the search list expands short names; the kubernetes plugin answers.

2. What is ndots and why tune it?

   Model answer: Names with fewer dots go through search domains first; lowering it cuts wasted queries.

3. How do you forward a corporate domain?

   Model answer: Add a server block in the Corefile with forward to the corporate DNS.

4. How do you debug DNS in a cluster?

   Model answer: Check CoreDNS pods and logs, resolv.conf, dnsPolicy, NetworkPolicy on port 53, upstream reachability.

5. What is NodeLocal DNSCache?

   Model answer: A per-node DNS cache DaemonSet that reduces latency and conntrack issues.

6. A service suddenly cannot resolve any names. What do you check first?

   Model answer: CoreDNS pod status and logs, recent Corefile changes, NetworkPolicies blocking port 53, the pod's dnsPolicy and resolv.conf, and whether upstream DNS answers from the nodes.

Samajla ka?