Kubernetes DNS at Scale — CoreDNS, NodeLocal DNSCache & the 5-Second SERVFAIL Bug

Kubernetes DNS at Scale: CoreDNS, NodeLocal DNSCache, and the 5-Second SERVFAIL Bug

Daily DevOps Mentor — 2026-09-09

1. Topic of the Day

DNS is the one dependency every workload has whether it knows it or not. A pod that never opens a socket to another Service still resolves kubernetes.default.svc.cluster.local on startup for client discovery, still resolves the RDS endpoint, still resolves the third-party API host in its HTTP_PROXY chain. It is the single most universally-shared, least-observed subsystem in a Kubernetes cluster, and it fails in ways that don't show up in the usual dashboards: CPU is fine, memory is fine, the Service is Running and 1/1 Ready, and yet checkout latency has a bimodal distribution with a second hump sitting at exactly 5 seconds.

CoreDNS exists because kube-dns (the original dnsmasq + kubedns + sidecar stack) didn't scale operationally — three containers per pod, a Go binary wrapping a fork of BIND-like resolution logic, and a plugin model that made anything beyond basic Service/Pod A-record serving painful. CoreDNS replaced it with a single static Go binary and a middleware chain (Caddy-style plugins: kubernetes, forward, cache, autopath, errors, health, ready, prometheus) that's declarative, testable, and — critically — horizontally scalable without touching client configuration, because every pod still talks to the same kube-dns ClusterIP.

NodeLocal DNSCache exists because CoreDNS-as-a-ClusterIP-Service has a structural weakness at scale: every DNS query, even a cache hit, still crosses the Service abstraction — iptables/IPVS DNAT, conntrack table insertion, and (in the worst case) a trip across node boundaries to a CoreDNS pod on a different node. At low QPS this is invisible. At the QPS a 2,000-node fleet with ndots:5 generates — where a single unqualified hostname lookup fans out into up to 5 actual queries before resolving or failing — it becomes conntrack pressure, UDP packet loss, and the infamous 5-second DNS timeout that shows up in literally every major cloud provider's Kubernetes troubleshooting guide, because glibc's resolver has no way to distinguish "the answer is NXDOMAIN" from "the UDP packet got dropped by a racing conntrack insert," so it just waits out the timeout.

Every large-scale Kubernetes operator — Uber's Networking team, Datadog's platform team, DoorDash, Reddit — has published a postmortem or engineering blog about hitting some variant of this problem, because it is baked into the intersection of Linux netfilter internals, the glibc resolver, and how Kubernetes composes Services. Today's session is a full production tour of that intersection: how it's built, how it fails, how you instrument it, and how you fix it permanently rather than papering over it with a longer timeout.

2. Real Business Problem

Scenario: You run a multi-tenant EKS platform for an e-commerce company. 340 nodes, ~9,000 pods, mixed workload — Node.js and Java checkout services, Python ML inference, a Kafka-backed order pipeline. During a flash-sale event, traffic to the checkout path spikes 6x over 20 minutes.

Symptoms reported by the on-call SRE:

  • Checkout API p50 latency is normal (~80ms), but p99 has a second cluster sitting at 5,000–5,200ms almost exactly.
  • The affected requests are not concentrated on any single node or AZ — they're diffuse, roughly 0.3% of requests cluster-wide, rising to 2% at peak.
  • Application logs show getaddrinfo ENOTFOUND or java.net.UnknownHostException for orders-db.internal.svc.cluster.local and for the external payment gateway's public hostname — both on the same pods that successfully resolved the same names milliseconds earlier.
  • CoreDNS pods show 40% CPU, no OOMKilled events, Ready the entire time.
  • kubectl top nodes looks unremarkable. No node is CPU or memory saturated.
  • Restarting the affected pods "fixes" it for a while, which is the classic decoy that sends people down the wrong path (pod-level resource limits, JVM DNS caching, connection pool exhaustion).

This is the shape of a conntrack-table-driven DNS failure, and it is deliberately picked because none of the standard signals (CPU, memory, Pod readiness, error rate on the Service itself) point at the actual bottleneck. The real constraint is a kernel-level table on specific nodes that isn't part of anyone's default dashboard, and the "randomly failing 0.3% of the time" pattern is the fingerprint of a race condition rather than a resource ceiling.

3. Production Architecture

Kubernetes DNS at Scale — Production Architecture

Why it's designed this way. The architecture has three concentric layers deliberately: (1) an in-pod resolver that does zero caching and trusts the kernel/network to be fast, (2) a per-node caching daemon that intercepts before any Service DNAT happens, and (3) a horizontally-scaled cluster-wide authority (CoreDNS) that is the only component that actually knows about Kubernetes API objects. This layering exists because a single global cache (one central CoreDNS fleet with no local tier) puts irreducible network hops and Service-DNAT/conntrack cost on every single lookup — including the 90%+ that are cache hits for the same ten hostnames every pod on a node resolves repeatedly (the node's own kubelet, the Service mesh sidecar, the database, the object store endpoint).

Component interactions and data flow. A pod's /etc/resolv.conf — written by the kubelet from the pod's dnsPolicy — points at a nameserver IP. In a NodeLocal DNSCache setup, that IP is not the CoreDNS ClusterIP directly; it's the link-local address 169.254.20.10 bound by a DaemonSet pod on the same node, using hostNetwork: true. This is the single most important architectural decision in the whole stack: because the NodeLocal DNSCache pod listens on the host network namespace at a link-local address, the kernel routes the query to it without going through the Service virtual IP abstraction at all for the common case. No iptables DNAT, no conntrack entry, no possibility of a cross-node hop. On a cache hit — which for a well-behaved workload is the large majority of traffic — the round trip never leaves the node.

On a cache miss, NodeLocal DNSCache itself becomes a client: it forwards to the kube-dns ClusterIP (still via iptables/IPVS, still creating a conntrack entry, but now only on cache misses instead of on every query, cutting Service-DNAT-path volume by 90%+ in typical fleets) or, in Cilium-eBPF-managed clusters, through the eBPF socket-level redirect that resolves the Service backend without a netfilter hop at all. CoreDNS receives the query, matches it against its plugin chain — kubernetes plugin serves anything under the cluster zone by watching the API server's Service/Endpoint/EndpointSlice objects in-memory (no etcd round trip per query), forward plugin sends anything else upstream to the VPC resolver (AmazonProvidedDNS on AWS, 168.63.129.16 on Azure) — and returns the answer, which NodeLocal DNSCache then caches locally with a TTL bounded by both the upstream TTL and NodeLocal DNSCache's own .override_ttl / negative-cache settings.

Security boundaries. The NodeLocal DNSCache DaemonSet runs privileged enough to bind a link-local address and manipulate the node's own iptables rules for the intercept — it needs NET_ADMIN and hostNetwork, which puts it squarely in the "trusted platform component, not tenant workload" bucket and it should be deployed via the platform team's GitOps pipeline, not left tenant-editable. CoreDNS itself should have a NetworkPolicy restricting ingress to udp/tcp:53 from cluster CIDR only, and a Kyverno/Gatekeeper policy preventing tenant namespaces from deploying their own hostNetwork pods that could spoof 169.254.20.10. In an eBPF/Cilium environment, CiliumNetworkPolicy with toFQDNs lets you enforce that specific pods may only resolve (and then connect to) an explicit allow-list of external hostnames — which is both a security control and a way to bound the fan-out amplification problem, because DNS-aware L7 policy means unauthorized lookups get an eBPF-level deny rather than five wasted upstream queries.

High availability and disaster recovery. CoreDNS runs as a Deployment with PodAntiAffinity spreading replicas across nodes and AZs, fronted by the kube-dns Service so a pod failure is invisible to clients beyond one retried query. Replica count is managed by cluster-proportional-autoscaler scaling on node/core count rather than a fixed number, because a fixed replica count that was correct at 100 nodes becomes a bottleneck at 1,000. NodeLocal DNSCache has no HA story of its own by design — it's a DaemonSet, so it's "HA" in the sense that every node has exactly one, and its failure mode is graceful: if the local cache pod is down, the DaemonSet's iptables rule that would redirect to it is removed as part of the pod's preStop lifecycle hook, and resolution falls through directly to the kube-dns Service, degrading performance but not correctness. Disaster recovery for the whole subsystem is really "how fast does a new CoreDNS pod become Ready and start serving" — which is why readinessProbe against the ready plugin and pre-warmed Corefile caches matter more than any snapshot/restore story; there's no persistent state to restore.

Multi-region and multi-cloud considerations. Each regional cluster runs its own CoreDNS + NodeLocal DNSCache stack; there is no cross-region DNS replication for internal cluster DNS by design — cross-cluster service discovery is handled at a different layer (a service mesh's multi-cluster control plane, or explicit ExternalName Services pointing at a global load balancer). For public-facing names, ExternalDNS running in each cluster reconciles Ingress/Service objects into Route 53 (or the equivalent), with a health-check-driven failover record so a region-down event routes public traffic elsewhere without any change to internal DNS. Multi-cloud clusters typically run a forward plugin split by zone — internal names for the local cloud go to the metadata-service resolver, all VPN-reachable on-prem or other-cloud names get forwarded to a dedicated Resolver rule set (AWS Route 53 Resolver rules, Azure Private DNS Resolver) mapped by domain suffix, so the Corefile becomes the single source of truth for "who is authoritative for what," and that file is exactly what platform engineers review most carefully in PRs.

Evolution at scale. At ~100 nodes, a single-tier CoreDNS-only setup with 2–3 replicas is fine. At ~500 nodes, NodeLocal DNSCache becomes mandatory, not optional, because Service-DNAT/conntrack pressure crosses from "occasional retry" to "measurable tail latency." At ~2,000+ nodes or in any environment doing kube-proxy replacement, Cilium's eBPF-based DNS proxy and FQDN-aware network policy replace the iptables intercept entirely, removing conntrack from the picture for the DNS path specifically (though conntrack is still relevant for other UDP/TCP traffic). Beyond that, the bottleneck shifts from the data path to the control plane: CoreDNS's kubernetes plugin watch load against the API server, and that's when cluster-proportional-autoscaler tuning and API server request-priority-and-fairness (APF) flow-schema configuration for DNS-related watches becomes the next constraint to design around.

4. Solution Design

Design decisions and the alternatives considered.

The core decision is whether to solve the conntrack/latency problem with (a) NodeLocal DNSCache, (b) a full kube-proxy replacement via Cilium eBPF, (c) simply scaling CoreDNS horizontally and hoping the extra replicas absorb the load, or (d) tuning ndots and client-side caching to reduce query volume instead of speeding up the query path.

(c) Horizontal CoreDNS scaling alone is the naive first move and it's insufficient on its own: it reduces per-pod contention for CoreDNS CPU, but it does nothing about the conntrack table on the client's own node, which is where the 5-second bug actually lives. You can have 50 CoreDNS replicas at 2% CPU each and still see conntrack races on a busy node, because the bottleneck isn't CoreDNS's ability to answer — it's the netfilter path getting the query there and back reliably.

(d) ndots/client-side tuning is cheap and should always be done regardless of the other choices: setting dnsConfig.options with ndots: 2 (or restructuring app config to always use FQDNs, avoiding the search-domain fan-out entirely) directly reduces query volume 3–5x for absolute-hostname lookups that don't need it, and is a five-line pod spec change with no infrastructure risk. Its downside is it's per-workload opt-in — you can't force it cluster-wide without breaking apps that rely on unqualified short names, so it doesn't fully solve the platform-wide problem by itself.

(a) NodeLocal DNSCache is the pragmatic, widely-adopted middle path — it's a Kubernetes SIG-sanctioned, well-documented DaemonSet that every major managed Kubernetes offering (EKS, GKE, AKS) supports as an add-on, it requires no CNI change, and it directly attacks the conntrack problem by removing the DNAT hop for cache hits. Its cost is operational surface area: one more DaemonSet to monitor, upgrade, and reason about, and it introduces its own cache-coherency question (stale answers after a Service's Endpoints change, bounded by TTL).

(b) Cilium eBPF DNS proxy / full kube-proxy replacement is the most complete fix — it removes iptables/conntrack from the data path for Service traffic broadly, not just DNS — but it's the highest-blast-radius change: replacing kube-proxy cluster-wide is a CNI-level migration with its own rollout risk (see any service-mesh-migration postmortem for how carefully these need to be staged), and it's not something you reach for purely to fix DNS if you're not already invested in Cilium for other reasons (network policy, observability via Hubble, WireGuard-based node-to-node encryption).

The recommendation used in this design: NodeLocal DNSCache as the default posture for any fleet over ~300 nodes, ndots tuning applied opportunistically per-workload as a further multiplier, and Cilium's eBPF path reserved for organizations already on Cilium for its networking/security benefits, where the DNS improvement is a bonus rather than the primary driver.

Scalability considerations. NodeLocal DNSCache scales linearly with node count by construction (one DaemonSet pod per node, bounded resource footprint per pod — typically 25m CPU / 20Mi memory at rest). CoreDNS scaling is bounded by API server watch cost, not query volume, once NodeLocal DNSCache absorbs the bulk of repeat queries — this is a favorable trade because API server watch load is far more predictable and far cheaper to provision for than raw QPS.

Cost implications. NodeLocal DNSCache costs are near-zero incrementally (small DaemonSet footprint, no additional nodes required). The real cost lever is reduced NAT gateway / cross-AZ data transfer charges: every DNS query that would have crossed an AZ boundary to reach a CoreDNS pod on a different node — and did so repeatedly for the same hostname — now resolves from a local cache, which is a genuinely underappreciated FinOps win on top of the reliability one.

Security implications are covered above (NetworkPolicy scoping, toFQDNs policy, privileged DaemonSet governance).

Performance implications. Cache-hit latency drops from single-digit-to-tens-of-milliseconds (Service DNAT + conntrack + cross-node hop) to sub-millisecond (local socket read). More importantly for tail latency, it removes the variance — the conntrack race that causes the 5-second outlier is structurally impossible for a query that never touches conntrack in the first place.

5. Deep Technical Walkthrough

Internal working and request flow, step by step, for a cache-miss lookup of orders-db.internal.svc.cluster.local from an application pod under ndots:5:

  1. The application calls getaddrinfo("orders-db.internal.svc.cluster.local"). Because the name has 4 dots and ndots:5 says "if the query has fewer than 5 dots, try the search list first," this name is treated as not absolute even though it visually looks like an FQDN — this is the single most misunderstood detail in Kubernetes DNS and the direct cause of most fan-out amplification. The resolver builds a query list: orders-db.internal.svc.cluster.local.<pod-namespace>.svc.cluster.local, then .svc.cluster.local, then .cluster.local, then whatever is in the node's own search domains (often a VPC-internal suffix), and finally the name as absolute.
  2. Each candidate is queried in order over UDP to the resolver in /etc/resolv.conf — the node-local 169.254.20.10 — until one returns a non-error answer or all are exhausted (returning the last error, typically NXDOMAIN).
  3. NodeLocal DNSCache receives the first candidate query. It checks its in-memory cache (bounded LRU, positive and negative entries tracked separately). Miss.
  4. NodeLocal DNSCache forwards to kube-dns ClusterIP. This is the one hop in the whole chain that still touches the Service abstraction: the packet gets DNAT'd by iptables/IPVS to a real CoreDNS pod IP, and a conntrack entry is created to track the UDP "connection" (UDP is connectionless but netfilter still tracks it for the return path DNAT'ing).
  5. CoreDNS's kubernetes plugin checks whether the query falls in cluster.local. — for the first three candidates it does — and looks up the Service orders-db in namespace internal from its in-memory object cache. If found: returns the ClusterIP A/AAAA record with a short TTL (typically 5s, controlled by the kubernetes plugin's ttl directive). If not found (namespace mismatch, e.g. the first candidate's assumed namespace is wrong): returns NXDOMAIN for that specific candidate, which is correct behavior but is exactly what drives the fan-out — 2–3 guaranteed NXDOMAINs before the real answer.
  6. The NXDOMAIN answers travel back through the same conntrack entry (or a new one, since each candidate is a separate UDP datagram and may get a fresh conntrack entry depending on timing) to NodeLocal DNSCache, which negative-caches them with a bounded TTL (protecting against NXDOMAIN storms from repeated bad lookups), and the resolver moves to the next candidate.
  7. Eventually the correct candidate resolves; NodeLocal DNSCache positive-caches it, and hands the answer back to the application's resolver, which returns it from getaddrinfo.

Where the 5-second bug actually comes from: under sustained high QPS, especially right after a burst of new pods/connections (a deploy, a scale-out event, a retry storm), the node's nf_conntrack table fills faster than entries expire. When nf_conntrack_count approaches nf_conntrack_max, new UDP "connections" — including DNS query/response pairs — get silently dropped by the kernel rather than erroring visibly. The application's UDP-based resolver has no way to distinguish "dropped packet" from "still in flight," so it just... waits. glibc's resolver default UDP timeout is 5 seconds before it retries or gives up (this exact constant — 5000ms — is why the failure mode has such a distinctively bimodal p99 signature). If the retry also races the same congested conntrack table, you get the double-hit: 5 seconds, then another 5, compounding into ENOTFOUND after glibc exhausts RES_OPTIONS attempts (default 2).

Failure scenarios and recovery mechanisms:

  • CoreDNS pod OOMKilled under an NXDOMAIN storm (e.g., a misconfigured client hammering a typo'd hostname thousands of times/sec): the cache plugin's negative-caching bounds memory growth, but if disabled or misconfigured, unbounded negative-answer generation can pressure CoreDNS memory. Recovery: kubernetes.io/limit-ranger resource limits plus a Corefile with explicit cache { success 9984 30, denial 9984 5 } bucket sizing.
  • NodeLocal DNSCache pod crash-loops: the DaemonSet's preStop hook removes the iptables intercept rule so traffic fails open to the ClusterIP directly — this is why the DaemonSet's terminationGracePeriodSeconds and the ordering of iptables rule teardown versus process exit matters; get it wrong and you get a window where queries are blackholed to a dead local address instead of falling through.
  • conntrack table exhaustion cluster-wide during a traffic spike: recovery requires either raising net.netfilter.nf_conntrack_max (a node-level sysctl, typically set via a DaemonSet init container or node bootstrap script, not something you can patch live cluster-wide without a rollout) or — the actual fix — reducing the volume of DNAT'd UDP flows via NodeLocal DNSCache so the table never gets close to the ceiling from DNS traffic in the first place.

Performance bottlenecks and scaling behavior: CoreDNS's kubernetes plugin is O(1) per query against its in-memory index once warm, so CoreDNS CPU scales with QPS, not cluster size directly — but watch load (processing Service/Endpoint/EndpointSlice updates from the API server) scales with cluster churn rate (pods being created/destroyed), which is why a cluster running a CI/CD-heavy, high-churn workload profile needs more CoreDNS headroom than a raw node-count comparison would suggest.

6. Production Troubleshooting

Symptoms (recap from Section 2): intermittent 5s-clustered latency spikes, ENOTFOUND/UnknownHostException on lookups that succeed moments later, no obvious resource saturation.

Step-by-step investigation, the way a senior SRE actually runs it:

Step 1 — confirm it's DNS, not the downstream service. Correlate failed request timestamps against CoreDNS and NodeLocal DNSCache metrics before touching application code.

rate(coredns_dns_responses_total{rcode="SERVFAIL"}[1m])
rate(coredns_dns_responses_total{rcode="NXDOMAIN"}[1m])
histogram_quantile(0.99, rate(coredns_dns_request_duration_seconds_bucket[5m]))

If CoreDNS's own p99 is healthy (single-digit ms) but application-observed latency shows the 5s cluster, the bottleneck is between the pod and CoreDNS — point straight at node-level conntrack/iptables, not the CoreDNS Deployment.

Step 2 — check node-level conntrack pressure on the nodes hosting the affected pods.

# on the node (via kubectl debug node/<node> or a privileged DaemonSet exec)
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
# ratio above ~80% is the red flag
conntrack -S | grep -E 'insert_failed|drop'

insert_failed incrementing is the smoking gun — it means the kernel is actively refusing new conntrack entries, which for UDP DNS traffic means silent packet drop, which is precisely the failure mode with no application-visible error until the client-side timeout fires.

Step 3 — confirm NodeLocal DNSCache is actually intercepting.

kubectl -n kube-system get pods -l k8s-app=node-local-dns -o wide
kubectl -n kube-system logs -l k8s-app=node-local-dns --tail=200 | grep -i error
# from an app pod's node, verify the iptables intercept rule exists:
iptables -t nat -L PREROUTING -n | grep 169.254.20.10

If the intercept rule is missing on a node (a stale DaemonSet rollout, a preStop race, or an IPVS-mode misconfiguration where iptables rules aren't the actual mechanism), queries from that node are silently falling through to the direct ClusterIP path and taking the slow, conntrack-heavy route without anyone realizing NodeLocal DNSCache isn't actually helping there.

Step 4 — quantify ndots fan-out amplification.

kubectl exec -it <affected-pod> -- cat /etc/resolv.conf
# look for: options ndots:5
# then, from inside the pod:
kubectl exec -it <affected-pod> -- sh -c 'time getent hosts orders-db.internal.svc.cluster.local'

Combine with a tcpdump on the node during a synthetic reproduction to literally count how many UDP datagrams one getaddrinfo call generates:

tcpdump -i any -n port 53 -c 50

Seeing 4–5 outbound queries for a single application-level DNS call, several ending in NXDOMAIN, confirms the amplification multiplier that's compounding conntrack pressure.

Step 5 — root cause and remediation checklist.

  • Confirm nf_conntrack_max is sized for the node's actual peak flow count (a common EKS/GKE default is too low for high-density nodes — bump it via node bootstrap sysctl, generally to 262144 or higher depending on pod density).
  • Confirm NodeLocal DNSCache is deployed to every node (not just newer node groups — a common gap after a partial rollout or a Karpenter-provisioned node group that missed the DaemonSet's node selector).
  • Set ndots: 2 or use FQDNs with a trailing dot for high-QPS lookups in the affected services' pod specs, cutting fan-out for those specific hot paths.
  • Verify Corefile negative-cache TTLs aren't set to 0 (a surprisingly common copy-pasted misconfiguration that disables the exact protection meant to bound NXDOMAIN-storm-driven query volume).
  • Add a standing alert on nf_conntrack_count / nf_conntrack_max > 0.8 per node — this metric predicts the failure before it happens and should page before customers notice, not after.

7. Hands-on Lab

Goal: reproduce conntrack-driven DNS latency on a local kind cluster, then fix it with NodeLocal DNSCache, observing the before/after with real metrics.

# 1. Create a kind cluster with a constrained conntrack table to make the bug reproducible locally
cat <<'EOF' > kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
EOF
kind create cluster --name dns-lab --config kind-config.yaml

# 2. Artificially lower conntrack max on the worker node to make exhaustion reproducible in minutes, not hours
docker exec dns-lab-worker sysctl -w net.netfilter.nf_conntrack_max=512

# 3. Deploy a small internal Service to resolve against
kubectl create deployment orders-db --image=nginx --replicas=1
kubectl expose deployment orders-db --port=80

# 4. Deploy a load generator pod that resolves the Service name in a tight loop, forcing ndots fan-out
kubectl run dns-load --image=busybox:1.36 --restart=Never -- \
  sh -c 'while true; do getent hosts orders-db.default.svc.cluster.local >/dev/null; done'
kubectl scale --replicas=40 deployment dns-load 2>/dev/null || \
  for i in $(seq 1 40); do kubectl run dns-load-$i --image=busybox:1.36 --restart=Never -- \
    sh -c 'while true; do getent hosts orders-db.default.svc.cluster.local >/dev/null; done'; done

# 5. Watch conntrack fill up on the worker node
watch -n1 'docker exec dns-lab-worker cat /proc/sys/net/netfilter/nf_conntrack_count'

# 6. Confirm degraded resolution time from a fresh pod under load
kubectl run dns-check --rm -it --image=busybox:1.36 --restart=Never -- \
  sh -c 'time getent hosts orders-db.default.svc.cluster.local'

# 7. Now install NodeLocal DNSCache and re-measure
curl -sL https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml -o nodelocaldns.yaml
# substitute __PILLAR__DNS__SERVER__ and __PILLAR__LOCAL__DNS__ per the file's own instructions, then:
kubectl apply -f nodelocaldns.yaml

# 8. Re-run the timing check — resolution should now be consistently sub-10ms, cache hits bypassing conntrack entirely
kubectl run dns-check-2 --rm -it --image=busybox:1.36 --restart=Never -- \
  sh -c 'time getent hosts orders-db.default.svc.cluster.local'

Validation. Compare the time output from step 6 (pre-fix, expect occasional multi-second stalls once conntrack fills) against step 8 (post-fix, expect consistent low-single-digit-millisecond resolution). Cross-check with docker exec dns-lab-worker conntrack -S to confirm insert_failed stopped incrementing once NodeLocal DNSCache is intercepting.

Cleanup:

for i in $(seq 1 40); do kubectl delete pod dns-load-$i --ignore-not-found; done
kubectl delete deployment orders-db
kubectl delete service orders-db
kubectl delete -f nodelocaldns.yaml
kind delete cluster --name dns-lab

8. Production Case Study

Netflix has publicly described DNS as one of the "long tail" reliability problems in their move to Titus/Kubernetes-adjacent infrastructure — their approach leans heavily on client-side caching (their internal service discovery client, Eureka-descended tooling, caches resolved addresses aggressively at the application layer, treating DNS as a bootstrap mechanism rather than a per-request dependency) precisely to avoid putting request-path latency at the mercy of the DNS subsystem at all, regardless of how well-tuned it is.

Uber, in their published networking postmortems, has described conntrack-table exhaustion from DNS-heavy microservice fan-out as a recurring theme in their move to larger, denser Kubernetes clusters, and their public engineering writing on their in-house service mesh explicitly calls out moving service-to-service discovery off of DNS-based mechanisms and onto their mesh control plane's own discovery protocol for exactly this reason — DNS remains for bootstrap and external names, not the hot path.

Google (GKE) ships NodeLocal DNSCache as a one-click cluster add-on specifically because this class of problem was common enough across their managed customer base that it warranted a supported, default-recommendable component rather than leaving every customer to rediscover the conntrack issue independently — their public GKE documentation on the topic reads, almost verbatim, like the troubleshooting checklist in Section 6, which is a strong signal of how common this exact failure mode is industry-wide.

The common thread across all three: none of them treat DNS as "fixed once, forget forever." The mature pattern is defense in depth — reduce query volume at the client (aggressive application-level caching, ndots discipline), remove the conntrack hop where possible (NodeLocal DNSCache or eBPF), and treat the remaining authoritative tier (CoreDNS) as a horizontally-scaled, boring, well-monitored commodity rather than a source of cleverness.

9. Architecture Review

Strengths. The three-tier design (in-pod resolver → node-local cache → cluster authority) correctly puts the cheapest, most local answer path closest to the caller, which is the right default for a read-heavy, highly-repetitive workload pattern (the same twenty hostnames get resolved millions of times a day across a fleet). It degrades gracefully — NodeLocal DNSCache failure falls through rather than blackholing, and CoreDNS failure is masked by replica count and Service-level load balancing. It requires no client-side code changes, which matters enormously for adoption across a multi-tenant platform with teams who can't all coordinate a simultaneous change.

Weaknesses. The reliance on client-side UDP retry semantics (glibc's 5-second timeout) means the failure signature is inherently ugly regardless of how well the rest of the stack is tuned — there's no way to make a dropped UDP packet fail fast from the application's perspective without changing resolver libraries or wrapping DNS calls in application-level timeouts, which most teams don't do. The ndots:5 default, inherited from upstream Kubernetes and rarely revisited, is a platform-wide tax that most application teams don't even know exists until they're debugging exactly this class of incident.

What fails first at 10x scale (from 340 to 3,400 nodes). CoreDNS's kubernetes plugin watch load against the API server becomes the first real constraint — at that density, Service/Endpoint/EndpointSlice churn from routine deploys and autoscaling generates enough watch event volume that CoreDNS's in-memory reconciliation and the API server's ability to serve that many watch streams both need explicit capacity planning (API Priority and Fairness flow-schema tuning, cluster-proportional-autoscaler coefficients revisited, not left at their default 100-nodes-per-replica-ish ratio). Secondarily, nf_conntrack_max sizing that was "good enough" per-node at 340 nodes needs re-validation, because pod density per node and cross-service fan-out both tend to grow with fleet size, not just node count.

What changes at 100 million users / hyperscale. At that scale, per-cluster CoreDNS stops being the right mental model entirely — you're running dozens to hundreds of clusters, and the actual architecture question shifts to cross-cluster and cross-region service discovery (a service mesh's own discovery plane, or a purpose-built discovery service that treats Kubernetes DNS as an implementation detail of a single cluster rather than the platform-wide answer), with Kubernetes DNS relegated to intra-cluster bootstrap only. eBPF-based approaches (Cilium) stop being an optional upgrade and become closer to mandatory, because at that node count the CPU and latency overhead of iptables rule evaluation for any Service traffic, DNS included, is no longer a rounding error.

What I'd redesign. Ship ndots: 2 (or explicit FQDN-with-trailing-dot conventions) as an org-wide default via a PodPresets-equivalent (a mutating admission webhook or Kyverno policy injecting dnsConfig) rather than leaving it as tribal knowledge individual teams discover during an incident — this is the highest-leverage, lowest-risk change available and it's routinely skipped because it requires cross-team coordination rather than a platform-team-only change.

10. Best Practices

Reliability. Run NodeLocal DNSCache on every node without exception — a partial rollout is worse than none, because it creates inconsistent latency profiles that are hard to correlate. Set PodDisruptionBudget on CoreDNS ensuring at least N-1 replicas survive voluntary disruption during node drains.

Scalability. Use cluster-proportional-autoscaler for CoreDNS replica count, not a fixed number; revisit its coefficients (nodesPerReplica, coresPerReplica) at each order-of-magnitude fleet growth milestone rather than assuming defaults hold.

Observability. Instrument all three layers: CoreDNS's own Prometheus plugin (rcode, duration, cache_hits_total), NodeLocal DNSCache's equivalent metrics (it exposes the same CoreDNS-derived metrics on its own port), and node-level conntrack saturation as a first-class SLI, not an afterthought debugged only during incidents.

Security. Scope NetworkPolicy around port 53 tightly; enforce toFQDNs-style egress allow-lists where available; treat the NodeLocal DNSCache DaemonSet's hostNetwork/NET_ADMIN privileges as a platform-governed component, never tenant-deployable.

Cost Optimization. NodeLocal DNSCache's reduction in cross-AZ Service-DNAT traffic is a legitimate, quantifiable data-transfer cost saving — track it, because it's an easy budget-friendly justification for platform investment that doesn't require a reliability incident to motivate.

Performance. Push ndots discipline and application-level DNS result caching wherever request-path latency is sensitive to it; don't rely solely on infrastructure-tier caching to compensate for chatty client-side resolution patterns.

Maintainability. Keep the Corefile under GitOps with mandatory peer review — it's small, rarely changed, and exactly the kind of file where a typo (a missing cache block, an accidentally-removed errors plugin) has cluster-wide blast radius with no immediate symptom until the next incident.

Operational Excellence. Bake conntrack saturation and DNS error-rate alerts into the standard cluster bootstrap/Terraform-adjacent tooling so every new cluster inherits them by default — this class of problem should never require a team to "remember" to add the alert after their first incident.

11. Common Production Mistakes

Leaving ndots:5 untouched cluster-wide and being surprised by fan-out amplification during the first real traffic spike — this is the single most common root contributor and the least often addressed proactively, because it requires understanding glibc resolver internals that most application engineers (reasonably) have never needed to think about.

Treating CoreDNS CPU/memory as the only health signal and missing that the actual bottleneck lives in node-level kernel tables that CoreDNS's own metrics can't see — experienced engineers instrument the path, not just the endpoint.

Partial NodeLocal DNSCache rollouts, especially after adding a new node group (Karpenter-provisioned nodes, a new instance-type node pool) that doesn't match the DaemonSet's node selector or taints/tolerations — silently reverting a subset of the fleet to the slow path with no alert firing.

Setting Corefile cache TTLs to 0 "to always get fresh data," which defeats the entire purpose of the caching tier and reintroduces both the latency and the conntrack pressure the architecture exists to remove — if you need faster propagation of Endpoint changes, tune the kubernetes plugin's own TTL down modestly rather than disabling caching wholesale.

Debugging the symptom instead of the mechanism — restarting pods, increasing application-level HTTP client timeouts, or bumping JVM DNS TTL caching (networkaddress.cache.ttl) as band-aids that mask the underlying conntrack/fan-out problem rather than fixing it, leaving the platform one traffic spike away from recurrence.

12. Interview Preparation

Q: Why does a Kubernetes pod see intermittent DNS failures under load even when CoreDNS shows healthy CPU and no errors? A: The bottleneck typically isn't CoreDNS itself but the node-level nf_conntrack table used by iptables/IPVS to DNAT queries to the kube-dns ClusterIP. Under high query volume — often amplified by ndots:5 search-domain fan-out — conntrack entries can be dropped silently when the table nears capacity, and because DNS uses UDP, the client has no error signal, only a resolver timeout (glibc defaults to 5 seconds), producing the classic 5-second-clustered latency spike with no corresponding CoreDNS-side error metric.

Q: How does NodeLocal DNSCache actually reduce load, mechanistically, not just "it caches"? A: It runs as a DaemonSet bound to a link-local address on each node's host network namespace, so the kernel routes queries to it directly without traversing the Service virtual IP / iptables DNAT / conntrack path at all. Cache hits — the majority of traffic for typical repetitive lookup patterns — never touch conntrack. Only cache misses fall through to the kube-dns Service, cutting DNAT'd query volume by the cache hit ratio, typically 90%+.

Q: What's the actual effect of ndots:5, and when would you change it? A: With ndots:5, any queried name with fewer than 5 dots is tried against the pod's search domains before being tried as absolute, meaning a lookup like orders-db.internal.svc.cluster.local (4 dots) generates multiple candidate queries — often 2–4 guaranteed NXDOMAINs — before resolving. Setting ndots: 2 or using fully-qualified names with a trailing dot for high-QPS external lookups collapses this to a single query, directly reducing both CoreDNS load and conntrack pressure. The trade-off is workloads relying on unqualified short-name resolution against the search list would break, so it's applied per-workload via dnsConfig, not blindly cluster-wide.

Q: How would you design CoreDNS's scaling model for a fleet growing from 300 to 3,000 nodes? A: Use cluster-proportional-autoscaler rather than a fixed replica count, tied to node/core count, and revisit its scaling coefficients at each growth milestone since API server watch load (driven by cluster churn, not just node count) becomes the dominant constraint before raw query-serving capacity does. Pair it with PodAntiAffinity across nodes/AZs for HA and a PodDisruptionBudget to protect availability during node drains.

Q: Compare NodeLocal DNSCache against a full Cilium eBPF kube-proxy replacement for solving DNS latency — when would you choose each? A: NodeLocal DNSCache is a narrowly-scoped, low-risk, SIG-supported add-on that directly targets the DNS conntrack problem with minimal operational surface area — the right default choice for most fleets. A full Cilium eBPF migration removes iptables/conntrack from the Service data path broadly, not just for DNS, which is more complete but carries CNI-level migration risk and should be adopted when you already want Cilium's other capabilities (network policy, WireGuard encryption, Hubble observability), with the DNS latency improvement as a secondary benefit rather than the primary driver.

Q: What metrics and alerts would you put in place to catch this class of failure before customers do? A: Node-level nf_conntrack_count / nf_conntrack_max ratio (alert above ~80%) as the leading indicator; CoreDNS's coredns_dns_responses_total{rcode="SERVFAIL"} rate and p99 coredns_dns_request_duration_seconds as the serving-side signal; and — most importantly, since neither of those alone captures the client-observed failure — application-level DNS resolution latency histograms (via a sidecar or eBPF-based passive DNS observability like Hubble) correlated against conntrack saturation to catch the specific failure signature described in this session before it shows up as a customer-facing incident.

13. Latest Industry Updates

  • CoreDNS and the kubernetes plugin continue converging on EndpointSlices over the legacy Endpoints API for watch efficiency at scale, reducing API server load in large, high-churn clusters — worth auditing your CoreDNS version and Corefile if you're running anything more than ~2 years old, since older defaults may still be Endpoints-based.
  • eBPF-based DNS observability (Cilium Hubble, and standalone eBPF DNS tracers) has matured into a genuinely practical way to get passive, per-query visibility (source pod, destination, rcode, latency) without sidecars or application changes — this closes exactly the observability gap described in Section 6, where CoreDNS-side metrics alone can't see client-observed latency.
  • AWS, GCP, and Azure have all continued investing in managed DNS caching add-ons (EKS's NodeLocal DNSCache add-on being one-click installable via the EKS add-on marketplace, GKE's equivalent) reflecting how common and well-understood this operational pattern has become as a default rather than an advanced tuning exercise.
  • The broader industry shift toward eBPF-based networking (Cilium, Isovalent's continued upstream contributions) is steadily reducing reliance on iptables/conntrack for Service traffic generally, which matters for this topic specifically because it's the long-term structural fix, not just a DNS-specific patch — worth tracking for any platform team planning multi-year CNI roadmaps.
  • Increased attention to ndots and DNS fan-out as a FinOps and reliability topic simultaneously — several platform engineering conference talks and vendor blog posts in the last year have framed excessive DNS query volume as both a latency risk and a measurable cost driver (NAT gateway data processing charges, cross-AZ transfer), giving platform teams a stronger business case to prioritize the ndots and NodeLocal DNSCache work discussed here.

14. Summary & Cheat Sheet

Key concepts: Kubernetes DNS is a three-tier system — in-pod resolver (glibc/musl, driven by /etc/resolv.conf and ndots), node-local cache (NodeLocal DNSCache, intercepting via a link-local address to avoid Service DNAT), and cluster authority (CoreDNS, serving cluster-zone records from an in-memory API watch and forwarding everything else upstream). The infamous 5-second DNS failure is a client-side UDP timeout triggered by silent packet drop from node-level nf_conntrack table exhaustion, amplified by ndots:5 search-domain fan-out multiplying query volume 2–5x per lookup.

Architecture: pod → (link-local intercept) → NodeLocal DNSCache → (cache miss only) → iptables/IPVS DNAT + conntrack → CoreDNS → forward plugin → upstream VPC resolver. Cilium eBPF environments can bypass the DNAT/conntrack hop entirely via socket-level redirect.

Commands:

# conntrack saturation check
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
conntrack -S | grep insert_failed

# confirm NodeLocal DNSCache intercept
iptables -t nat -L PREROUTING -n | grep 169.254.20.10

# CoreDNS health
kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100

# quantify ndots fan-out
tcpdump -i any -n port 53 -c 50

Best practices: roll out NodeLocal DNSCache fleet-wide with no gaps; tune ndots per-workload (or org-wide via admission policy) rather than leaving the default; keep Corefile caching enabled with sane TTLs; scale CoreDNS via cluster-proportional-autoscaler; instrument conntrack saturation as a first-class SLI, not an incident-time discovery.

Design patterns: local-cache-in-front-of-shared-authority (the same pattern shows up in CDN edge caching, database read replicas, and client-side SDK caching — recognize it as a general scaling primitive, not a DNS-specific trick).

Troubleshooting checklist:

  1. Confirm failure is DNS via CoreDNS metrics correlation, not application-code assumption.
  2. Check node-level conntrack saturation and insert_failed counters.
  3. Verify NodeLocal DNSCache is deployed and actively intercepting on the affected nodes.
  4. Measure ndots fan-out multiplier via tcpdump.
  5. Remediate: right-size nf_conntrack_max, close DaemonSet rollout gaps, apply ndots tuning, validate Corefile cache/negative-cache settings.
  6. Add standing alerts on conntrack ratio and SERVFAIL rate so the next occurrence pages before customers notice.

Daily DevOps Mentor is a running series on production-grade cloud-native and AI infrastructure engineering — architecture, trade-offs, debugging, and scaling decisions at the level expected of Principal/Staff engineers operating platforms at enterprise scale.