Your compute scaled out — now DNS is timing out: taming the CoreDNS bottleneck under load

Your load test ramps, Karpenter and the HPA do their job, and within ninety seconds you have three times the pods you started with. Compute looks great. Then the application logs start bleeding:

dial tcp: lookup payments.prod.svc.cluster.local on 10.100.0.10:53:
  read udp 10.0.14.2:41500->10.100.0.10:53: i/o timeout

Your microservices can't find each other. Requests that normally take single-digit milliseconds hang for a flat five seconds and then succeed. The two CoreDNS pods are pegged at 100% CPU. Nothing is wrong with your code, your Services, or your network policy — you've just discovered that standard Linux DNS resolution, at concurrency, will happily DDoS your own cluster's service registry. Here's exactly how the storm forms, and the three defenses that stop it.

The anatomy of a DNS storm: how one lookup becomes eight

Start where the kernel starts. When a pod boots, the kubelet writes it an /etc/resolv.conf that looks like this (namespace prod shown):

nameserver 10.100.0.10
search prod.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Two lines do all the damage. The search list gives three suffixes to try. And ndots:5 tells the stub resolver: if a name has fewer than 5 dots, treat it as a relative name and walk the search list first, before trying the name as given. That threshold of five exists so that in-cluster shorthands like payments or payments.prod resolve without a fully-qualified suffix — a genuine convenience. The trap is that it applies to every name, including the external ones.

Count the dots in api.stripe.com: two. Two is less than five, so the resolver does not try it as-is first. It appends each search domain in order and queries those, and only tries the bare name once the search list is exhausted:

Now multiply. That fan-out happens for every external hostname, from every pod, on every connection where the resolver cache has expired. A service that opens connections to api.stripe.com, sqs.us-east-1.amazonaws.com, and a database endpoint is generating dozens of DNS queries per request cycle, most of them destined to fail with NXDOMAIN. Scale from 40 pods to 400 during a surge and you have raised CoreDNS's query rate by a factor of eight on top of the 10× pod growth. That is why the graph looks like a wall: DNS QPS goes vertical while your actual traffic only tripled.

Why the timeout is exactly five seconds — and why CoreDNS pins at 100%

The amplification explains the load. It doesn't explain the flat five-second hang, and that detail is the tell that separates a DNS storm from ordinary slowness. The five seconds come from the kernel, not from CoreDNS.

DNS is UDP. UDP is connectionless, but kube-proxy in iptables mode still creates a conntrack entry for every DNS packet so it can reverse the DNAT that rewrites the kube-dns Service ClusterIP (10.100.0.10) to a real CoreDNS pod IP. Here's the race: glibc fires the A and AAAA queries at nearly the same instant, from the same source port. Two packets with an identical 5-tuple arrive at the conntrack layer before either has an entry, both try to insert, and the kernel drops one as a duplicate. The application never gets that answer, so it sits on its default 5-second resolver timeout before retrying. This is a documented kernel-level conntrack race, not a Kubernetes bug — two of its three variants were only fixed in Linux 5.1.

Meanwhile CoreDNS is genuinely overworked, and there's a hard ceiling behind it that many teams never learn about until it bites. CoreDNS forwards anything it can't answer locally to the Amazon-provided VPC resolver (the “.2” address). AWS enforces a limit of 1,024 packets per second per network interface to that resolver. The two CoreDNS replicas EKS ships by default sit on two ENIs, so your entire cluster's external DNS is capped at roughly 2,048 packets/second to the VPC resolver — and the 8× amplification means you hit that cap with surprisingly little real traffic. Past the cap, packets are silently dropped, which feeds back into the 5-second-timeout loop. CoreDNS pins at 100% CPU trying to shovel the amplified load, and the VPC resolver quietly throttles what gets through.

Defense 1: put a DNS cache on every node with NodeLocal DNSCache

The single highest-leverage change is to stop sending DNS across the node boundary at all. NodeLocal DNSCache runs CoreDNS in cache mode as a DaemonSet — one pod per node in kube-system — listening on the link-local address 169.254.20.10. Pods on that node resolve against the local cache instead of the cluster-wide kube-dns Service. Two things change, and both matter.

It takes conntrack out of the path. Because the cache lives on the node's own loopback-scoped link-local IP, queries to it never traverse the kube-proxy DNAT rules and are never connection-tracked. The parallel-A/AAAA race that produced the 5-second stalls simply cannot happen for a cache hit. And with the vast majority of queries served from the local cache, the number of packets that ever reach conntrack collapses.

It converts flaky UDP into persistent TCP upstream. On a cache miss, NodeLocal forwards to CoreDNS — but it upgrades that connection from UDP to TCP. Per the Kubernetes docs, TCP conntrack entries are removed on connection close, whereas UDP entries have to time out (the default nf_conntrack_udp_timeout is 30 seconds), so TCP both avoids the race and stops the conntrack table from filling with stale UDP entries. Critically, the cache still listens on UDP toward the pods, so applications need no changes at all.

On EKS the manifest is templated with __PILLAR__ placeholders you fill in with sed before applying: the link-local IP, the cluster domain, and your kube-dns Service ClusterIP. One subtlety worth knowing: in kube-proxy iptables mode the node-local pod listens on both 169.254.20.10 and the kube-dns Service IP, so existing pods that still point at the ClusterIP transparently get the local cache. In IPVS mode it listens only on the link-local address, so pods must be pointed at 169.254.20.10 explicitly.

# Render the upstream manifest for EKS (kube-proxy iptables mode).
# kubedns = your kube-dns Service ClusterIP: 10.100.0.10 OR 172.20.0.10
kubedns=$(kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}')
domain=cluster.local
localdns=169.254.20.10

curl -sO https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml

# iptables mode: keep listening on BOTH the ClusterIP and the link-local IP
sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; \
        s/__PILLAR__DNS__DOMAIN__/$domain/g; \
        s/__PILLAR__DNS__SERVER__/$kubedns/g" nodelocaldns.yaml

kubectl apply -f nodelocaldns.yaml
kubectl -n kube-system rollout status daemonset/node-local-dns

The rendered Corefile for the DaemonSet is where the UDP→TCP upgrade lives — note force_tcp on the upstream forwarders and the cache blocks per zone:

# node-local-dns Corefile (essentials)
cluster.local:53 {
    errors
    cache {
        success  9984 30
        denial   9984 5
    }
    forward . 10.100.0.10 {      # kube-dns ClusterIP
        force_tcp                # UDP -> persistent TCP upstream
    }
    prometheus :9253
    health 169.254.20.10:8080
}
. :53 {                          # everything else, incl. external names
    errors
    cache 30
    forward . /etc/resolv.conf {
        force_tcp
    }
    prometheus :9253
}

Defense 2: size CoreDNS to the cluster, not to a CPU graph

NodeLocal DNSCache slashes the query volume that reaches CoreDNS, but the central tier still has to be big enough for cache misses and the cold-start stampede at the beginning of a surge. The instinct is to slap an HPA on CoreDNS and scale on CPU. Don't. CoreDNS CPU is spiky and reactive scaling is always a step behind the spike — by the time the HPA reacts, the timeouts have already fired. You want capacity provisioned ahead of load, proportional to how big the cluster is.

That's exactly what the cluster-proportional-autoscaler (CPA) does. It runs as its own Deployment, watches the number of schedulable nodes and cores, and resizes the CoreDNS Deployment on a simple linear formula. No metrics-server, no CPU thresholds — if the cluster is big, CoreDNS is big.

The CPA is configured with a single ConfigMap. AWS's own guidance is to add a replica for every 256 cores or 16 nodes, whichever triggers first — that's coresPerReplica: 256 and nodesPerReplica: 16 in linear mode:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-autoscaler
  namespace: kube-system
data:
  linear: |-
    {
      "coresPerReplica": 256,
      "nodesPerReplica": 16,
      "min": 2,
      "max": 20,
      "preventSinglePointFailure": true,
      "includeUnschedulableNodes": true
    }

The controller evaluates replicas = max(ceil(cores / 256), ceil(nodes / 16)), then clamps to [min, max]. preventSinglePointFailure: true guarantees at least two replicas whenever the cluster has more than one node, and includeUnschedulableNodes: true counts cordoned/draining nodes so a rollout doesn't briefly under-provision DNS. Deploy the controller itself pointed at the CoreDNS Deployment:

# cluster-proportional-autoscaler controller args (excerpt)
    - --namespace=kube-system
    - --target=deployment/coredns          # scale the CoreDNS Deployment
    - --configmap=coredns-autoscaler       # the ConfigMap above
    - --logtostderr=true
    - --v=2

Defense 3: stop the amplification at the source

The platform defenses make CoreDNS survivable. The application-level fixes make the storm smaller in the first place — and they're free. Start with the one that pays back the most.

Set ndots:2 for external-heavy workloads

If a workload mostly talks to external endpoints, override ndots down from 5. AWS recommends 2: high enough that in-cluster short names like svc.namespace still resolve, low enough that api.stripe.com (two dots) is tried as an absolute name first, skipping the three wasted search-domain queries. Set it per-pod with dnsConfig:

apiVersion: v1
kind: Pod
metadata:
  name: checkout
spec:
  dnsPolicy: "ClusterFirst"       # keep cluster DNS for in-cluster names
  dnsConfig:
    options:
      - name: ndots
        value: "2"                # api.stripe.com is tried as-is first
      - name: single-request-reopen  # A and AAAA on separate sockets:
      - name: attempts                # sidesteps the conntrack race
        value: "2"

single-request-reopen is a quiet gem: it forces glibc to use a fresh socket for the AAAA query, so the two lookups no longer share a 5-tuple and the conntrack insertion race can't drop a packet. It's a per-pod band-aid for the 5-second stalls when you can't roll out NodeLocal DNSCache everywhere yet.

Use fully-qualified names with a trailing dot

The most surgical fix costs one character. A trailing dot marks a name as absolute, so the resolver skips the search list entirely — api.stripe.com. is one query pair, not four. Teams have measured pod DNS time dropping by up to 70% from this change alone, and at 10,000 pods it's the difference between millisecond lookups and multi-second stalls. Where you control the hostnames (config, connection strings, service meshes), prefer the trailing dot.

Complete the search path server-side with autopath

If you can't touch every workload's ndots, move the intelligence into CoreDNS. The autopath plugin makes CoreDNS walk the client's search path on the server: on the first query it follows the search chain, and when it finds a real answer it returns a CNAME pointing there — so the client stops after one query instead of walking all four. It trades CPU and memory on CoreDNS (it needs the Kubernetes pod cache to map client IP to namespace) for far fewer round-trips:

.:53 {
    errors
    health
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods verified          # required for autopath's IP -> namespace map
        fallthrough in-addr.arpa ip6.arpa
    }
    autopath @kubernetes       # complete the search path server-side
    cache 30
    forward . /etc/resolv.conf {
        force_tcp
        prefer_udp
    }
    prometheus :9153
    loop
    reload
    loadbalance
}

Disable AAAA where you'll never use IPv6

In an IPv4-only cluster, every AAAA query is pure waste — it's the second factor of two in the eight. You can't easily stop glibc from asking, but you can stop CoreDNS from forwarding pointless AAAA lookups upstream by returning an immediate empty answer for them, which keeps those queries off the 1,024-pps VPC-resolver budget. Combined with ndots:2, this can halve the residual external query load.

Pitfalls that bite in production