DNS Failures on Amazon EKS: Fix "no such host", i/o timeout and SERVFAIL

One card per cause. Each shows what you see, the commands that confirm it, the fix, how to keep it from coming back, and the walkthrough that explains the mechanism.

Wrong name, or the Service is in another namespace

no such host NXDOMAIN kubernetes.default resolves

Back to the tree

What you see

$ kubectl logs deploy/checkout -n shop | tail -1
Post "http://payments/charge": dial tcp: lookup payments on 172.20.0.10:53: no such host

$ kubectl debug -it checkout-7d4b9c6f5-x2kqp -n shop --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 -- nslookup payments
Server:         172.20.0.10
Address:        172.20.0.10#53

** server can't find payments: NXDOMAIN

NXDOMAIN is an answer: CoreDNS is up and reachable, and it says the name doesn't exist. A short name is tried with the Pod's own namespace first (payments.shop.svc.cluster.local), then the other search domains, so payments only finds a Service in shop.

Confirm

# Does the Service exist, and in which namespace?
kubectl get service -A --field-selector metadata.name=payments
# The search domains the client Pod uses
kubectl exec deploy/checkout -n shop -- cat /etc/resolv.conf
# The name with its namespace, from the same Pod
kubectl debug -it checkout-7d4b9c6f5-x2kqp -n shop --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 -- nslookup payments.billing.svc.cluster.local

If get service -A finds nothing, the Service doesn't exist under that name: a typo, not deployed yet, or another cluster. If the full name resolves and the call then fails with refused or a timeout, DNS is done: go to the Service not reachable playbook.

Fix

Call the Service with its namespace: payments.billing works through the search list, and payments.billing.svc.cluster.local is the full name. Put it in the app's config.
Fix the typo, or deploy the Service. A Service name resolves as soon as the Service exists; a Pod's own name doesn't resolve unless it sets hostname and subdomain behind a headless Service.
A cross-namespace call also has to pass any Network Policies in the target namespace; that failure is a timeout on the connection, after the name resolved.

Prevent

Use <service>.<namespace> or the full .svc.cluster.local name for anything outside the Pod's own namespace, even when both happen to share one today.
In the deploy pipeline, smoke-test the app's configured hostnames from a Pod in its namespace.

Understand it

CoreDNS Namespaces, labels and selectors

The VPC resolver can't answer a private or on-premises name

only corp names fail NXDOMAIN or SERVFAIL dnsPolicy: Default fails too

Back to the tree

What you see

$ kubectl logs deploy/checkout -n shop | tail -1
dial tcp: lookup orders-db.corp.internal on 172.20.0.10:53: no such host

# Skip CoreDNS: ask the resolver the node uses
$ kubectl run upstream-test -n shop --rm -it --restart=Never --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 \
    --overrides='{"spec":{"dnsPolicy":"Default"}}' -- nslookup orders-db.corp.internal
Server:         10.0.0.2
Address:        10.0.0.2#53

** server can't find orders-db.corp.internal: NXDOMAIN

CoreDNS answers cluster.local names itself and forwards everything else to the resolver in the node's /etc/resolv.conf: on EKS, the VPC's Route 53 Resolver at the VPC's primary IPv4 CIDR base plus two (10.0.0.2 here). When that resolver can't see the zone, the answer is NXDOMAIN; when a forwarder it relies on can't be reached, CoreDNS returns SERVFAIL, which Go reports as server misbehaving.

Confirm

# Private hosted zones associated with the cluster's VPC
aws route53 list-hosted-zones-by-vpc --vpc-id vpc-0abc1234def567890 --vpc-region us-east-1
# Resolver forwarding rules associated with it (on-premises or other-VPC domains)
aws route53resolver list-resolver-rule-associations --filters Name=VPCId,Values=vpc-0abc1234def567890
# Custom DHCP options mean the nodes use other DNS servers
aws ec2 describe-vpcs --vpc-ids vpc-0abc1234def567890 --query 'Vpcs[0].DhcpOptionsId'
aws ec2 describe-vpc-attribute --vpc-id vpc-0abc1234def567890 --attribute enableDnsSupport

If the dnsPolicy: Default Pod fails the same way, CoreDNS isn't the problem; the VPC's resolver is. If it resolves there but not through CoreDNS, look for a custom forward block for that zone in the CoreDNS Corefile.

Fix

Private hosted zone: associate it with the cluster's VPC (aws route53 associate-vpc-with-hosted-zone; a zone in another account needs an authorization from that account first). Private zones also need enableDnsSupport and enableDnsHostnames on the VPC.
On-premises or other-VPC names: a Route 53 Resolver forwarding rule for the domain, through an outbound endpoint, associated with the cluster's VPC. The target DNS servers must be reachable from the endpoint on port 53.
Custom DHCP options: their DNS servers must resolve both public and internal names. Nodes pick up a change within a few hours as they renew their DHCP lease (renew it, or replace the nodes, to go faster). CoreDNS Pods copy the node's resolv.conf when they start: kubectl rollout restart deployment coredns -n kube-system.

Prevent

Manage hosted zone associations and Resolver rule associations in the same infrastructure code as the cluster's VPC.
Run a scheduled lookup from a Pod for one name in each private zone, and alert when it fails.

Understand it

CoreDNS

CoreDNS isn't running: kube-dns has no ready endpoints

coredns 0/2 read: connection refused FailedScheduling

Back to the tree

What you see

$ kubectl logs deploy/checkout -n shop | tail -1
dial tcp: lookup orders.shop.svc.cluster.local on 172.20.0.10:53: read udp 10.0.1.55:41234->172.20.0.10:53: read: connection refused

$ kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-5b8cc885bc-7xk2p   0/1     Pending   0          25m
coredns-5b8cc885bc-m4q9t   0/1     Pending   0          25m

$ kubectl describe pod coredns-5b8cc885bc-7xk2p -n kube-system
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  25m   default-scheduler  0/3 nodes are available: 3 node(s) had untolerated taint(s). preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

The event text is Kubernetes 1.35 and later; earlier versions name the taint, for example untolerated taint {dedicated: batch}. With no ready CoreDNS Pod behind the kube-dns Service, kube-proxy (in its default iptables mode) rejects packets to the Service IP, so lookups fail at once: connection refused in Go, Temporary failure in name resolution from glibc. Every name fails, in every namespace.

Confirm

kubectl get deployment coredns -n kube-system
kubectl get endpointslices -n kube-system -l kubernetes.io/service-name=kube-dns
# Why Pending (Events), or why it crashes (logs)
kubectl describe pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --previous --tail=30
# The add-on's version and health
aws eks describe-addon --cluster-name prod --addon-name coredns --query 'addon.{version:addonVersion,status:status,health:health}'

Pending is a scheduling problem: the event names it, and the Pod stuck in Pending playbook walks through each kind. A Corefile edit that won't parse is logged and ignored by running Pods (the reload plugin keeps the old config); CoreDNS then crashes on its next restart, printing the bad line.

Fix

Every node is tainted (a new node group with dedicated=batch:NoSchedule, for example): keep some untainted nodes for system Pods, or give CoreDNS a matching toleration through the add-on's tolerations configuration value.
Fargate only: create a Fargate profile for kube-system that selects k8s-app=kube-dns, then kubectl rollout restart -n kube-system deployment coredns so the Pods are recreated on Fargate.
Crashing on restart after a Corefile edit: revert it. With the EKS add-on, make Corefile changes through its corefile configuration value; an add-on update can overwrite direct edits to the coredns ConfigMap.

Prevent

Keep at least two CoreDNS replicas on different nodes and zones, and capacity they can schedule on.
Alert when the kube-dns Service has fewer ready endpoints than the Deployment's replicas.

Understand it

CoreDNS EKS add-ons

A Network Policy blocks egress to CoreDNS on port 53

i/o timeout one namespace only policyTypes: Egress

Back to the tree

What you see

$ kubectl logs deploy/checkout -n shop | tail -1
dial tcp: lookup orders.shop.svc.cluster.local on 172.20.0.10:53: read udp 10.0.1.55:41234->172.20.0.10:53: i/o timeout

$ kubectl get networkpolicy -n shop
NAME                  POD-SELECTOR   AGE
default-deny-egress   <none>         2h
allow-to-orders       app=checkout   2h

Once an egress policy selects a Pod, only the traffic some policy allows gets out. Forget DNS and every lookup is dropped without an answer, so the client waits for its timeout. CoreDNS is healthy and Pods in other namespaces resolve fine; only Pods the policy selects fail.

Confirm

# What do the egress rules allow? Look for port 53 to kube-system
kubectl get networkpolicy -n shop -o yaml
# The labels the policies select on
kubectl get pods -n shop -l app=checkout --show-labels
# Policies around CoreDNS itself (an ingress policy there blocks every namespace)
kubectl get networkpolicy -n kube-system
# Cluster-wide policies (VPC CNI 1.21+), which can deny
kubectl get clusternetworkpolicies
# VPC CNI: the resolved rules the node agent enforces
kubectl get policyendpoints -n shop

Kubernetes NetworkPolicies only allow, so look for the missing allow. Also check kubectl get clusternetworkpolicies for an Admin-tier Deny, which no namespace policy can override (or Calico or Cilium deny policies, if you run them). On EKS, the VPC CNI enforces policies only when enableNetworkPolicy is "true" in the add-on's configuration; if you run Calico or Cilium, they enforce them instead.

Fix

Allow egress to CoreDNS: a to entry with namespaceSelector kubernetes.io/metadata.name: kube-system and podSelector k8s-app: kube-dns, on UDP 53 and TCP 53. DNS falls back to TCP for large answers.
Write the rule for port 53 on the CoreDNS Pods, not only for the Service IP: policies match Pods and their ports.
If kube-system has an ingress policy that selects CoreDNS, it must allow every namespace that resolves names.

Prevent

Ship an allow-dns egress policy with every default-deny egress policy, in the same chart or namespace template.
Test from the real client Pod (kubectl debug into it), not a fresh debug Pod: policies select by label, so a debug Pod can be treated differently.

Understand it

Network Policies CoreDNS

The Pod's security group blocks DNS to CoreDNS

i/o timeout SecurityGroupPolicy other Pods resolve

Back to the tree

What you see

$ kubectl logs deploy/orders -n shop | tail -1
dial tcp: lookup payments.billing.svc.cluster.local on 172.20.0.10:53: read udp 10.0.2.31:50112->172.20.0.10:53: i/o timeout
# Only the orders Pods fail; they have their own security group
$ kubectl get securitygrouppolicy orders-sgp -n shop -o jsonpath='{.spec.securityGroups.groupIds}'
["sg-0a1b2c3d4e5f60718"]

$ aws ec2 describe-security-group-rules --filters Name=group-id,Values=sg-0a1b2c3d4e5f60718 \
    --query 'SecurityGroupRules[?IsEgress==`true`].[IpProtocol,FromPort,ToPort,CidrIpv4]' --output text
tcp     5432    5432    10.0.0.0/16
tcp     443     443     0.0.0.0/0

A SecurityGroupPolicy gives these Pods their own branch network interface, so in the default strict mode the Pod's security group filters all their traffic, DNS included. Here someone replaced the default allow-all outbound rule with two specific ones, and port 53 isn't among them. The packets are dropped, so lookups time out.

Confirm

# Outbound rules on the Pod's security group: UDP and TCP 53?
aws ec2 describe-security-group-rules --filters Name=group-id,Values=sg-0a1b2c3d4e5f60718 \
  --query 'SecurityGroupRules[?IsEgress==`true`].[IpProtocol,FromPort,ToPort,ReferencedGroupInfo.GroupId,CidrIpv4]' --output table
# Inbound rules where CoreDNS runs (the node's security group): 53 from the Pod's group?
aws ec2 describe-security-group-rules --filters Name=group-id,Values=sg-05e6f7a8b9c0d1e2f \
  --query 'SecurityGroupRules[?IsEgress==`false`].[IpProtocol,FromPort,ToPort,ReferencedGroupInfo.GroupId,CidrIpv4]' --output table

Two places can drop it: the Pod's group on the way out, and the security group on the CoreDNS Pods' network interface (the node's security group, unless custom networking sets another) on the way in. A Pod without a SecurityGroupPolicy on the same node resolving fine points here.

Fix

Allow outbound UDP and TCP 53 from the Pod's security group to the security group the CoreDNS Pods use.
Allow inbound UDP and TCP 53 on that group from the Pod's security group (sg-0a1b2c3d4e5f60718). Rule changes take effect at once.
Or, if CoreDNS runs on nodes that use the cluster security group (managed node groups do by default), add that group to the SecurityGroupPolicy's groupIds: it allows all traffic between its members. A changed policy only applies to newly created Pods, so restart the Deployment.

Prevent

Start every Pod security group from the rules the EKS guide requires: probes from the node security group, DNS to CoreDNS, then the app's own rules.

Understand it

Security groups for Pods CoreDNS

The VPC resolver's 1024 packets-per-second limit drops lookups

intermittent i/o timeout SERVFAIL linklocal_allowance_exceeded

Back to the tree

What you see

# Cluster names resolve; external names fail now and then, worst at peak
$ kubectl logs -n kube-system coredns-5b8cc885bc-7xk2p --tail=2
[ERROR] plugin/errors: 2 sqs.us-east-1.amazonaws.com. A: read udp 10.0.2.15:43211->10.0.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 sqs.us-east-1.amazonaws.com. AAAA: read udp 10.0.2.15:52004->10.0.0.2:53: i/o timeout

# On the node running that CoreDNS Pod (through Session Manager)
$ ethtool -S ens5 | grep allowance_exceeded
     bw_in_allowance_exceeded: 0
     bw_out_allowance_exceeded: 0
     pps_allowance_exceeded: 0
     conntrack_allowance_exceeded: 0
     linklocal_allowance_exceeded: 48211

The Amazon-provided DNS server (10.0.0.2 here, also 169.254.169.253) sits behind a limit of 1024 packets per second per network interface, shared with the Instance Metadata Service and Amazon Time Sync; packets above it are dropped. CoreDNS's forward times out and answers the Pod SERVFAIL: server misbehaving in Go, Temporary failure in name resolution from glibc. With two replicas, every external lookup in the cluster leaves through at most two network interfaces.

Confirm

# Which nodes host CoreDNS?
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
# On each of those nodes: is the counter climbing while lookups fail?
ethtool -S ens5 | grep linklocal_allowance_exceeded

The counter is cumulative, so read it twice a minute apart. It also counts dropped Instance Metadata Service and Time Sync packets, so read it alongside the CoreDNS upstream timeouts. ens5 is the primary interface on Amazon Linux 2023 (eth0 on Amazon Linux 2); a CoreDNS Pod whose IP sits on a secondary interface uses that one, so check every interface. The CloudWatch agent can publish these ENA counters as metrics.

Fix

Run NodeLocal DNSCache: a DNS cache on every node answers repeats locally and sends misses out through each node's own network interface, so the limit is spread across all of them.
Run more CoreDNS replicas and spread them one per node (the add-on's replicaCount, autoScaling and topologySpreadConstraints configuration values), so more interfaces share the upstream traffic.
Cut the number of queries: external names under ndots:5 try the search list first, and <name>.ec2.internal goes upstream too (card 8).

Prevent

Alert on linklocal_allowance_exceeded growing on any node, not only on application errors.
Load-test DNS with the cluster's real query mix before a known peak.

Understand it

CoreDNS CoreDNS under load (field guide)

CoreDNS is overloaded or OOMKilled and doesn't scale

OOMKilled RESTARTS climbing cluster names slow too

Back to the tree

What you see

$ kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS        AGE
coredns-5b8cc885bc-7xk2p   1/1     Running   14 (3m ago)     9d
coredns-5b8cc885bc-m4q9t   1/1     Running   11 (21m ago)    9d

$ kubectl describe pod coredns-5b8cc885bc-7xk2p -n kube-system | grep -A3 'Last State'
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

The CoreDNS add-on starts with two replicas, and its default manifest sets a 170Mi memory limit. Many more nodes and Pods than it was sized for mean more queries and more cached records. While one replica restarts, the other takes every query; lookups time out, and cluster names fail too, not only external ones.

Confirm

kubectl top pods -n kube-system -l k8s-app=kube-dns
kubectl get deployment coredns -n kube-system \
  -o jsonpath='{.spec.replicas} {.spec.template.spec.containers[0].resources}{"\n"}'
# Is the add-on's autoscaling turned on?
aws eks describe-addon --cluster-name prod --addon-name coredns --query 'addon.configurationValues'

OOMKilled with exit code 137 is the container hitting its memory limit. No restarts, but timeouts only on external names and a climbing linklocal_allowance_exceeded, points at card 6 instead.

Fix

Turn on the add-on's autoscaling: aws eks update-addon --cluster-name prod --addon-name coredns --configuration-values '{"autoScaling":{"enabled":true,"minReplicas":2,"maxReplicas":10}}'. EKS then scales CoreDNS with the number of nodes and CPU cores; it needs a recent enough add-on and platform version.
Raise the memory limit through the add-on's resources configuration value.
Take load off CoreDNS: NodeLocal DNSCache, and fewer search-list queries (card 8).

Prevent

Alert on CoreDNS restarts and on OOMKilled in kube-system.
Revisit CoreDNS sizing whenever the node count grows several-fold, or leave it to the add-on's autoscaling.

Understand it

CoreDNS EKS add-ons CoreDNS under load (field guide)

ndots:5 sends up to 10 queries for one external name

options ndots:5 slow external lookups NXDOMAIN-heavy

Back to the tree

What you see

$ kubectl exec deploy/checkout -n shop -- cat /etc/resolv.conf
search shop.svc.cluster.local svc.cluster.local cluster.local ec2.internal
nameserver 172.20.0.10
options ndots:5

# What the resolver sends for api.stripe.com (2 dots, fewer than 5)
api.stripe.com.shop.svc.cluster.local    NXDOMAIN
api.stripe.com.svc.cluster.local         NXDOMAIN
api.stripe.com.cluster.local             NXDOMAIN
api.stripe.com.ec2.internal              NXDOMAIN   (forwarded to the VPC resolver)
api.stripe.com                           NOERROR
# 5 names, an A and an AAAA query each: up to 10 queries for one answer

A name with fewer dots than ndots is tried with every search domain before it's tried as written. That isn't an outright failure: lookups succeed, slowly, and the extra queries load CoreDNS and the VPC resolver's per-interface limit. At peak, that's what turns into the timeouts of cards 6 and 7.

Confirm

# The resolver config the kubelet wrote for the Pod
kubectl exec deploy/checkout -n shop -- cat /etc/resolv.conf
# Every query the search list sends, from the same Pod
kubectl debug -it checkout-7d4b9c6f5-x2kqp -n shop --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 -- dig +search +showsearch api.stripe.com
# CoreDNS's answers by response code: NXDOMAIN far ahead of NOERROR is the search list
kubectl port-forward -n kube-system deploy/coredns 9153:9153 &
curl -s localhost:9153/metrics | grep '^coredns_dns_responses_total'

ndots:5 and the cluster search domains come from the default dnsPolicy: ClusterFirst; the last search domain is the VPC's DHCP domain (ec2.internal in us-east-1, <region>.compute.internal elsewhere).

Fix

Lower ndots for the Pod: dnsConfig: {options: [{name: ndots, value: "2"}]}. Names with two or more dots are then tried as written first; payments.billing (one dot) still uses the search list.
Or write external names fully qualified, with a trailing dot (api.stripe.com.): one query. Test it first; some HTTP clients and servers reject a trailing dot in the Host header or TLS name.
Cache closer to the app: NodeLocal DNSCache, or the language runtime's own DNS cache.

Prevent

Set dnsConfig with a lower ndots in the base chart for services that call many external endpoints.

Understand it

CoreDNS

DNS Failures on Amazon EKS

1Start here

Run this first

2Decision tree

3Causes and fixes

Wrong name, or the Service is in another namespace

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

The VPC resolver can't answer a private or on-premises name

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

CoreDNS isn't running: kube-dns has no ready endpoints

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

A Network Policy blocks egress to CoreDNS on port 53

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

The Pod's security group blocks DNS to CoreDNS

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

The VPC resolver's 1024 packets-per-second limit drops lookups

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

CoreDNS is overloaded or OOMKilled and doesn't scale

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

ndots:5 sends up to 10 queries for one external name

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

4Diagnose it

Related

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it