Service Not Reachable on Amazon EKS: Fix "connection refused" and Timeouts Calling a Service

One card per cause. Each shows what you see, the commands that confirm it, the fix, how to keep it from coming back, and the walkthrough that explains the mechanism.

The Service's selector matches no Pods

Endpoints: (empty) ENDPOINTS <unset> Connection refused

Back to the tree

What you see

$ kubectl describe service orders -n shop
# (trimmed to the lines that matter)
Name:                     orders
Namespace:                shop
Selector:                 app=orders,tier=backend
Type:                     ClusterIP
IP:                       172.20.45.12
Port:                     http  80/TCP
TargetPort:               8080/TCP
Endpoints:
Session Affinity:         None

$ kubectl get endpointslices -n shop -l kubernetes.io/service-name=orders
NAME           ADDRESSTYPE   PORTS     ENDPOINTS   AGE
orders-5xq8k   IPv4          <unset>   <unset>     2d

The Endpoints: line lists only ready Pod IPs, and here it's empty (older kubectl prints <none>). The EndpointSlice's ENDPOINTS <unset> says more: no Pod matched at all, ready or not. With nothing to send to, kube-proxy rejects connections to the ClusterIP, so callers get refused at once.

Confirm

# The Service's selector
kubectl get service orders -n shop -o jsonpath='{.spec.selector}{"\n"}'
# Does any Pod match it? Use the selector exactly as printed
kubectl get pods -n shop -l app=orders,tier=backend
# Which labels do the Pods really have?
kubectl get pods -n shop --show-labels

No resources found from the second command settles it. A selector is an AND: a Pod needs every key with the same value. It also only looks in the Service's own namespace.

Fix

Labels were renamed (the Deployment moved to app.kubernetes.io/name: orders, the Service still says app: orders): change the Service's spec.selector to labels the Pod template really has. Don't label running Pods by hand; their replacements won't have it.
An extra key (here tier: backend, which the Pods don't carry): remove it from the selector, or add it to the Deployment's spec.template.metadata.labels.
No Pods at all (scaled to 0, or every Pod is Pending): fix the workload first. A Pending Pod has no IP, so it can't be an endpoint; see the Pod stuck in Pending playbook.

Prevent

Keep the Service and the Deployment in one chart or kustomization, with the selector and the Pod labels set from the same value.
After each rollout, check that the Service has ready endpoints, or alert when one has none.

Understand it

Namespaces, labels and selectors Services and kube-proxy

Pods match, but none of them is Ready

0/1 Running Readiness probe failed

Back to the tree

What you see

$ kubectl get pods -n shop -l app=orders
NAME                     READY   STATUS    RESTARTS   AGE
orders-6d8f9c7b4-7tmxq   0/1     Running   0          12m
orders-6d8f9c7b4-q2w9z   0/1     Running   0          12m

$ kubectl describe pod orders-6d8f9c7b4-7tmxq -n shop
Events:
  Type     Reason     Age                From     Message
  ----     ------     ----               ----     -------
  Warning  Unhealthy  2m (x31 over 12m)  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 503

# The EndpointSlice still lists both Pods ...
$ kubectl get endpointslices -n shop -l kubernetes.io/service-name=orders
NAME           ADDRESSTYPE   PORTS   ENDPOINTS             AGE
orders-5xq8k   IPv4          8080    10.0.2.31,10.0.3.17   2d
# ... but describe service shows an empty Endpoints: line, because neither is ready

A Pod that isn't Ready stays in the EndpointSlice marked ready: false, and kube-proxy sends it no traffic. When no endpoint is ready, kube-proxy rejects connections to the ClusterIP, so the client sees refused, the same as an empty selector.

Confirm

# Ready condition of every endpoint
kubectl get endpointslices -n shop -l kubernetes.io/service-name=orders \
  -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]} ready={.conditions.ready}{"\n"}{end}'
# What the probe calls
kubectl get pod orders-6d8f9c7b4-7tmxq -n shop -o jsonpath='{.spec.containers[0].readinessProbe}{"\n"}'
# What it got back: the Unhealthy events
kubectl describe pod orders-6d8f9c7b4-7tmxq -n shop

The event text names the failure: statuscode: 503 (the app answered no), connect: connection refused (nothing listening on that address and port, see card 4), or context deadline exceeded (slower than timeoutSeconds, 1 second by default).

Fix

The app really isn't ready, often because a dependency such as its database is down and /ready returns 503: fix the dependency, and the Pods turn Ready without a restart.
The probe is wrong: path, port or scheme don't match the app, or timeoutSeconds is too short for the endpoint. Fix the probe in the Pod template.
Pods with their own security group: kubelet probes come from the node, so the Pod's security group must allow the node's security group on the probe port (card 6).

Prevent

Make readiness answer "can this Pod serve?", not "is every downstream up?". If a shared database blip fails readiness on every replica at once, the Service is left with no endpoints at all.
Alert on Deployments with unavailable replicas, not only on restarts: a failing readiness probe never restarts a container.

Understand it

Liveness, readiness and startup probes Services and kube-proxy

targetPort isn't the port the app listens on

TargetPort: 80/TCP app listens on 8080 Connection refused

Back to the tree

What you see

$ kubectl describe service orders -n shop | grep -E 'Port|Endpoints'
Port:                     http  80/TCP
TargetPort:               80/TCP
Endpoints:                10.0.2.31:80,10.0.3.17:80

$ kubectl logs orders-6d8f9c7b4-7tmxq -n shop | head -1
listening on 0.0.0.0:8080

The endpoints are listed, so the selector matches and the Pods are Ready. kube-proxy sends the traffic to port 80 on each Pod, nothing listens there, and the Pod's kernel answers with a reset: refused.

Confirm

# Service port -> targetPort
kubectl get service orders -n shop -o jsonpath='{range .spec.ports[*]}{.port} -> {.targetPort}{"\n"}{end}'
# The ports the container declares, by name and number
kubectl get pod orders-6d8f9c7b4-7tmxq -n shop -o jsonpath='{.spec.containers[*].ports}{"\n"}'
# Skip the Service: from a debug Pod, call one Pod IP on each port
curl -sS --max-time 3 http://10.0.2.31:80/
curl -sS --max-time 3 http://10.0.2.31:8080/

Refused on 80 and an answer on 8080 settles it. containerPort is informational: declaring a port doesn't make the app listen on it, so check the app's own config or logs.

Fix

Set the Service's targetPort to the port the process listens on (8080). Clients keep calling the Service port (80). If you leave targetPort out, it defaults to the same value as port, which is how this bug usually starts.
Better: name the container port (name: http, containerPort: 8080) and set targetPort: http, so the Service follows the Pod when the number changes. The name must exist in the Pod's container ports, or that Pod isn't an endpoint for this port.

Prevent

Name container ports and refer to them by name in Services and probes.
In the deploy pipeline, smoke-test through the Service name, not only the Pod.

Understand it

Services and kube-proxy

The app listens on 127.0.0.1, not 0.0.0.0

127.0.0.1:8080 port-forward works Connection refused

Back to the tree

What you see

# From a debug Pod: refused on the Pod IP itself
$ curl -sS --max-time 3 http://10.0.2.31:8080/healthz
curl: (7) Failed to connect to 10.0.2.31:8080 after 1 ms: Could not connect to server
# Inside the Pod's own network namespace, it answers
$ kubectl exec orders-6d8f9c7b4-7tmxq -n shop -- wget -qO- http://127.0.0.1:8080/healthz
ok

$ kubectl debug -it orders-6d8f9c7b4-7tmxq -n shop --image=nicolaka/netshoot -- ss -ltn
State   Recv-Q  Send-Q   Local Address:Port    Peer Address:Port
LISTEN  0       4096         127.0.0.1:8080         0.0.0.0:*

Bound to 127.0.0.1, the app only accepts connections from inside its own Pod, so anything arriving on the Pod IP is refused. kubectl port-forward usually still works, because it connects from inside the Pod's network namespace. That's what makes this one easy to miss.

Confirm

# Which address does the process listen on? An ephemeral container shares the Pod's network
kubectl debug -it orders-6d8f9c7b4-7tmxq -n shop --image=nicolaka/netshoot -- ss -ltn
# 0.0.0.0:8080, *:8080 or [::]:8080 = every interface; 127.0.0.1:8080 = loopback only

With a readiness probe on that port, you'd see card 2 instead: the kubelet probes the Pod IP, so the probe fails with dial tcp 10.0.2.31:8080: connect: connection refused and the Pod never turns Ready. You land on this card when there's no readiness probe, it checks another port, or it's an exec probe (that runs inside the container, so loopback works).

Fix

Make the app bind every interface: 0.0.0.0 (or :: for IPv6 too). Defaults that bind loopback only include the Flask dev server (--host=0.0.0.0), Uvicorn (--host 0.0.0.0) and Rails in development (-b 0.0.0.0).
Check environment-driven settings too: a HOST=localhost or BIND_ADDRESS=127.0.0.1 copied from a laptop's .env file does the same thing.

Prevent

Give every Service-backed container a readiness probe on its serving port. A loopback-only app then never turns Ready, which shows up in kubectl get pods instead of as refused calls.
Default the bind address to 0.0.0.0 in the image, and override it only for local runs.

Understand it

Services and kube-proxy Liveness, readiness and startup probes

A Network Policy drops the traffic

Connection timed out Pod IP times out too Verdict: DENY

Back to the tree

What you see

$ kubectl exec deploy/checkout -n shop -- curl -sS --max-time 5 http://orders/
curl: (28) Connection timed out after 5002 milliseconds
# The Pod IP times out too, so the Service itself isn't the problem
$ kubectl exec deploy/checkout -n shop -- curl -sS --max-time 5 http://10.0.2.31:8080/
curl: (28) Connection timed out after 5001 milliseconds

# On the server's node, with VPC CNI policy event logs turned on
# /var/log/aws-routed-eni/network-policy-agent.log
{"level":"info","timestamp":"2026-11-02T09:14:07.512Z","logger":"ebpf-client","msg":"Flow Info: ","Src IP":"10.0.1.55","Src Port":41234,"Dest IP":"10.0.2.31","Dest Port":8080,"Proto":"TCP","Verdict":"DENY"}

A Network Policy drops packets without answering, so the client waits until its timeout. If the client's namespace has an egress policy that forgets DNS, even the lookup hangs: curl: (28) Resolving timed out after 5000 milliseconds.

Confirm

# Policies in the server's namespace (and the client's, for egress)
kubectl get networkpolicy -n shop
kubectl describe networkpolicy -n shop
# Which policies select the server Pods? Compare their podSelector with these labels
kubectl get pods -n shop -l app=orders --show-labels
# VPC CNI: the resolved rules the node agent enforces
kubectl get policyendpoints -n shop

Once any policy selects a Pod for ingress, only traffic that some policy allows gets in. Policies only allow; there's no deny rule to find, so look for the missing allow. On EKS, the VPC CNI enforces policies only when enableNetworkPolicy is "true" in the add-on's configuration; if you run Calico or Cilium, they enforce them instead.

Fix

Allow the client in: an ingress rule on the server Pods with a podSelector for the client in the same namespace, or a namespaceSelector (for example kubernetes.io/metadata.name: checkout) for another namespace.
Use the Pod's port in the rule (port: 8080, the targetPort), not the Service port 80: the policy sees the traffic after it has been translated to the Pod's IP and port.
If the client's namespace denies egress by default, also allow egress to the server Pods, and to CoreDNS in kube-system on UDP and TCP 53.

Prevent

Ship the allow policies with the app, in the same chart, so a default-deny namespace never gets the app without them.
Test from the real client Pod (kubectl exec deploy/checkout …), not a debug Pod: policies select by label, so a debug Pod can be blocked or allowed when the client isn't.

Understand it

Network Policies

The Pod's security group doesn't allow the client

Connection timed out SecurityGroupPolicy aws-k8s-branch-eni

Back to the tree

What you see

$ kubectl exec deploy/checkout -n shop -- curl -sS --max-time 5 http://orders/
curl: (28) Connection timed out after 5001 milliseconds
# The server Pods have their own branch network interface
$ kubectl get pod orders-6d8f9c7b4-7tmxq -n shop -o jsonpath='{.metadata.annotations.vpc\.amazonaws\.com/pod-eni}'
[{"eniId":"eni-0c1d2e3f4a5b6c7d8","ifAddress":"02:5e:1a:7b:3c:41","privateIp":"10.0.2.31","vlanId":1,"subnetCidr":"10.0.2.0/24"}]

$ aws ec2 describe-network-interfaces --network-interface-ids eni-0c1d2e3f4a5b6c7d8 \
    --query 'NetworkInterfaces[0].{Desc:Description,Groups:Groups[].GroupId}'
{
    "Desc": "aws-k8s-branch-eni",
    "Groups": [
        "sg-0a1b2c3d4e5f60718"
    ]
}

A SecurityGroupPolicy gave these Pods their own branch network interface, so the Pod's security group decides what reaches them, not the node's. A security group drops what it doesn't allow, so the client times out.

Confirm

# Which policies apply in the namespace, and which security groups they name
kubectl get securitygrouppolicies -n shop -o yaml
# Inbound rules on the Pod's security group: is the client's group allowed on 8080?
aws ec2 describe-security-group-rules --filters Name=group-id,Values=sg-0a1b2c3d4e5f60718 \
  --query 'SecurityGroupRules[?IsEgress==`false`].[IpProtocol,FromPort,ToPort,ReferencedGroupInfo.GroupId,CidrIpv4]' --output table
# The VPC CNI's enforcing mode (empty output means the default, strict)
kubectl get daemonset aws-node -n kube-system \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="aws-node")].env[?(@.name=="POD_SECURITY_GROUP_ENFORCING_MODE")].value}{"\n"}'

A client Pod without its own security group sends from an IP on the node's network interface, so a rule that references the node's security group lets it in. A client with its own SecurityGroupPolicy needs a rule for its group instead. On managed node groups without a custom launch template, the node's interface carries the EKS cluster security group; with custom networking, the ENIConfig sets it.

Fix

Add an inbound rule to the Pod's security group for each client on the container port, for example TCP 8080 from the node security group sg-05e6f7a8b9c0d1e2f. Rule changes take effect at once.
Allow the node security group on every probe port too, or kubelet probes fail and the Pods drop out of the Service (card 2). With probes in strict mode, also set DISABLE_TCP_EARLY_DEMUX=true on the VPC CNI's aws-vpc-cni-init container, as the EKS guide describes.
Allow outbound UDP and TCP 53 to CoreDNS's security group, and let CoreDNS's group accept it, or the Pod's own lookups fail.
Changed the SecurityGroupPolicy itself? It applies only to newly scheduled Pods: restart the Deployment.

Prevent

Start every Pod security group from the rules the EKS guide requires: probes from the node security group, DNS to CoreDNS, and one inbound rule per caller.
Know your mode: in strict (the default) the Pod's security group filters all its traffic, even from Pods on the same node. standard doesn't apply it between Pods on the same node.

Understand it

Security groups for Pods

kube-proxy isn't programming the client's node

Pod IP works ClusterIP times out

Back to the tree

What you see

# From the same client Pod: the Pod IP answers, the ClusterIP times out
$ curl -sS --max-time 5 http://10.0.2.31:8080/healthz
ok
$ curl -sS --max-time 5 http://172.20.45.12/healthz
curl: (28) Connection timed out after 5001 milliseconds

$ kubectl get daemonset kube-proxy -n kube-system
NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-proxy   3         3         2       3            2           <none>          41d

A ClusterIP isn't on any machine; kube-proxy's rules on the client's node translate it to a Pod IP. Without those rules, packets head for an address nobody owns and are dropped. Often only clients on one node are affected.

Confirm

# Is a kube-proxy Pod running and Ready on the client's node?
kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide
kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=50
# The add-on's version and health
aws eks describe-addon --cluster-name prod --addon-name kube-proxy --query 'addon.{version:addonVersion,status:status,health:health}'

Rare, so rule out cards 5 and 6 first. Rules kube-proxy already wrote survive when it stops, so old Services often keep working; new or changed Services and endpoints are what break. On EKS kube-proxy runs in iptables mode unless you chose another mode. EKS Auto Mode and Fargate nodes don't run it as a DaemonSet Pod you can inspect.

Fix

Get kube-proxy Running and Ready on every node: read the logs of the failing Pod, and update the EKS add-on to the version recommended for your cluster's Kubernetes version.
If it broke right after a mode change (for example to IPVS), check the node prerequisites for that mode, or go back to iptables.

Prevent

Run kube-proxy as an EKS add-on and update it with every cluster upgrade.

Understand it

Services and kube-proxy EKS add-ons

The Service is in another namespace

Could not resolve host no such host

Back to the tree

What you see

# checkout runs in namespace shop and calls http://payments/charge
$ kubectl logs deploy/checkout -n shop | tail -1
Post "http://payments/charge": dial tcp: lookup payments on 172.20.0.10:53: no such host
$ kubectl exec deploy/checkout -n shop -- curl -sS http://payments/charge
curl: (6) Could not resolve host: payments

$ kubectl get service -A --field-selector metadata.name=payments
NAMESPACE   NAME       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
billing     payments   ClusterIP   172.20.88.40   <none>        80/TCP    9d

A short name is looked up with the client's own namespace first (payments.shop.svc.cluster.local), then the other search domains. The Service lives in billing, so every try comes back NXDOMAIN, and the call fails before any connection.

Confirm

# The search domains the client Pod uses
kubectl exec deploy/checkout -n shop -- cat /etc/resolv.conf
# search shop.svc.cluster.local svc.cluster.local cluster.local ec2.internal
# (outside us-east-1 the last entry is <region>.compute.internal)
# The name with its namespace resolves
kubectl run dnstest -n shop --rm -it --restart=Never --image=busybox:1.36 -- nslookup payments.billing.svc.cluster.local

If the full name doesn't resolve either, or lookups time out, the problem is CoreDNS or the path to it, not the namespace. Start with the CoreDNS walkthrough; a DNS failures playbook is coming on the troubleshooting index.

Fix

Call the Service with its namespace: payments.billing works through the search list, and payments.billing.svc.cluster.local is the full name. Put it in the app's config.
Or, if the two apps belong together, run them in the same namespace.
A cross-namespace call also has to pass any Network Policies in billing (card 5).

Prevent

Use <service>.<namespace> or the full .svc.cluster.local name for anything outside the Pod's own namespace, even when it works today because both happen to share one.

Understand it

CoreDNS Namespaces, labels and selectors

Service Not Reachable on Amazon EKS

1Start here

Run this first

2Decision tree

3Causes and fixes

The Service's selector matches no Pods

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

Pods match, but none of them is Ready

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

targetPort isn't the port the app listens on

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

The app listens on 127.0.0.1, not 0.0.0.0

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

A Network Policy drops the traffic

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

The Pod's security group doesn't allow the client

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

kube-proxy isn't programming the client's node

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

The Service is in another namespace

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

4Diagnose it

Related

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it