Why Ingress Exists: Follow One Packet on Amazon EKS

1Inside the cluster: why a Service exists

On EKS the VPC CNI gives every pod a secondary IP from its node's subnet. Pod IPs are ordinary VPC addresses, so the web pod can call a shop pod at 10.0.1.15:8080 on another node with no overlay and no NAT. That works until the shop pod is replaced.

Pods are replaced, not repaired. Each of these creates a new pod with a new IP:

A rolling update or a rollback
An eviction or a node drain
Karpenter consolidation
A scale-down followed by a scale-up

A container restart inside the same pod keeps its IP; a new pod never does. Anything that stored 10.0.1.15 now times out, and with two replicas the caller would also need to pick one and skip any that aren't ready.

A Service fixes both problems. Its selector, app: shop, finds the pods, and the EndpointSlice controller keeps a list of their IPs with a ready flag on each. The Service gets a fixed virtual IP, 172.20.14.7, and a DNS name that CoreDNS answers: shop.default.svc.cluster.local, or just shop from the same namespace.

No machine owns the ClusterIP. kube-proxy on every node turns the Service into iptables rules that say "a packet to 172.20.14.7:80 gets its destination rewritten to one ready endpoint on :8080". The rewrite (DNAT) happens on the node that sends the packet, which then crosses the VPC straight to the chosen pod IP. That's also why you can't ping a ClusterIP in iptables mode, the EKS default: only the Service's TCP or UDP ports are translated.

content_copyCopy

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop
spec:
  replicas: 2
  selector:
    matchLabels: { app: shop }
  template:
    metadata:
      labels: { app: shop }
    spec:
      containers:
      - name: shop
        image: shop:1.4
        ports:
        - containerPort: 8080

apiVersion: v1
kind: Service
metadata:
  name: shop
spec:
  selector: { app: shop }   # finds the pods
  ports:
  - port: 80                # the ClusterIP port
    targetPort: 8080        # the pod port
# type: ClusterIP is the default

$ kubectl get pods -l app=shop -o wide
NAME          READY  IP          NODE
shop-7d9f-a   1/1    10.0.1.47   node-1    # was 10.0.1.15 before the rollout
shop-7d9f-b   1/1    10.0.2.31   node-2

$ kubectl get svc shop
NAME  TYPE       CLUSTER-IP    PORT(S)
shop  ClusterIP  172.20.14.7   80/TCP

$ kubectl get endpointslices -l kubernetes.io/service-name=shop
NAME         ADDRESSTYPE  PORTS  ENDPOINTS
shop-x7k2p   IPv4         8080   10.0.1.47,10.0.2.31

Readiness is what makes this safe during deploys. A pod whose readiness probe fails stays in the EndpointSlice but is marked not ready, kube-proxy stops sending to it, and traffic shifts to the other replica.

warning

Always define a readiness probe. A pod without one counts as ready as soon as its containers start, so a rollout sends traffic to an app that's still booting.

2From outside: NodePort, then LoadBalancer

The ClusterIP range (172.20.0.0/16 on this cluster, 10.100.0.0/16 on others) isn't in any VPC route table. It exists only as rules on the nodes, so a client on the internet, or in another VPC, has no route to it. For debugging, kubectl port-forward svc/shop 8080:80 tunnels one connection through the API server, but that isn't a way to serve users.

NodePort is a building block, not a front door. Changing the Service to type: NodePort makes kube-proxy open the same port, from 30000–32767, on every node, including nodes that run no shop pod, and traffic to any node on :30080 is rewritten to a shop pod anywhere in the cluster. Clients would need node IPs that change as nodes come and go, a high port, and a security group that allows it, and EKS nodes usually sit in private subnets anyway.

Setting externalTrafficPolicy: Local changes two things: only nodes that run a shop pod answer, and the client's source IP survives.

type: LoadBalancer keeps the NodePort and asks a controller to put a cloud load balancer in front of it. Which load balancer you get depends on what's watching:

With the AWS Load Balancer Controller (v2.5 and later), new LoadBalancer Services get an NLB. In the default instance mode, the NLB sends to the NodePort on healthy nodes and kube-proxy picks the pod.
With aws-load-balancer-nlb-target-type: ip, the NLB registers pod IPs directly. That skips the NodePort hop and is the only option for Fargate pods.
Without the controller, the legacy in-tree cloud provider creates a Classic Load Balancer.
EKS Auto Mode runs the load balancer controller for you.

Prefer ip mode in production: it removes a hop and works for every compute type. The player uses instance mode only so you can see the NodePort doing its job.

content_copyCopy

spec:
  type: NodePort
  selector: { app: shop }
  ports:
  - port: 80
    targetPort: 8080
    nodePort: 30080     # 30000-32767; omit it to get one assigned

apiVersion: v1
kind: Service
metadata:
  name: shop
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance   # prefer ip in production
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
  type: LoadBalancer
  selector: { app: shop }
  ports:
  - port: 80
    targetPort: 8080

3Many apps, one entry: Ingress and its controller

A LoadBalancer Service is fine for one app. When api needs to be public too, it gets a second NLB with its own DNS name. At twenty microservices that's twenty load balancers at roughly $16 a month each in us-east-1 before traffic, twenty DNS names and twenty places to configure TLS. An NLB also works at layer 4: it sees IPs and ports, never the URL path or the Host header, so it can't send /api to one Service and / to another.

Keep a per-Service NLB when the traffic isn't HTTP, such as raw TCP, databases or MQTT, or when you need static IPs at layer 4. For HTTP, use an Ingress.

An Ingress describes layer-7 routing for many Services: hosts, paths and TLS. Each backend still points at a Service, which the controller uses to find the pods, so the Services stay. They just go back to type: ClusterIP.

warning

An Ingress on its own does nothing. kubectl apply succeeds, the object is stored, and its ADDRESS stays empty. Kubernetes defines the Ingress API but ships no implementation, because the right one depends on where you run. ingressClassName names the controller that owns it.

On EKS that controller is usually the AWS Load Balancer Controller, a Deployment in kube-system that reconciles in a loop:

Watch Ingresses with ingressClassName: alb, plus their Services and EndpointSlices.
Work out the ALB they need: listeners, one rule per host and path, and a target group per backend Service.
Call the Elastic Load Balancing API to create or update it, using IAM permissions from IAM Roles for Service Accounts (IRSA) or EKS Pod Identity.
Keep the target groups in step as pods come and go, and write the ALB's DNS name into the Ingress ADDRESS.

Three settings decide how that ALB behaves:

alb.ingress.kubernetes.io/target-type: ip fills the target groups with pod IPs from the EndpointSlices, so the ALB connects to the pod directly, with no NodePort and no kube-proxy hop.
TLS terminates on the ALB with an ACM certificate, set with alb.ingress.kubernetes.io/certificate-arn or discovered from the host name.
alb.ingress.kubernetes.io/group.name lets several Ingresses share one ALB.

warning

Without group.name, each Ingress gets its own ALB. Ten teams with ten Ingresses bring back the one-load-balancer-per-app problem this section set out to fix.

content_copyCopy

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
  - host: shop.example.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend: { service: { name: api, port: { number: 80 } } }
      - path: /
        pathType: Prefix
        backend: { service: { name: shop, port: { number: 80 } } }

$ kubectl get ingress shop
NAME  CLASS  HOSTS             ADDRESS
shop  alb    shop.example.com  k8s-default-shop-9f8e.us-east-1.elb.amazonaws.com

$ curl -H 'Host: shop.example.com' \
    http://k8s-default-shop-9f8e.us-east-1.elb.amazonaws.com/api/cart
# ALB rule /api*  -> target group api  -> 10.0.1.31, 10.0.2.44
# ALB rule /*     -> target group shop -> 10.0.1.47, 10.0.2.31

4Which object fixes which failure

Failure	Object	What actually does the work
Pod IP changes when the pod is replaced; two replicas to choose from	Service (ClusterIP)	CoreDNS answers a stable name; kube-proxy on every node DNATs the ClusterIP to a ready endpoint
ClusterIP isn't routable from outside the cluster	type: NodePort	kube-proxy opens the same port, 30000–32767, on every node
Clients shouldn't need node IPs or high ports	type: LoadBalancer	A controller creates one NLB per Service in front of the nodes or pods
One load balancer per app; no routing by path	Ingress	Nothing on its own; it stores HTTP routing rules
The Ingress has no ADDRESS	Ingress controller	The AWS Load Balancer Controller builds one ALB and registers pod IPs as targets

Type	Reachable from	Address	Layer
ClusterIP	Pods and nodes in the cluster	Virtual IP and name.namespace.svc.cluster.local	L4
NodePort	Anything that can reach a node IP	Any node IP on a port from 30000–32767	L4
LoadBalancer	The internet or the VPC, by scheme	One NLB DNS name per Service	L4
Ingress (not a Service type)	The internet or the VPC, by scheme	One ALB DNS name for many Services	L7: host, path, TLS

5Debugging a 502 or 503 from the ALB

Walk the same chain backwards, from the pod out to the Ingress:

Pods: are they running, and does the readiness probe pass?
EndpointSlice: does kubectl get endpointslices -l kubernetes.io/service-name=api list ready IPs? An empty list usually means the Service selector doesn't match the pod labels.
Target group: are the targets healthy in the ELB console, and does the health check path return 200?
Security groups: can the ALB's security group reach the pod port, 8080 here, not just 80?
Ingress: kubectl describe ingress shop shows the controller's events, including permission and annotation errors.

6What comes after Ingress

On a new platform, start with the Gateway API. On an existing one, Ingress keeps working and there's no rush to migrate: the Ingress API is stable and frozen, and new features go into the Gateway API instead.

The reason to move is ownership. Gateway API splits what Ingress mixes together: a platform team owns the Gateway (the load balancer itself), and app teams attach their own HTTPRoutes to it. The AWS Load Balancer Controller supports the Gateway API too, so the chain on this page stays the same: Pod, Service, routing rules, and a controller that builds the load balancer.

Conclusion

Each object in the chain exists because the one before it fails at something specific, so remembering the failure brings back the object. For HTTP on EKS, run ClusterIP Services behind one ALB Ingress with target-type: ip and a shared group.name, keep per-Service NLBs for non-HTTP traffic, and when something breaks, walk the chain in reverse from the pod out.

References

Kubernetes documentation: Service, EndpointSlices, Virtual IPs and service proxies, DNS for Services and Pods.
Kubernetes documentation: Ingress, Ingress controllers, Gateway API.
Amazon EKS User Guide: Route TCP and UDP traffic with Network Load Balancers, Route application and HTTP traffic with Application Load Balancers, Assign IPs to Pods with the Amazon VPC CNI.
AWS Load Balancer Controller documentation: Ingress annotations and Service annotations.

Assumptions for the player and the examples: Amazon EKS with the VPC CNI, kube-proxy in iptables mode, and the AWS Load Balancer Controller v2 (an NLB in instance mode for the LoadBalancer Service, target-type: ip on the Ingress). IPs, the ClusterIP range and names are examples. Compare with your own cluster using kubectl get svc,endpointslices,ingress -A.

Why Ingress exists: follow one packet from a Pod to an ALB on Amazon EKS

1Inside the cluster: why a Service exists

2From outside: NodePort, then LoadBalancer

3Many apps, one entry: Ingress and its controller

4Which object fixes which failure

5Debugging a 502 or 503 from the ALB

6What comes after Ingress

Conclusion

References