ALB 502, 503 and 504 Errors on Amazon EKS: Fix "502 Bad Gateway", "503 Service Temporarily Unavailable" and Unhealthy Targets

One card per cause. Each shows what you see, the commands that confirm it, the fix, how to keep it from coming back, and the walkthrough that explains the mechanism. Commands use $TG (the target group ARN, from Start here) and $LB_ARN, the load balancer ARN:

LB_ARN=$(aws elbv2 describe-load-balancers --names k8s-shop-orders-5d1c6a0f2b --query 'LoadBalancers[0].LoadBalancerArn' --output text)

No controller claims the Ingress

ADDRESS empty no events Ingress Class: <none>

Back to the tree

What you see

$ kubectl get ingress orders -n shop
NAME     CLASS    HOSTS   ADDRESS   PORTS   AGE
orders   <none>   *                 80      20m

$ kubectl describe ingress orders -n shop | tail -2
Annotations:      alb.ingress.kubernetes.io/scheme: internet-facing
Events:           <none>

No address and not a single event: nothing is even trying. The AWS Load Balancer Controller only reconciles an Ingress whose class points at it, and it records an event on every attempt, success or failure. Silence means the class is wrong or missing, or the controller isn't running.

Confirm

# Which IngressClasses exist, and which controller owns each?
kubectl get ingressclass
# NAME   CONTROLLER            PARAMETERS   AGE
# alb    ingress.k8s.aws/alb   <none>       41d
# The class this Ingress asks for (empty = none set)
kubectl get ingress orders -n shop -o jsonpath='{.spec.ingressClassName}{"\n"}'
# Is the controller running?
kubectl get deployment aws-load-balancer-controller -n kube-system

The Ingress's ingressClassName must name a class whose controller is ingress.k8s.aws/alb (the AWS Load Balancer Controller). An Ingress with no class is only picked up if some IngressClass is marked default with ingressclass.kubernetes.io/is-default-class: "true". On EKS Auto Mode the built-in controller is eks.amazonaws.com/alb, and you create its IngressClass yourself.

Fix

No or wrong class: set spec.ingressClassName: alb (or whatever kubectl get ingressclass lists for the controller). The old kubernetes.io/ingress.class annotation still works and wins over the field, so remove a stale one that names another controller.
Controller not installed: install the AWS Load Balancer Controller with Helm, with an IAM role (Pod Identity or IRSA) that has its published IAM policy, and with clusterName set to your cluster's name. On EKS Auto Mode it's built in; create an IngressClass for eks.amazonaws.com/alb.
Controller installed but not Ready: read its logs (kubectl logs -n kube-system deploy/aws-load-balancer-controller). A crash right at start is usually a missing clusterName, or the controller can't reach the instance metadata service to find its VPC (set vpcId and region in the Helm values).

Prevent

Always set ingressClassName explicitly in your charts; don't rely on a default class that a cluster may or may not have.
Install the controller with the cluster (as code), and alert when its Deployment has no ready replicas.

Understand it

How Ingress works Gateway API

The controller can't find subnets for the ALB

FailedBuildModel couldn't auto-discover subnets

Back to the tree

What you see

$ kubectl describe ingress orders -n shop | sed -n '/Events/,$p'
Events:
  Type     Reason            Age                From     Message
  ----     ------            ----               ----     -------
  Warning  FailedBuildModel  40s (x12 over 9m)  ingress  Failed build model due to couldn't auto-discover subnets: subnets count less than minimal required count: 1 < 2

# Or, when no subnet qualifies at all
Failed build model due to couldn't auto-discover subnets: unable to resolve at least one subnet. Evaluated 0 subnets: 0 are tagged for other clusters, and 0 have insufficient available IP addresses

Without an alb.ingress.kubernetes.io/subnets annotation, the controller discovers subnets by their role tag, picks one per Availability Zone, and an ALB needs at least two zones. Tagging only one subnet is a classic: it stops the fallback (discovery by route table) and leaves one zone.

Confirm

# The scheme decides the tag: internet-facing -> kubernetes.io/role/elb, internal -> kubernetes.io/role/internal-elb
kubectl get ingress orders -n shop -o jsonpath='{.metadata.annotations.alb\.ingress\.kubernetes\.io/scheme}{"\n"}'
# Subnets in the cluster's VPC carrying the public role tag, with their zones
aws ec2 describe-subnets --filters Name=vpc-id,Values=vpc-0a1b2c3d4e5f60718 Name=tag-key,Values=kubernetes.io/role/elb \
  --query 'Subnets[].[SubnetId,AvailabilityZone,AvailableIpAddressCount]' --output text

You need tagged subnets in two or more zones for the scheme you asked for. The controller also skips subnets tagged kubernetes.io/cluster/<other-cluster> and subnets with too few free IPs (an ALB needs at least 8). The scheme defaults to internal; an internet-facing ALB needs public subnets, with a route to an internet gateway.

Fix

Tag one public subnet per zone, in at least two zones, with kubernetes.io/role/elb = 1 (internet-facing), or private subnets with kubernetes.io/role/internal-elb = 1 (internal). The controller retries on its own; no restart needed.
Or name the subnets on the Ingress: alb.ingress.kubernetes.io/subnets: subnet-0aa…,subnet-0bb… (two zones or more).
Wanted a public ALB but got an internal one (its DNS name starts with internal-) or none? Check alb.ingress.kubernetes.io/scheme: internet-facing is set; without it the controller looks for internal-elb subnets.

Prevent

Tag subnets when you create the VPC (in the same Terraform or CloudFormation), in every zone the cluster uses.
Keep a few dozen free IPs in load balancer subnets; ALBs add nodes as they scale.

Understand it

How Ingress works Internet to Pod traffic path

The controller's IAM role is missing a permission

FailedDeployModel AccessDenied is not authorized to perform

Back to the tree

What you see

$ kubectl describe ingress orders -n shop | sed -n '/Events/,$p'
Events:
  Type     Reason             Age                From     Message
  ----     ------             ----               ----     -------
  Warning  FailedDeployModel  25s (x9 over 6m)   ingress  Failed deploy model due to operation error Elastic Load Balancing v2: DescribeListenerAttributes, https response error StatusCode: 403, RequestID: 2f8c1e4a-7b3d-4c6e-9a1f-5d2b8e0c7a43, api error AccessDenied: User: arn:aws:sts::111122223333:assumed-role/AmazonEKSLoadBalancerControllerRole/1730541247 is not authorized to perform: elasticloadbalancing:DescribeListenerAttributes because no identity-based policy allows the elasticloadbalancing:DescribeListenerAttributes action

Build worked (the Ingress made sense), deploy failed: the controller called an AWS API and was refused. The event names the exact action. A common trigger is upgrading the controller without updating its IAM policy: newer versions call actions that older policies don't grant.

Confirm

# The controller's own log has the same error, with more context
kubectl logs -n kube-system deploy/aws-load-balancer-controller | grep -i accessdenied | tail -3
# Which role does it use? (IRSA annotation; with Pod Identity, list the associations)
kubectl get serviceaccount aws-load-balancer-controller -n kube-system -o yaml | grep role-arn
aws eks list-pod-identity-associations --cluster-name prod --namespace kube-system

Read whose role the error names. A node group role means IRSA or Pod Identity isn't wired up and the controller fell back to the node's instance role. AssumeRoleWithWebIdentity with AccessDenied means the role exists but its trust policy or the cluster's OIDC provider is wrong. no EC2 IMDS role found means no role at all. Calls made while building the model (for example ec2:DescribeSubnets or certificate discovery) fail as FailedBuildModel instead, with the same AccessDenied text.

Fix

Update the role's policy to the iam_policy.json published for your controller version (in the controller's GitHub repo, under docs/install/). Don't hand-add one action at a time; the next missing one will fail the next reconcile.
No role: create one with that policy and bind it to kube-system/aws-load-balancer-controller with EKS Pod Identity or IRSA, then restart the controller Pods so they pick up credentials.

Prevent

Treat the IAM policy as part of the controller upgrade: bump the Helm chart and the policy in the same change.

Understand it

EKS Pod Identity and IRSA How Ingress works

Target type doesn't fit the Service or the network

TargetGroup port is empty instance needs NodePort

Back to the tree

What you see

$ kubectl describe ingress orders -n shop | sed -n '/Events/,$p'
Events:
  Type     Reason            Age               From     Message
  ----     ------            ----              ----     -------
  Warning  FailedBuildModel  1m (x8 over 5m)   ingress  Failed build model due to TargetGroup port is empty. When using Instance targets, your service be must of type 'NodePort' or 'LoadBalancer'

$ kubectl get service orders -n shop
NAME     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
orders   ClusterIP   172.20.45.12   <none>        80/TCP    3d

The default target type is instance: the ALB sends to every node's NodePort, and kube-proxy forwards to a Pod. A ClusterIP Service has no NodePort, so there's nothing to register. (The message's word order, "be must", is the controller's own.)

Confirm

# Which target type does the Ingress (or the Service) ask for? Empty = instance
kubectl get ingress orders -n shop -o jsonpath='{.metadata.annotations.alb\.ingress\.kubernetes\.io/target-type}{"\n"}'
kubectl get service orders -n shop -o jsonpath='{.spec.type}{"\n"}'
# Are Pod IPs real VPC addresses? With the VPC CNI they're in the node subnets
kubectl get pods -n shop -l app=orders -o wide
kubectl get daemonset aws-node -n kube-system

The other mismatch: target-type: ip registers Pod IPs directly, so they must be routable from the VPC. That's true with the Amazon VPC CNI; with an overlay CNI (Calico or Cilium in overlay mode), the ALB is created and the targets register, but they never answer and show up as Target.Timeout (the unhealthy-targets branch of the tree, not this one). Fargate Pods need ip.

Fix

Prefer alb.ingress.kubernetes.io/target-type: ip with the VPC CNI: it works with ClusterIP Services, sends straight to Pods, and the health check tests the Pod itself.
Staying on instance? Change the Service to type: NodePort, and allow the ALB to reach that NodePort on the nodes (the controller adds a rule for that port when it manages backend security group rules, card 7).
On an overlay CNI, use instance targets (the NodePort is on the node's VPC address), or move to the VPC CNI.

Prevent

Set target-type on every Ingress, or once on the IngressClass with IngressClassParams, so nobody gets the default by accident.

Understand it

Services and kube-proxy VPC CNI How Ingress works

No ready Pods, so the target group is empty

503 Service Temporarily Unavailable orders:80 ()

Back to the tree

What you see

$ curl -si http://k8s-shop-orders-5d1c6a0f2b-1187654321.us-east-1.elb.amazonaws.com/ | head -2
HTTP/1.1 503 Service Temporarily Unavailable
Server: awselb/2.0

$ kubectl describe ingress orders -n shop | grep -A3 Backends
  Host        Path  Backends
  ----        ----  --------
  *
              /   orders:80 ()

$ aws elbv2 describe-target-health --target-group-arn "$TG" \
    --query 'TargetHealthDescriptions[].[Target.Id,Target.Port,TargetHealth.State,TargetHealth.Reason]' --output text
# (no output: the target group has no registered targets)

The ALB answers 503 itself when the target group has no registered targets. In ip mode the controller registers only ready Pods, so an empty () after the backend in describe ingress means the same thing as an empty target group: the Service's selector matches nothing, or no Pod is Ready.

Confirm

# Ready endpoints of the backend Service (empty = nothing to register)
kubectl get endpointslices -n shop -l kubernetes.io/service-name=orders
# Do the Service's selector and the Pods' labels match? Are the Pods Ready?
kubectl get service orders -n shop -o jsonpath='{.spec.selector}{"\n"}'
kubectl get pods -n shop -l app=orders
# The controller's TargetGroupBinding for this Service, and its events
kubectl describe targetgroupbinding -n shop

Instance mode looks different: nodes stay registered even with no Pods behind the Service, the health check to the NodePort fails, and you get unhealthy targets and 502s instead of a 503. Targets in an Availability Zone the ALB has no subnet in show unused with Target.NotInUse, and if every target is unused you also get 503.

Fix

Selector matches nothing (labels renamed, an extra key): make the Service's spec.selector match the Pod template's labels.
Pods aren't Ready: fix why the readiness probe fails; the controller registers them as soon as they turn Ready.
Pods are Pending or crashing: see the Pod stuck in Pending and CrashLoopBackOff playbooks. Inside the cluster the same Service refuses connections; the Service not reachable playbook covers that side.
Targets unused: give the ALB a subnet in every zone your Pods run in.

Prevent

Alarm on HTTPCode_ELB_503_Count, and on the target group's HealthyHostCount with missing data treated as breaching: an empty target group stops reporting that metric instead of reporting 0.
Keep the Service and Deployment in one chart, with the selector and labels from one value.

Understand it

Services and kube-proxy Liveness, readiness and startup probes Namespaces, labels and selectors

The health check path or port is wrong

Target.ResponseCodeMismatch Health checks failed with these codes: [404] Target.FailedHealthChecks

Back to the tree

What you see

$ aws elbv2 describe-target-health --target-group-arn "$TG" \
    --query 'TargetHealthDescriptions[].[Target.Id,TargetHealth.State,TargetHealth.Reason,TargetHealth.Description]' --output text
10.0.2.31   unhealthy   Target.ResponseCodeMismatch   Health checks failed with these codes: [404]
10.0.3.17   unhealthy   Target.ResponseCodeMismatch   Health checks failed with these codes: [404]

The ALB's health check is GET / with success code 200 unless you say otherwise. Many APIs answer / with 404, and web apps answer it with a 301 or 302 redirect to a login page or to HTTPS; both count as failures. Target.FailedHealthChecks instead means the check couldn't get an HTTP answer at all: nothing listens on the health check port, or the app listens only on 127.0.0.1.

Confirm

# What does the target group check?
aws elbv2 describe-target-groups --target-group-arns "$TG" \
  --query 'TargetGroups[0].[HealthCheckPath,HealthCheckPort,Matcher.HttpCode,HealthCheckTimeoutSeconds]' --output text
# /   traffic-port   200   5
# Ask the Pod the same question, from a debug Pod (ip mode: the Pod IP and its port)
kubectl run hc-test -n shop --rm -it --restart=Never --image=curlimages/curl -- \
  curl -s -o /dev/null -w '%{http_code}\n' http://10.0.2.31:8080/

If / gives 404 but /healthz gives 200, it's the path. If every target is unhealthy, the ALB fails open and sends traffic to all of them anyway, so a wrong path alone can leave users unaffected while the console shows everything red. The day one Pod passes, the ALB sends traffic only to it.

Fix

Point the check at a real health endpoint: alb.ingress.kubernetes.io/healthcheck-path: /healthz on the Ingress (or on the Service, to set it per backend).
If the endpoint redirects by design, accept it: alb.ingress.kubernetes.io/success-codes: 200-399. Better to give the check a path that answers 200 directly.
Wrong port: alb.ingress.kubernetes.io/healthcheck-port defaults to traffic-port (the port traffic goes to). Set it only if the health endpoint is on another port, and make sure the app listens there on 0.0.0.0.
Slow health endpoint (Target.Timeout with no security group problem): keep it cheap, or raise alb.ingress.kubernetes.io/healthcheck-timeout-seconds (5 s by default).

Prevent

Use the same path for the ALB health check and the readiness probe, and keep it free of downstream checks, so one shared dependency can't take every target out at once.
Set healthcheck-path in the chart's Ingress template, never left to the default.

Understand it

Liveness, readiness and startup probes How Ingress works

A security group blocks the ALB from the targets

Target.Timeout Request timed out 504 after about 10 s

Back to the tree

What you see

$ aws elbv2 describe-target-health --target-group-arn "$TG" \
    --query 'TargetHealthDescriptions[].[Target.Id,Target.Port,TargetHealth.State,TargetHealth.Reason]' --output text
10.0.2.31   8080   unhealthy   Target.Timeout
10.0.3.17   8080   unhealthy   Target.Timeout

$ curl -s -o /dev/null -w '%{http_code} after %{time_total}s\n' http://k8s-shop-orders-5d1c6a0f2b-1187654321.us-east-1.elb.amazonaws.com/
504 after 10.04s

Health checks get no answer at all, so a security group (or a network ACL) is dropping the packets. Every target is unhealthy, the ALB fails open and tries them anyway, and each request waits out the ALB's 10-second connect timeout: 504 after about 10 s. A slow app gives 504 after the 60-second idle timeout instead (card 10).

Confirm

# The security groups on the ALB: the controller's managed one plus the shared backend SG
aws elbv2 describe-load-balancers --names k8s-shop-orders-5d1c6a0f2b --query 'LoadBalancers[0].SecurityGroups'
# Does the Ingress bring its own security groups? Then the controller stops adding backend rules
kubectl get ingress orders -n shop -o yaml | grep -E 'security-groups|manage-backend'
# Inbound rules on the target's security group (the Pod's or node's): is the backend SG allowed on the target port?
aws ec2 describe-security-group-rules --filters Name=group-id,Values=sg-05e6f7a8b9c0d1e2f \
  --query 'SecurityGroupRules[?IsEgress==`false`].[FromPort,ToPort,ReferencedGroupInfo.GroupId,Description]' --output text
# Controller errors adding the rule show on the TargetGroupBinding
kubectl get events -n shop --field-selector reason=FailedNetworkReconcile

By default the controller attaches a shared backend security group to every ALB and adds an inbound rule for it, on the target port, to the security group on each target's network interface. For ip targets that's the node network interface holding the Pod's IP (usually the EKS cluster security group, or the ENIConfig's groups with custom networking); with security groups for Pods it's the Pod's own branch interface and its security group. If that interface has several security groups, exactly one must carry the tag kubernetes.io/cluster/<cluster-name>, or the TargetGroupBinding gets expected exactly one securityGroup tagged with kubernetes.io/cluster/prod … and no rule is added.

Fix

Custom alb.ingress.kubernetes.io/security-groups: the controller then manages no backend rules. Add alb.ingress.kubernetes.io/manage-backend-security-group-rules: "true", or add an inbound rule yourself from the ALB's security group on the target port (the container port in ip mode, the Service's NodePort in instance mode).
Security groups for Pods: the Pod's security group must allow the ALB. Let the controller manage it (one cluster-tagged group on the Pod), or add an inbound rule from the backend security group on the container port.
Several cluster-tagged groups on the node or Pod interface: keep the kubernetes.io/cluster/<name> tag on exactly one of them.
Check network ACLs too: the target subnets must allow the ALB's subnets in on the target port and back out on ephemeral ports (1024 to 65535).

Prevent

Let the controller own frontend and backend security groups unless you have a reason not to; if you bring your own, set manage-backend-security-group-rules: "true" in the same change.
Include a SecurityGroupPolicy's ALB rule in the same chart as the policy itself.

Understand it

Security groups for Pods How Ingress works

Pods stop before the ALB stops sending to them

502 Bad Gateway during rollouts draining

Back to the tree

What you see

# 5xx counted by the ALB itself, per minute, during a 10:40 rollout
$ aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name HTTPCode_ELB_502_Count \
    --dimensions Name=LoadBalancer,Value=app/k8s-shop-orders-5d1c6a0f2b/50dc6c495c0c9188 \
    --start-time 2026-11-02T10:36:00Z --end-time 2026-11-02T10:48:00Z --period 60 --statistics Sum \
    --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text
2026-11-02T10:40:00+00:00   37.0
2026-11-02T10:41:00+00:00   52.0
# minutes with no 502s have no datapoint at all

$ aws elbv2 describe-target-health --target-group-arn "$TG" \
    --query 'TargetHealthDescriptions[].[Target.Id,Target.Port,TargetHealth.State,TargetHealth.Reason]' --output text
10.0.2.31   8080   draining   Target.DeregistrationInProgress
10.0.3.17   8080   initial    Elb.InitialHealthChecking
10.0.1.84   8080   initial    Elb.RegistrationInProgress

Two races, both at rollout time. Ending: a Pod gets SIGTERM and exits at the same moment the controller starts deregistering it, so the ALB still has requests on it or sends new ones, and the closed connection becomes a 502. Starting: new Pods turn Ready (to Kubernetes) before their targets turn healthy (to the ALB), the rollout deletes the old Pods, and for a moment the ALB has only initial and draining targets.

Confirm

# Readiness gates injected? READINESS GATES shows 1/1 (0/1 while pending), or <none> without them
kubectl get pods -n shop -l app=orders -o wide
kubectl get namespace shop --show-labels | grep pod-readiness-gate-inject
# Does the container wait before shutting down, and how long may it take?
kubectl get deployment orders -n shop -o jsonpath='{.spec.template.spec.containers[0].lifecycle}{"\n"}{.spec.template.spec.terminationGracePeriodSeconds}{"\n"}'
# The target group's deregistration delay (300 s by default)
aws elbv2 describe-target-group-attributes --target-group-arn "$TG" \
  --query "Attributes[?Key=='deregistration_delay.timeout_seconds']"

502s that line up with rollouts, scale-ins or node drains, and nothing in between, point here. Readiness gates need ip targets; the preStop delay helps with either target type.

Fix

Readiness gates: label the namespace elbv2.k8s.aws/pod-readiness-gate-inject=enabled, then restart the Deployment (gates are injected only when a Pod is created). A new Pod is then Ready only once its target is healthy in the ALB, so the rollout can't remove old Pods first. Works with target-type: ip.
A preStop delay: lifecycle.preStop.sleep.seconds: 15 (or an exec sleep 15 if the image has a shell). The Pod keeps serving while the controller deregisters it, and only then gets SIGTERM.
Graceful shutdown: on SIGTERM, stop accepting new connections and finish in-flight requests. Keep terminationGracePeriodSeconds (30 s by default) longer than the preStop sleep plus that drain time.
Deregistration delay: 300 s by default. It doesn't hold the Pod: the Pod is killed when terminationGracePeriodSeconds runs out, so a delay longer than the grace period buys nothing. Set it a little above your longest request and below the grace period (for example alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30). Requests still running when it ends are cut off with a 502.

Prevent

Make readiness gates, a preStop delay and SIGTERM handling part of every Deployment behind an ALB.
Watch HTTPCode_ELB_502_Count on every deploy; a clean rollout leaves it at zero.

Understand it

Deployments and ReplicaSets Liveness, readiness and startup probes

The app closes idle connections before the ALB does

502 Bad Gateway target_status_code - keep-alive 5 s < idle timeout 60 s

Back to the tree

What you see

# ALB access log: a 502 the target never answered (target_status_code is "-")
http 2026-11-02T14:02:11.384512Z app/k8s-shop-orders-5d1c6a0f2b/50dc6c495c0c9188 203.0.113.24:51344 10.0.3.17:8080 -1 -1 -1 502 - 412 277 "GET http://orders.example.com:80/api/orders/1189 HTTP/1.1" "Mozilla/5.0" - - arn:aws:elasticloadbalancing:us-east-1:111122223333:targetgroup/k8s-shop-orders-8f3a2c1d9e/6d0ecf831eec9f09 "Root=1-6725f0a3-4c1e2b7d9a0f3e5c6b8d1a2f" "-" "-" 1 2026-11-02T14:02:11.383000Z "forward" "-" "-" "10.0.3.17:8080" "-" "-" "-" TID_8c3a1f0e

# The app: Node.js, whose HTTP server closes idle keep-alive connections after 5 s by default
$ kubectl exec deploy/orders -n shop -- node -e 'console.log(require("http").createServer().keepAliveTimeout)'
5000

The ALB reuses connections to targets and keeps them open for its idle timeout, 60 s by default. If the app closes an idle connection sooner, the ALB can send a request just as the app closes it, and that request gets a 502. A small, steady trickle of 502s, at any time of day, with no deploy and healthy targets, is the signature.

Confirm

# The ALB's idle timeout
aws elbv2 describe-load-balancer-attributes --load-balancer-arn "$LB_ARN" \
  --query "Attributes[?Key=='idle_timeout.timeout_seconds']"
# The app's keep-alive timeout: check its server config. Common defaults:
#   Node.js http.Server keepAliveTimeout 5 s   Gunicorn --keep-alive 2 s (threaded or async workers)

In the access logs, these 502s have target_status_code - and processing times of -1: the ALB sent the request but never got a response. The rate is low (often well under 1%) and rises with traffic that has gaps of a few seconds between requests on a connection.

Fix

Make the app's keep-alive (idle) timeout longer than the ALB's idle timeout: for example 65 s when the ALB uses 60 s. Node.js: server.keepAliveTimeout = 65000 (on older Node versions, also set server.headersTimeout above it). Gunicorn with gthread, gevent or eventlet workers: --keep-alive 65 (the default sync worker doesn't keep connections open).
If you raise the ALB idle timeout (card 10), raise the app's keep-alive above the new value too.

Prevent

Put the keep-alive setting in the service template for every HTTP framework you run behind an ALB, with a comment pointing at the ALB's idle timeout.

Understand it

How Ingress works Internet to Pod traffic path

The request outlasts the ALB's idle timeout

504 Gateway Time-out after 60 s

Back to the tree

What you see

$ curl -s -o /dev/null -w '%{http_code} after %{time_total}s\n' http://k8s-shop-orders-5d1c6a0f2b-1187654321.us-east-1.elb.amazonaws.com/api/reports/export
504 after 60.02s

# ALB access log: connected, sent the request, no response before the idle timeout
http 2026-11-02T15:20:41.902117Z app/k8s-shop-orders-5d1c6a0f2b/50dc6c495c0c9188 203.0.113.24:50122 10.0.2.31:8080 0.001 -1 -1 504 - 389 299 "GET http://orders.example.com:80/api/reports/export HTTP/1.1" "curl/8.11.1" - - ...

Exactly the idle timeout (60 s unless changed) means the connection to the Pod worked and the request was sent, but the app sent nothing back in time. Compare with card 7: there the 504 comes after about 10 s, because the connection itself never opened.

Confirm

# The current idle timeout
aws elbv2 describe-load-balancer-attributes --load-balancer-arn "$LB_ARN" \
  --query "Attributes[?Key=='idle_timeout.timeout_seconds']"
# How long does the endpoint take when called directly, skipping the ALB?
kubectl run slow-test -n shop --rm -it --restart=Never --image=curlimages/curl -- \
  curl -s -o /dev/null -w '%{time_total}s\n' --max-time 300 http://10.0.2.31:8080/api/reports/export

The idle timeout counts time with no bytes in either direction, not the total request time. A response that streams data every few seconds never hits it; one that thinks for 61 s before the first byte does.

Fix

Make long work asynchronous: return 202 Accepted with a job ID and let the client poll, or stream progress so bytes keep moving.
If the request really must take longer, raise the timeout: alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=180. In an IngressGroup every Ingress shares one ALB, so the value applies to all of them.
After raising it, keep the app's keep-alive above it (card 9).
Find why it's slow: a starved Pod (CPU throttling, too few replicas) shows as rising TargetResponseTime for every path, not just one.

Prevent

Set a budget per endpoint (for example p99 under 10 s) well inside the idle timeout, and alert on TargetResponseTime approaching it.

Understand it

How Ingress works Internet to Pod traffic path

ALB 502, 503 and 504 Errors on Amazon EKS

1Start here

Run this first

2Decision tree

3Causes and fixes

No controller claims the Ingress

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

The controller can't find subnets for the ALB

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

The controller's IAM role is missing a permission

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

Target type doesn't fit the Service or the network

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

No ready Pods, so the target group is empty

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

The health check path or port is wrong

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

A security group blocks the ALB from the targets

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

Pods stop before the ALB stops sending to them

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

The app closes idle connections before the ALB does

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

The request outlasts the ALB's idle timeout

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

4Diagnose it

Related

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it