Out of Pod IP Addresses on Amazon EKS: Fix "failed to assign an IP address to container"

One card per cause. Each shows what the event and ipamd log look like, the commands that confirm it, the fix, how to keep it from coming back, and the walkthrough that explains the mechanism.

max Pods is higher than the node's ENIs can give IPs for

Skipping ENI allocation as the max ENI limit is already reached

Back to the tree

What you see

$ kubectl get pods -n shop -o wide --field-selector spec.nodeName=ip-10-0-1-23.ec2.internal
NAME                  READY   STATUS              NODE
web-7d9f8c6b5-2bq7d   1/1     Running             ip-10-0-1-23.ec2.internal
...
web-7d9f8c6b5-x2k4q   0/1     ContainerCreating   ip-10-0-1-23.ec2.internal

# /var/log/aws-routed-eni/ipamd.log on the node
{"level":"debug","caller":"ipamd/ipamd.go:1031","msg":"Skipping ENI allocation as the max ENI limit is already reached"}
{"level":"error","caller":"datastore/data_store.go:858","msg":"DataStore has no available IP/Prefix addresses"}

The scheduler only checks the node's max Pods. If that's set higher than the instance's network interfaces can hold IPs for, Pods keep landing on a node that has no IP left to give them. An m5.large has 3 ENIs with 10 IPs each; one per ENI is the ENI's own, so 27 Pods can get an IP, and its default max Pods is 29 (27 plus 2 host-network Pods, aws-node and kube-proxy).

Confirm

# The kubelet's max Pods, and the instance type
kubectl get node ip-10-0-1-23.ec2.internal -o jsonpath='{.status.allocatable.pods} {.metadata.labels.node\.kubernetes\.io/instance-type}{"\n"}'
# ENIs per instance, and IPv4 addresses per ENI
aws ec2 describe-instance-types --instance-types m5.large \
  --query 'InstanceTypes[].NetworkInfo.[MaximumNetworkInterfaces,Ipv4AddressesPerInterface]' --output text
# Pods on the node that need a VPC IP (host-network Pods use the node's IP)
kubectl get pods -A --field-selector spec.nodeName=ip-10-0-1-23.ec2.internal,status.phase=Running -o custom-columns=NAME:.metadata.name,HOSTNET:.spec.hostNetwork --no-headers | grep -vc true

Here it's the instance's limit, not the subnet: the subnet still has free IPs, and the ipamd log has no EC2 error, just the ENI limit. Max Pods gets set too high by a custom --max-pods or maxPods in node user data, spec.kubelet.maxPods in a Karpenter EC2NodeClass, or by turning prefix delegation off (or custom networking on) without replacing the nodes.

Fix

Make max Pods match the IPs: work it out with the EKS max-pods-calculator.sh script (it takes the instance type, the VPC CNI version, and flags for prefix delegation and custom networking), and set that on new nodes. Managed node groups that use the EKS AMI without a custom AMI in the launch template set it for you. In Karpenter, remove maxPods: with the AL2, AL2023 and Bottlerocket AMI families Karpenter then calculates it from the ENI limits.
Or give the node more IPs: turn on prefix delegation (ENABLE_PREFIX_DELEGATION=true, Nitro instances only). Each ENI slot then holds a /28 prefix of 16 addresses, so a higher max Pods is real. Only new nodes get the new max Pods; replace the old ones.
Custom networking: the primary ENI no longer hosts Pods, so an m5.large gives (3 − 1) × (10 − 1) + 2 = 20 Pods, not 29. Lower max Pods to match (with Karpenter, set its reservedENIs setting to 1), or pair it with prefix delegation.

Prevent

Never set max Pods by hand without the calculator, and change it in the same rollout as any VPC CNI mode change.
Choose instance types for Pod count as well as CPU and memory; small instances run out of IPs first.

Understand it

The VPC CNI Prefix delegation and custom networking Managed node groups

The subnet is out of free IP addresses

InsufficientFreeAddressesInSubnet AvailableIpAddressCount 0

Back to the tree

What you see

# /var/log/aws-routed-eni/ipamd.log on the node
{"level":"error","caller":"ipamd/ipamd.go:1099","msg":"Failed to increase pool size due to not able to allocate ENI AllocENI: failed to create ENI: failed to create network interface: operation error EC2: CreateNetworkInterface, https response error StatusCode: 400, RequestID: 5b1e…, api error InsufficientFreeAddressesInSubnet: The specified subnet does not have enough free addresses to satisfy the request."}
{"level":"error","caller":"ipamd/ipamd.go:1104","msg":"Unable to attach IPs/Prefixes for the ENI, subnet doesn't seem to have enough IPs/Prefixes. Consider using new subnet or carve a reserved range using create-subnet-cidr-reservation"}

$ aws ec2 describe-subnets --subnet-ids subnet-0a1b2c3d4e5f60718 --query 'Subnets[].[SubnetId,AvailabilityZone,CidrBlock,AvailableIpAddressCount]' --output text
subnet-0a1b2c3d4e5f60718   us-east-1a   10.0.1.0/24   0

When an existing ENI has free slots, ipamd asks EC2 for more IPs on it (AssignPrivateIpAddresses) and the same error code comes back from that call instead. Either way the subnet has no addresses left, and new nodes can't launch in it either.

Confirm

# Which subnet is the node in? (with custom networking, check the ENIConfig's subnet instead)
aws ec2 describe-instances --instance-ids i-0abc123def4567890 --query 'Reservations[].Instances[].SubnetId' --output text
aws ec2 describe-subnets --subnet-ids subnet-0a1b2c3d4e5f60718 --query 'Subnets[].[SubnetId,AvailabilityZone,CidrBlock,AvailableIpAddressCount]' --output text
# Who holds the addresses? Count network interfaces in the subnet by type and description
aws ec2 describe-network-interfaces --filters Name=subnet-id,Values=subnet-0a1b2c3d4e5f60718 \
  --query 'NetworkInterfaces[].[InterfaceType,Status,Description]' --output text | sort | uniq -c | sort -rn | head

VPC CNI ENIs have the description aws-K8S-<instance-id>. If they hold far more IPs than you run Pods, the warm pool is the problem (card 3). Load balancers, VPC endpoints, Lambda functions and databases share the subnet too, and AWS reserves 5 addresses in every subnet.

Fix

Quickest: add a subnet in the same Availability Zone (from a new VPC CIDR if the current one is full) and tag it kubernetes.io/role/cni=1. With subnet discovery (VPC CNI 1.18+, on by default), new ENIs go to the subnet with the most free IPs. Pods keep the IPs they have.
Free addresses: shrink the warm pool if nodes hold idle IPs (card 3), and delete network interfaces stuck in available that nothing uses. Ready nodes running few Pods also hold whole ENIs; consolidating them gives addresses back.
With custom networking: Pods take IPs from the ENIConfig's subnet for that zone, not the node subnet. A subnet can't be resized, so create a new, larger one in that zone and point the zone's ENIConfig at it (new nodes pick it up).

Prevent

Size subnets for Pods, not just nodes; every Pod takes a VPC address. Work backwards from max nodes × max Pods per node.
Monitor free IPs: the CNI metrics helper publishes the cluster's IP and ENI counts to CloudWatch, and you can alarm on each subnet's AvailableIpAddressCount from a script or on Amazon VPC IPAM utilization.

Understand it

The VPC CNI Prefix delegation and custom networking

Long-term fixes: add a secondary CIDR to the VPC (AWS suggests 100.64.0.0/10 space, for example 100.64.0.0/16) and use custom networking so Pods take IPs from it instead of the node subnets; use IPv6, where each Pod gets an address from the subnet's /64 (a new cluster: the IP family is set when you create it); or rebuild on larger subnets. AWS recommends better VPC design over tuning the warm pool, which is a stopgap.

The warm pool holds IPs that no Pod uses

Total IPs/Prefixes = 58/0, AssignedIPs/CooldownIPs: 4/0 WARM_ENI_TARGET=1

Back to the tree

What you see

# /var/log/aws-routed-eni/ipamd.log: an m5.4xlarge with 4 Pods holds 58 IPs
{"level":"debug","caller":"ipamd/ipamd.go:1521","msg":"IP pool stats for network card 0: Total IPs/Prefixes = 58/0, AssignedIPs/CooldownIPs: 4/0, c.maxIPsPerENI = 29"}

# ...while the subnet is empty, and other nodes' Pods can't get one
$ aws ec2 describe-subnets --subnet-ids subnet-0a1b2c3d4e5f60718 --query 'Subnets[].[SubnetId,AvailabilityZone,CidrBlock,AvailableIpAddressCount]' --output text
subnet-0a1b2c3d4e5f60718   us-east-1a   10.0.1.0/24   0

By default (WARM_ENI_TARGET=1) ipamd keeps one whole ENI's worth of IPs free on every node, and attaches another ENI when it dips below that. An m5.4xlarge ENI holds 29 Pod IPs, so a node with 4 Pods holds 58 addresses. Multiply by the nodes and small subnets drain while most addresses sit idle.

Confirm

# The warm settings on aws-node (no WARM_IP_TARGET or MINIMUM_IP_TARGET means WARM_ENI_TARGET=1 applies)
kubectl get daemonset aws-node -n kube-system -o jsonpath='{range .spec.template.spec.containers[?(@.name=="aws-node")].env[*]}{.name}={.value}{"\n"}{end}' | grep -E 'WARM|MINIMUM|PREFIX'
# On a node: IPs attached vs IPs used by Pods
curl -s http://localhost:61678/metrics | grep -E '^awscni_(total|assigned)_ip_addresses'

awscni_total_ip_addresses far above awscni_assigned_ip_addresses on every node, and an empty subnet, is this cause. Query the metrics from the node (localhost:61678) or at the node's IP, since aws-node uses host networking; across the cluster, the CNI metrics helper sends the same numbers to CloudWatch.

Fix

Keep fewer IPs warm: set MINIMUM_IP_TARGET a little above the Pods you expect per node, and WARM_IP_TARGET to a few spare IPs. For example aws eks update-addon --cluster-name prod --addon-name vpc-cni --configuration-values '{"env":{"MINIMUM_IP_TARGET":"12","WARM_IP_TARGET":"2"}}'. Either one overrides WARM_ENI_TARGET.
The trade-off: ipamd now calls EC2 to add and remove IPs as Pods come and go. In large clusters or with high Pod churn, those calls can be throttled. EC2 throttles per account and Region, so that slows IP allocation for every node, and can hit other clusters too; bursts of Pods also start slower. AWS calls this a stopgap; fix the subnets for good (card 2's note).
Don't set MINIMUM_IP_TARGET alone: without WARM_IP_TARGET the node stops at that many IPs and never grows.
With prefix delegation, the default WARM_PREFIX_TARGET=1 keeps one whole free /28 (16 IPs) on top of the prefix in use, so a node can hold up to about 31 idle IPs. A WARM_IP_TARGET below 16 stops it keeping an entire spare prefix.

Prevent

Set these in the EKS add-on's configuration values, not by editing the DaemonSet, so an add-on update doesn't undo them.
Revisit the targets when you change instance sizes: bigger instances have bigger ENIs, so the default warm pool grows with them.

Understand it

The VPC CNI

Prefix delegation is on, but the subnet has no free /28

InsufficientCidrBlocks

Back to the tree

What you see

# /var/log/aws-routed-eni/ipamd.log on the node
{"level":"error","caller":"awsutils/awsutils.go:2250","msg":"Failed to allocate a private IP/Prefix addresses on ENI eni-0f1e2d3c4b5a69788: operation error EC2: AssignPrivateIpAddresses, https response error StatusCode: 400, RequestID: 9c2d…, api error InsufficientCidrBlocks: There are not enough free cidr blocks in the specified subnet to satisfy the request."}

# The subnet still has free addresses, just not 16 in a row
$ aws ec2 describe-subnets --subnet-ids subnet-0a1b2c3d4e5f60718 --query 'Subnets[].[SubnetId,AvailabilityZone,CidrBlock,AvailableIpAddressCount]' --output text
subnet-0a1b2c3d4e5f60718   us-east-1a   10.0.1.0/24   37

With ENABLE_PREFIX_DELEGATION=true, ipamd asks EC2 for /28 prefixes: 16 contiguous addresses on a /28 boundary. A subnet whose free addresses are scattered between nodes, load balancers and older secondary IPs can have dozens free and still no whole /28.

Confirm

# Prefix delegation on?
kubectl get daemonset aws-node -n kube-system -o jsonpath='{.spec.template.spec.containers[?(@.name=="aws-node")].env[?(@.name=="ENABLE_PREFIX_DELEGATION")].value}{"\n"}'
# Free addresses in the subnet, and any reservations
aws ec2 describe-subnets --subnet-ids subnet-0a1b2c3d4e5f60718 --query 'Subnets[].[SubnetId,AvailabilityZone,CidrBlock,AvailableIpAddressCount]' --output text
aws ec2 get-subnet-cidr-reservations --subnet-id subnet-0a1b2c3d4e5f60718

Fix

Give prefixes their own space: create a new subnet in the same zone for Pods (tag it kubernetes.io/role/cni=1 for subnet discovery, or point custom networking at it). If other things will share that subnet, reserve a range inside it for prefixes before they do: aws ec2 create-subnet-cidr-reservation --subnet-id subnet-0f9e8d7c6b5a43210 --cidr 10.0.64.0/20 --reservation-type prefix (for a new 10.0.64.0/18 subnet). EC2 then takes prefixes from the reservation. A reservation doesn't free addresses already in use, and reserved space stops counting in AvailableIpAddressCount.
Until then: a node that already holds prefixes keeps serving IPs from them, so lowering WARM_PREFIX_TARGET or using WARM_IP_TARGET lets nodes share what's left for longer.

Prevent

Before turning on prefix delegation, check that the subnets have contiguous free space, and reserve it with a prefix CIDR reservation.
Keep Pod subnets for Pods: with custom networking or subnet discovery, nodes, load balancers and endpoints don't fragment them.

Understand it

Prefix delegation and custom networking The VPC CNI

aws-node's IAM role can't call the EC2 APIs it needs

MissingIAMPermissions UnauthorizedOperation

Back to the tree

What you see

$ kubectl get events -n kube-system --field-selector reason=MissingIAMPermissions
LAST SEEN   TYPE      REASON                  OBJECT               MESSAGE
2m          Warning   MissingIAMPermissions   pod/aws-node-7xk2p   Unauthorized operation: failed to call ec2:CreateNetworkInterface due to missing permissions. Please refer https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/iam-policy.md to attach relevant policy to IAM role

# /var/log/aws-routed-eni/ipamd.log on the node
{"level":"error","caller":"ipamd/ipamd.go:1099","msg":"Failed to increase pool size due to not able to allocate ENI AllocENI: failed to create ENI: failed to create network interface: operation error EC2: CreateNetworkInterface, https response error StatusCode: 403, RequestID: 1f3a…, api error UnauthorizedOperation: You are not authorized to perform this operation."}

ipamd records the event on its own aws-node Pod, not on yours. Here it can fill the ENI it has, so the node is Ready and the first Pods run; the next ENI is refused, and Pods past that get no IP. If ipamd can't call EC2 at all (a broken trust policy, or AssignPrivateIpAddresses denied), a new node never gets ready: see the Node NotReady playbook. Nodes that were already running may stay Ready while aws-node crash-loops, and new Pods fail with add cmd: Error received from AddNetwork gRPC call.

Confirm

# Which role does aws-node use? IRSA annotation, or a Pod Identity association
kubectl get serviceaccount aws-node -n kube-system -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}{"\n"}'
aws eks list-pod-identity-associations --cluster-name prod --namespace kube-system --service-account aws-node
# What's attached to that role (or, with neither, to the node role)?
aws iam list-attached-role-policies --role-name prod-vpc-cni

Fix

Attach the managed policy: AmazonEKS_CNI_Policy covers IPv4 (CreateNetworkInterface, AttachNetworkInterface, AssignPrivateIpAddresses and the rest). IPv6 clusters need the IPv6 policy from the EKS docs (AssignIpv6Addresses) instead.
The policy is there, but calls are still denied: look for a service control policy or permissions boundary that denies these EC2 actions, and for a broken trust policy if the role moved to IRSA or Pod Identity (then every call fails: the aws-node logs show an STS AccessDenied, not a MissingIAMPermissions event).
Restart aws-node after fixing IRSA or Pod Identity, so it picks up the new credentials: kubectl rollout restart daemonset aws-node -n kube-system.

Prevent

Give aws-node its own role (IRSA or Pod Identity) with AmazonEKS_CNI_Policy, managed in the same IaC as the add-on, as AWS recommends.
Alert on MissingIAMPermissions events in kube-system.

Understand it

Pod Identity and IRSA The VPC CNI

Security groups for Pods: no branch ENI left

Insufficient vpc.amazonaws.com/pod-eni

Back to the tree

What you see

$ kubectl get pod payments-5f7c8d9b6-q2w4e -n shop
NAME                       READY   STATUS    RESTARTS   AGE
payments-5f7c8d9b6-q2w4e   0/1     Pending   0          4m

Events:
  Type     Reason            From               Message
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 3 Insufficient vpc.amazonaws.com/pod-eni. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

A Pod matched by a SecurityGroupPolicy gets its own branch ENI. A webhook adds a vpc.amazonaws.com/pod-eni: 1 request to the Pod, and each node offers as many as its instance type allows (9 on an m5.large, 18 on an m5.xlarge). When they're used up the Pod isn't placed at all: it's Pending, not ContainerCreating. Prefix delegation doesn't raise this limit.

Confirm

# Branch ENIs each node offers (empty = not set up for security groups for Pods)
kubectl get nodes -o custom-columns='NAME:.metadata.name,TYPE:.metadata.labels.node\.kubernetes\.io/instance-type,PODENI:.status.allocatable.vpc\.amazonaws\.com/pod-eni,TRUNK:.metadata.labels.vpc\.amazonaws\.com/has-trunk-attached'
# Which Pods use them: every Pod a SecurityGroupPolicy selects
kubectl get securitygrouppolicies -A

If some nodes show no pod-eni at all: the instance type isn't supported (no t family instance is), ENABLE_POD_ENI isn't true on aws-node, or the cluster role lacks AmazonEKSVPCResourceController.

Fix

More branch ENIs: add nodes, or use larger supported instance types; the limit per type is in the VPC resource controller's limits.go. Karpenter counts pod-eni when it picks instance types.
Fewer Pods that need one: narrow the SecurityGroupPolicy's selector to the Pods that really need their own security group.
It's still a scheduling failure, so the Pod stuck in Pending playbook applies to the rest of the message.

Prevent

Plan branch ENIs per node like max Pods: count the Pods with security groups that can land on each node.
Branch ENIs take their IPs from the node's subnet too, so they count against free addresses.

Understand it

Security groups for Pods

Out of Pod IP Addresses on Amazon EKS

1Start here

Run this first

2Decision tree

3Causes and fixes

max Pods is higher than the node's ENIs can give IPs for

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

The subnet is out of free IP addresses

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

The warm pool holds IPs that no Pod uses

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

Prefix delegation is on, but the subnet has no free /28

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

aws-node's IAM role can't call the EC2 APIs it needs

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

Security groups for Pods: no branch ENI left

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

4Diagnose it

Related

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it