Your EKS nodes run out of IPs long before they run out of compute — here's how to fix it

Out of the box, EKS feels magical — until a flash sale scales your microservices from 10 to 150 replicas and half of them wedge in Pending. You check the nodes and nothing looks wrong: CPU and memory sit at a lazy 40%, nothing throttled, nothing OOM. Then the kubelet log: FailedCreatePodSandBox ... failed to assign an IP address to container. The node didn't run out of compute — it ran out of IP addresses. That's a ceiling most teams never see until it drops on them in production.

Why a full node still can't place a pod — reading the failure

Before changing anything, work out which wall you hit — the two share an error text but have different fixes:

Failed create pod sandbox: rpc error: ... networkPlugin cni
failed to set up pod ... network: add cmd: failed to assign
an IP address to container

The message means one thing — the CNI had no IP to hand the pod — but for two independent reasons. Tell them apart with one look at the subnet's free-IPv4 count in the VPC console:

One trap: EC2 will launch an instance into a nearly-full subnet, and only then does the CNI find no IPs to assign. The instance reports healthy; the pods are stranded.

How the default CNI spends IPs — and where the wall is

On EKS, every pod gets a real, routable VPC IP, not an overlay address — to the rest of your VPC a pod looks exactly like an EC2 instance. That's why Flow Logs, security groups, and existing tooling "just work" for pods, and it's also the root of exhaustion. The aws-node daemonset does it with two parts: a CNI binary that wires up each pod, and ipamd, a daemon that keeps a warm pool of IPs ready and hands them out from ENIs attached to the node.

So the pod count a node can run comes down to two hardware facts of the instance type — how many ENIs it attaches, and how many IPs each ENI supports:

max-pods = (number of ENIs for the instance type
            × (IPs per ENI − 1))
           + 2

The − 1 is the ENI's own primary address (never given to a pod); the + 2 adds back kube-proxy and aws-node, which run hostNetwork on the node's IP. An m5.large (3 ENIs, 10 IPs/ENI) gives (3 × 9) + 2 = 29 pods — on 2 vCPUs and 8 GiB that could run far more modest pods. The IP model, not the silicon, stops you.

A t3.medium full of small pods — sidecars, a mesh, event-driven jobs — would pack 30–50 pods of compute, but pod 18 has no IP. That is the "40% idle, still Pending" symptom in one line.

Fix 1 — give each ENI slot a /28, not a single IP (prefix delegation)

Reach for this one first — it's the least disruptive, and it hits the node ceiling head-on. Flip ENABLE_PREFIX_DELEGATION=true and the CNI gives each ENI slot a whole /28 prefix — 16 IPs — instead of one. ENI and slot counts stay the same, but each slot is now 16 pods, so addressable capacity becomes ENIs × (slots per ENI − 1) × 16 — roughly 432 on an m5.large. You're not meant to run 432 pods on 2 vCPUs: AWS recommends capping max-pods at 110 for ≤30 vCPUs and 250 above that — limits from Kubernetes/EKS scalability testing (kubelet, kube-proxy, control-plane load), not IP availability (per the AWS containers blog). The point is that IPs stop being what stops you; the bottleneck moves back to CPU and memory.

Two hard prerequisites: VPC CNI 1.9.0+ and Nitro-based instances (unsupported on older Xen types). Existing nodes won't pick up prefixes until recycled, so roll them after enabling it. The warm knob is now WARM_PREFIX_TARGET (default 1, one spare /28); WARM_IP_TARGET or MINIMUM_IP_TARGET override it for finer control on tight subnets, at the cost of more EC2 API calls.

kubectl set env daemonset aws-node -n kube-system \
  ENABLE_PREFIX_DELEGATION=true

./max-pods-calculator.sh \
  --instance-type m5.large \
  --cni-version 1.9.0 \
  --cni-prefix-delegation-enabled

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  # ... amiFamily, role, subnetSelectorTerms, securityGroupSelectorTerms ...
  kubelet:
    maxPods: 110        # match your instance density under prefix delegation

The one real caveat — prefix fragmentation

A /28 needs 16 contiguous addresses. On a subnet used for a while — IPs allocated and freed in a patchwork — there may be plenty free but no contiguous block of 16, so prefix attachment fails:

InsufficientCidrBlocks: There are not enough free cidr
blocks in the specified subnet

Keep prefix space contiguous with VPC subnet CIDR reservations, or run the cluster in fresh, dedicated subnets — this is what catches teams retrofitting prefix delegation onto an old, busy cluster.

If you just need to unblock scaling, prefix delegation is the fix — stop here. The rest is for teams protecting scarce routable IPv4 and operating this at fleet scale.

Fix 2 — park pods on non-routable space (custom networking)

Prefix delegation fixes how many pods fit on a node. It does nothing for the other wall: running out of routable corporate IPv4 entirely. When your VPC is carved from a tightly-managed RFC1918 range shared across an org, every pod IP is a scarce routable address someone had to budget for. Custom networking attaches a secondary, non-routable CIDR and puts pods there — the recommended range is the CG-NAT shared space 100.64.0.0/10 (RFC 6598), unlikely to collide with corporate RFC1918. Only the node's secondary ENIs move to it; the primary ENI stays routable, carrying the node plus host-network pods (aws-node, kube-proxy).

The cost — you lose one ENI's worth of pods

Because the primary ENI no longer hands its IPs to pods, the formula loses an ENI:

without custom networking:  (ENIs × (IPs − 1)) + 2
with custom networking:     ((ENIs − 1) × (IPs − 1)) + 2

For an m5.large that's 29 → 20 pods in plain secondary-IP mode — which is exactly why you combine it with prefix delegation: with /28 prefixes on the secondary ENIs, ((3 − 1) × ((10 − 1) × 16)) + 2 = 290 addressable pods (still capped at 110 for compute). The lost primary ENI barely matters once each slot holds 16 IPs.

Egress — how a non-routable pod still gets out

So how does a pod on a non-routable 100.64.x address ever reach the internet or the corporate network? Source NAT on the primary ENI. The CNI default AWS_VPC_K8S_CNI_EXTERNALSNAT=false SNATs any traffic bound outside the VPC's CIDRs to the node's routable primary IP on the way out — the pod's non-routable address never appears on the wire off-VPC.

Standing it up

You configure it with one env var plus an ENIConfig per AZ, but in practice bake it into Terraform. It's disruptive and forward-only — only new nodes carrying the k8s.amazonaws.com/eniConfig label are affected, so plan a node-group replacement (enable, provision fresh nodes, drain the old). before_compute = true on the CNI add-on lands the config before any node joins, so nodes come up already on the secondary CIDR.

kubectl set env daemonset aws-node -n kube-system \
  AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

# auto-assign the right ENIConfig to each node by AZ
kubectl set env daemonset aws-node -n kube-system \
  ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone

apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: us-west-2a
spec:
  securityGroups:
    - sg-0123456789abcdef0
  subnet: subnet-0a1b2c3d4e5f6a7b8

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  secondary_cidr_blocks = [local.secondary_vpc_cidr]  # e.g. "100.64.0.0/16"

  azs = local.azs
  private_subnets = concat(
    [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)],
    [for k, v in local.azs : cidrsubnet(local.secondary_vpc_cidr, 2, k)],
  )
}

module "eks" {
  source = "terraform-aws-modules/eks/aws"

  cluster_addons = {
    vpc-cni = {
      before_compute = true        # configure the CNI before nodes join
      most_recent    = true
      configuration_values = jsonencode({
        env = {
          AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG = "true"
          ENI_CONFIG_LABEL_DEF               = "topology.kubernetes.io/zone"
        }
      })
    }
  }
}

# One ENIConfig per AZ, mapped to the secondary-CIDR subnets
resource "kubectl_manifest" "eni_config" {
  for_each = zipmap(local.azs, slice(module.vpc.private_subnets, 3, 6))

  yaml_body = yamlencode({
    apiVersion = "crd.k8s.amazonaws.com/v1alpha1"
    kind       = "ENIConfig"
    metadata   = { name = each.key }
    spec = {
      securityGroups = [module.eks.node_security_group_id]
      subnet         = each.value
    }
  })
}

Stacking them, and the lighter escapes — what to run at fleet scale

The standard large-cluster topology is both fixes together: /28 prefixes on secondary ENIs on the non-routable CIDR — ~290 addressable pods/node on an m5.large, capped at 110. Beyond that, pick the escape that matches your constraint:

Karpenter — where the max-pods trap moves

Karpenter deliberately sets --use-max-pods=false --max-pods=110, because the ENI-based max-pods-calculator.sh doesn't understand prefixes. That works until it doesn't: in a documented case (#6102) prefix-delegated Karpenter nodes came up with a too-low max-pods — a c7g.medium reporting 8 pods when it should support ~98 — while the identical AMI on a managed node group computed it correctly. Pin maxPods in the EC2NodeClass kubelet block (§4, tab 3), and enable prefix delegation as usual:

kubectl set env daemonset aws-node -n kube-system \
  ENABLE_PREFIX_DELEGATION=true WARM_PREFIX_TARGET=1

Karpenter's general guidance for IP pressure is three levers: enable prefix delegation, keep maxPods at or below real instance density, and use larger subnets (/20+) or steer subnetSelectorTerms away from IP-starved ones.

IPv6 — the clean slate that makes this moot

For a new cluster, AWS best-practices guidance is to consider IPv6 first. A /64 pod subnet holds ~18 quintillion addresses, so exhaustion simply doesn't exist; prefix mode is the default and only option (a /80 per ENI slot), assigned only at node startup, which removes the API-throttling large IPv4 clusters feel. The catches are real: you cannot use custom networking with IPv6 (and don't need to — scarcity is its whole reason), and IPv6 is a one-way decision with dual-stack and app-compatibility implications. Right first choice for greenfield; rarely a quick retrofit.

Pitfalls — the ways these fixes silently don't work

The short version: turn on prefix delegation today, and remember to raise max-pods. Add custom networking when routable IPv4 is the real constraint. Reach for Enhanced Subnet Discovery when you just need a little more routable space. And if you're starting fresh, look hard at IPv6 before you build any of this.