Managing Hundreds of Amazon EKS Clusters at Fleet Scale · Part 8 of 18

EKS Networking at Fleet Scale: IP Planning Before It Bites

At one or two clusters, IP addressing is an afterthought. At hundreds, running out of addresses is the most common avoidable outage there is — and the one you can't fix after the fact, because a cluster is welded to the VPC it was born in. This is how to size the subnets before they bite.

Why a busy fleet runs out of addresses before it runs out of compute

The failure looks like a compute problem and isn't. A deployment scales up, new pods sit in Pending, and the events show InsufficientCidrBlocks — the subnet has no more IP addresses to hand out. The nodes have CPU and memory to spare. What they've run out of is address space.

This happens because of how the Amazon VPC CNI works: every pod gets a real, routable VPC IP address from the node's subnet, drawn from a pool the node pre-attaches to its elastic network interfaces. That's what makes EKS pod networking fast and observable — a pod is a first-class citizen on the VPC — but it also means your pods and your nodes are spending from the same finite pool of subnet addresses.

At one cluster you never notice. At fleet scale the arithmetic turns hostile: hundreds of clusters, each with dozens of nodes, each node claiming a block of IPs whether or not its pods use them. Stamp out a fleet from a template that gave every cluster a tidy /24 per subnet and you'll meet the ceiling in production, on the busiest cluster, at the worst time. IP exhaustion is the number one avoidable at-scale outage precisely because it's invisible until the subnet is full.

The rest of this piece is about doing the address math before the fleet does it for you — and about the levers that buy headroom, in the order you should reach for them.

The address math you owe yourself before sizing a subnet

Start with what a single node can hold. Without prefix delegation, a node's pod capacity is capped by its network interfaces: ENIs × (IPs per ENI − 1) + 2. The subtraction is the interface's own primary address; the + 2 covers the node itself. Both numbers come from the instance type, and they are smaller than people expect.

Take the canonical example the series uses throughout: an m5.large. Run that formula and you get 29 usable pod IP addresses per node.¹ Not 110, not 250 — 29. That is the real per-node pod ceiling on that instance until you change the CNI's behaviour.

Two other limits sit above the per-node number and are easy to forget. Kubernetes itself caps a node at 110 pods by default (250 for instances with more than 30 vCPUs), for scheduler and kubelet stability reasons rather than IP reasons.¹ And on Nitro instances the total device attachment budget — EBS volumes, ENIs, and NVMe combined — is 28, which quietly limits how many interfaces (and therefore how many IPs) a node can attach at all.⁷ Figure 2 lays the three numbers side by side.

The point of the math is planning, not trivia. Multiply the real per-node IP count by your densest node count, add headroom for rolling upgrades that briefly double a node group, and you have the subnet size you actually need — usually far larger than the template default.

How prefix delegation buys headroom — and the subnet it demands

Prefix delegation is the first lever, and for most fleets it's the one that ends the problem. Instead of attaching pod IP addresses one at a time, the VPC CNI (version 1.9.0 and later) attaches a /28 prefix — 16 contiguous addresses — to each interface slot.² A single ENI slot now backs 16 pods instead of one.

The effect on the canonical node is dramatic. That same m5.large jumps from 29 usable pod IPs to a theoretical ceiling of about 431.² You won't reach 431, because the Kubernetes 110-pod cap (250 above 30 vCPUs) stops you first — and that's the whole point. Once the address ceiling is well above the pod cap, IP addresses stop being the constraint and pod density becomes a scheduling decision again.

There's an operational bonus that matters at scale. Attaching a prefix takes under a second, versus roughly ten seconds to attach a whole new ENI, so nodes reach pod-ready state faster during scale-up.²

The catch is in the word contiguous. A /28 prefix can only be assigned from sixteen consecutive free addresses. In a subnet that's been chewed up by other workloads, the addresses exist but not in an unbroken run — and the CNI fails to attach the prefix with the same InsufficientCidrBlocks you were trying to avoid. Fragmentation, not raw scarcity, is what bites here.

The fix is structural: give pods a dedicated subnet whose whole job is to hand out /28 prefixes, kept free of anything that would fragment it.² That's the pod subnet in Figure 1. Turn prefix delegation on before you feel density pressure, not after — retrofitting it onto a live, fragmented subnet is exactly the situation it's meant to prevent.

Where you find IP space when routable addresses run out — secondary CIDRs

Prefix delegation solves density on a node. It doesn't create routable address space that isn't there. In a real organization the VPC's primary CIDR is carved from a corporate range that has to stay unique and routable across on-prem, other VPCs, and partners — and pods, which churn constantly and never need to be reached from outside, are a wasteful thing to spend that scarce routable space on.

The lever here is custom networking with a secondary CIDR. You attach a second, non-routable range to the VPC — the carrier-grade 100.64.0.0/10 block is the usual choice — and configure the CNI to place pod IPs there while nodes keep their routable addresses.³ Pods now spend from an effectively private pool that never collides with the corporate range, and your scarce routable space goes only to things that actually need to be routed.

• Nodes keep routable addresses from the primary CIDR — they're reachable across on-prem, other VPCs, and partners, so they earn that scarce space.
• Pods draw from the non-routable secondary range — they churn constantly and never need to be reached from outside, so they don't.

This used to require fiddly per-AZ ENIConfig objects. From VPC CNI 1.18.0 onward, Enhanced Subnet Discovery is the default: the CNI finds the right pod subnets by tag automatically, so custom networking is far less error-prone to run across a fleet.³ (Version current as of 2026-06-30; re-verify before you rely on it.)

The tradeoff is honest and worth stating. Because those pod IPs are non-routable, anything that must reach a pod directly from outside the VPC — some ingress paths, certain mesh or cross-cluster patterns — now needs a NAT or load-balancer hop it wouldn't have needed on routable pod IPs. For the vast majority of workloads, which are reached through a load balancer anyway, that's a non-issue. Multi-cluster connectivity, where pods do talk across boundaries, is its own article later in the series.

The one-way door — IPv6 at creation, and why the VPC is forever

Every lever so far is a way to stretch IPv4. The way to stop playing the game entirely is IPv6, where the address space is large enough that pod exhaustion simply doesn't happen. On an IPv6 cluster the CNI assigns addresses from a range so vast that the density math above stops mattering.⁴

But IPv6 comes with the hardest constraint in this whole article: it must be enabled when the VPC and cluster are created. There is no converting an IPv4 cluster to IPv6 afterward.⁴ So IPv6 is a greenfield decision — excellent for a new fleet you expect to grow past IPv4's limits, unavailable as a rescue for one that's already tight.

That irreversibility is the theme of the section, and it's bigger than IPv6. An EKS cluster is permanently bound to the VPC and subnets it was created in. You cannot move a running cluster into a larger or differently-shaped VPC.⁵ If a cluster outgrows its address plan, the remedy is to build a new cluster in a properly-sized VPC and migrate workloads to it — which, across a fleet, is real cost and real risk.

This is why networking is the decision the earlier articles told you to reserve for. The greenfield reference architecture (Part 7) deliberately over-provisions VPC and subnet space up front for exactly this reason, and the fleet reference architecture (Part 1) flags IP planning as one of the three decisions that are expensive to reverse. Subnet shape is a one-way door. Walk through it deliberately.

What public IPv4 now costs a fleet — and why nodes belong in private subnets

Addressing isn't only a capacity question at fleet scale; it's a line item. Since February 2024, AWS charges $0.005 per hour for every public IPv4 address in use, in-use or idle.⁶ One address is a rounding error. Across a fleet of hundreds of clusters, each node that carries a public IP is a small recurring charge multiplied by a large number. (Pricing current as of 2026-06-30; re-verify at publish time.)

The design that answers both the cost and the security question is the same one good practice already recommends: put nodes in private subnets and give them egress through a NAT gateway, one per Availability Zone for resilience.⁷ Nodes then need no public IPv4 at all; only the NAT gateways and any public-facing load balancers do. That's the layout in Figure 1 — private node subnets, a dedicated pod subnet on the secondary CIDR, and a NAT per AZ.

NAT gateways aren't free either — they carry an hourly and a per-GB charge — but one-per-AZ is a small, predictable cost, and consolidating public IPv4 behind them is almost always cheaper at fleet scale than a public address on every node. Cost management across the fleet, including the four EKS billing streams, is its own deep-dive later in the series.

Which lever to pull, and when

Here's the decision to carry out of this piece, in the order you should apply it (Figure 3). The levers aren't alternatives you pick between once; they layer, and the right starting point depends on your single tightest constraint.

Building greenfield and you expect to outgrow IPv4? Choose IPv6 at VPC creation and most of the rest of this article becomes moot. This is the only decision you truly cannot revisit, so make it first and make it deliberately.

Staying on IPv4 with scarce routable space? Add a secondary 100.64.0.0/10 CIDR with custom networking so pods stop spending your routable range. On VPC CNI 1.18.0 or later, Enhanced Subnet Discovery makes this fleet-friendly.

Routable space is fine, you just need pod density? Turn on prefix delegation and give pods a dedicated, contiguous /28 subnet. That alone takes most clusters off the exhaustion path.

And underneath all three, one rule holds regardless: dedicate contiguous /28 pod subnets, put nodes in private subnets behind a NAT per AZ, and size the VPC generously — because the one thing you can't buy back later is the shape of the VPC itself. Get that right at creation and InsufficientCidrBlocks stays a diagram in an article instead of a page at 2 a.m.

The next article picks up where a well-shaped VPC leaves off: multi-cluster connectivity — load balancing, service mesh, and wiring pods across cluster and account boundaries.