Managing Hundreds of Amazon EKS Clusters at Fleet Scale · Part 7 of 18

Designing a Single EKS Cluster Right: The Greenfield Reference Architecture

Parts 1–6 designed the platform that stamps out clusters. This one designs the thing being stamped: one greenfield cluster you would be happy to replicate three hundred times — and the compute-model choice you have to defend to do it.

Why the cluster template is the fleet's real unit of work

The opening parts of this series were about the platform — the GitOps hub, the IaC core module, the golden-path template that emits clusters. But a template is only as good as the design it encodes. Get the unit wrong and you don't get one bad cluster; you get three hundred copies of the same mistake, each one expensive to unwind because the fleet tooling is now actively keeping them all consistent.

So this article zooms all the way in, to a single greenfield cluster, and treats it as a design review. The bar is concrete: could you defend every choice on this one cluster if you knew it would be cloned across the estate? Four decisions carry almost all the weight — compute, data-plane sizing, identity, and the VPC binding — and each one is either cheap to get right now or painful to change once it has been replicated. Figure 1 is the shape we are designing toward; the rest of the piece is the argument for each part of it.

Two of these choices are genuinely permanent. The VPC a cluster is created in can never be swapped out.⁸ And the pods-per-node and IP math you bake into the node template quietly sets the ceiling for how far each cluster scales. Neither shows up in a demo; both show up at fleet scale.

The decision that shapes everything else — who runs your compute

The compute model is the choice you make first and regret longest, because it decides who operates the data plane, how nodes are sized and replaced, and which parts of the bill you can even see. EKS gives you four honest options, and the point of this section is that they are genuinely different tools, not a good-better-best ladder.

Managed node groups are the conservative baseline: EKS manages the lifecycle of an EC2 Auto Scaling group for you, but you still choose instance types up front, you still own the AMI-upgrade cadence, and scaling is per-group rather than workload-aware. That predictability is exactly what regulated environments want — a fixed, pre-approved AMI and a controlled upgrade workflow.

Fargate removes nodes from the picture entirely: each pod runs in its own right-sized, isolated compute sandbox, so there is nothing to patch and strong per-pod isolation by default. The price is real constraints — no DaemonSets, IRSA-only identity (EKS Pod Identity does not cover Fargate pods),⁹ and per-pod pricing that stops being cheap for dense, steady workloads.

Karpenter is the fleet standard, and the rest of this series treats it as the default. It is groupless and workload-aware: it looks at pending pods and provisions best-fit EC2 directly, mixes Spot and On-Demand in a single NodePool, consolidates idle or underused nodes (emptiness-first consolidation landed in v1.5, July 2025), and replaces nodes on drift.¹ The catch is stated plainly by AWS: there is no SLA for Karpenter. You run the controller, you own its failures, and that operational responsibility is the price of its flexibility.

EKS Auto Mode is AWS answering “fine, you run it for me.” It runs Karpenter, the AWS Load Balancer Controller, and the EBS CSI driver off-cluster as a managed component, on Bottlerocket nodes with a 21-day maximum runtime.

You pay a management fee of roughly 12% on top of the underlying EC2 cost for it.² (The ~12% is grounded in AWS’s own pricing examples but varies by instance type — treat it as approximate, current as of 2026-06-30.)

Figure 2 places the four on the one axis that actually matters for a fleet decision — how much of the data plane you operate versus how much AWS does — and the table underneath is the full comparison.

Picking a default you can defend — Karpenter, Auto Mode, and the exceptions

A reference architecture has to commit, so here is the commitment: Karpenter is the default, and the other three are named exceptions you reach for on purpose.

Karpenter earns default status because it optimizes the two things that dominate a fleet — cost and node fit — without a per-node fee and without asking AWS to gatekeep your data plane. One NodePool can span dozens of instance types and blend Spot with On-Demand; consolidation claws back the idle capacity that per-group autoscalers leave stranded. The practitioner evidence points the same way, with the caveat that it is practitioner evidence: Salesforce reported migrating more than 1,000 EKS clusters onto Karpenter,¹⁰ and Grover reported pushing Spot usage to around 80% in production.¹¹ Neither is an AWS benchmark or a guaranteed outcome; both are directional, from teams operating at real scale.

The reason to name it a default rather than a mandate is that “no AWS SLA” is a real sentence. You are taking on operating the controller — its upgrades, its permissions, its failure modes. For most teams that is a fair trade for the control and the savings. For some, it isn’t, and that is what the exceptions are for.

Reach for the alternatives on these thresholds:

• EKS Auto Mode when you would rather AWS operate Karpenter, the load-balancer controller, and the EBS CSI driver than run them yourself, and the roughly 12% premium plus reduced visibility is a price you will happily pay to delete that operational surface.² It is the same engine — you are buying the operations, not a different capability.
• Managed node groups when capacity is stable and predefined, or when a regulated AMI-upgrade workflow — fixed, pre-approved images promoted on a controlled cadence — matters more than best-fit provisioning.
• Fargate when strong per-pod isolation without any node operations is the goal, and the workload tolerates the constraints (no DaemonSets, IRSA-only identity, per-pod pricing).⁹

A note on the bill, because the compute model quietly picks which billing stream you feed. Compute is one of four independent EKS billing streams — separate from the per-cluster control-plane fee — and it bundles EC2 or Fargate charges, EBS, cross-AZ transfer, and $0.005/hour for every public IPv4 address.² Karpenter and managed node groups spend into EC2; Fargate spends per pod; Auto Mode adds its fee on top. The full cost breakdown is Part 15’s job; the point here is that the compute choice is also a cost-shape choice.

Sizing the data plane before you clone it — pods, IPs, and the ceilings

Compute decides who runs the nodes; this decides how much each node can actually hold, and it is the sizing that gets frozen into the template. Two numbers do most of the damage if you ignore them.

The first is pods per node. EKS defaults to a maximum of 110 pods per node — the upstream Kubernetes scalability threshold — and only lifts it to 250 on instances with more than 30 vCPUs.³ That is a scheduler ceiling, and it interacts with a lower, sneakier one: IP addresses.

By default the VPC CNI gives each pod a real VPC IP, and the number of IPs a node can hand out is capped by its ENIs. The canonical example the whole series uses is the m5.large: without prefix delegation it tops out at just 29 pod IPs, far below the 110-pod scheduler limit.³ Your node has CPU and memory to spare and still stops scheduling pods, because it ran out of addresses.

Prefix delegation is the fix, and it belongs in the template from day one. Instead of assigning IPs one at a time, the CNI (v1.9.0+) hands each ENI slot a /28 prefix of 16 addresses, lifting that same m5.large to roughly 431 theoretical IPs — comfortably past the 110-pod cap it will actually hit.⁴ Attaching a prefix takes under a second versus about ten for a fresh ENI, so pod startup gets faster too.

The one condition: prefixes need contiguous /28 blocks, so a fragmented subnet throws InsufficientCidrBlocks. Give pods a dedicated subnet and the problem disappears. Figure 3 is that story in two bars.

Beyond the node sit the cluster-wide ceilings, and a greenfield design should know where they are even if it never touches them. The upstream etcd datastore is limited to 8 GiB, a cluster tops out around 10,000 services (5,000 per namespace), and Nitro instances share a single limit of 28 volume attachments across EBS, ENIs, and NVMe.⁵ None of these bite a small cluster; all of them shape how densely you should pack one before you decide the answer is a second cluster instead. The deep networking treatment — secondary CIDRs, IPv6, custom networking — is Part 8; here it is enough to size the node right and reserve room to grow.

Reserving the network shape you can't take back

Here is the one truly irreversible decision on a greenfield cluster, and it is easy to sleepwalk through: an EKS cluster is permanently bound to the VPC and subnets it is created in. You cannot migrate a cluster to another VPC — not with a flag, not with a support ticket. The only way to “move” is to build a new cluster and shift workloads to it.⁸

That permanence is why subnet layout is a greenfield decision, not a day-2 one. Pick private subnets for nodes across at least two Availability Zones, size the pod subnets for prefix delegation’s contiguous-block appetite, and leave CIDR headroom for the density you sketched in the last section. If IPv6 is on the table — and at fleet scale, for IP exhaustion, it often should be — that too is a create-time choice, covered in Part 8, but flagged here because you cannot convert an IPv4 VPC later either.

The practical move is to treat the VPC/subnet layout as part of the golden-path template itself, provisioned by the same IaC that stamps the cluster, so every cluster inherits a shape you have already decided you can live with permanently. Getting this wrong is not a tuning problem; it is a rebuild.

Baking identity in at creation — Pod Identity and access entries

Identity is the other choice that is cheap at creation and expensive to retrofit, so a greenfield cluster wires it in before any workload lands. Two mechanisms, both set from day one:

• EKS Pod Identity — for pods reaching AWS APIs.
• Access entries — for humans and roles reaching the cluster’s API.

For pods reaching AWS APIs, the default is EKS Pod Identity. It replaces IRSA as the recommended path for clusters running in AWS: no per-cluster OIDC provider to stand up, IAM roles that are reusable across clusters, session tags, and a clean separation between IAM administration and cluster administration.⁶ It needs Kubernetes 1.24 or newer and the EKS Pod Identity Agent add-on — so you enable that add-on at creation rather than bolting it on later. IRSA does not disappear; it is still required for Fargate pods, Windows, and portability to non-EKS or self-managed clusters, and the two can coexist.⁹ For a standard Linux-on-EC2 greenfield cluster, though, Pod Identity is the default.

For humans and roles reaching the cluster’s API, the mechanism is access entries, which replace the deprecated aws-auth ConfigMap. Create the cluster in API_AND_CONFIG_MAP mode (the console default) or go straight to API, and grant access with entries rather than by hand-editing a ConfigMap that has no audit trail and one bad-YAML-edit-from-lockout failure mode.⁷ Access entries take precedence over the ConfigMap for the same principal, and managed node groups and Fargate profiles get their entries created automatically. Authentication-mode transitions are one-way — one more reason to decide it at creation. The full identity deep-dive — the exact IRSA-versus-Pod-Identity boundary — is Part 10; the greenfield rule is simply that both are configured before cluster #1 ships.

What to lock in before you stamp cluster #2

Pull it together into the checklist you would defend in a design review, because everything here is about to be replicated.

Default to Karpenter and own it. Best-fit nodes, one NodePool blending Spot and On-Demand, consolidation on — and eyes open about the no-SLA operational load. Switch to Auto Mode only when you will gladly pay ~12% and give up some visibility to have AWS run the data plane; to managed node groups for stable or regulated capacity; to Fargate for per-pod isolation without nodes.

Size the node before you clone it. Prefix delegation and a dedicated pod subnet from day one, so an m5.large fills to 110 pods instead of stalling at 29, with the etcd, service-count, and Nitro-attachment ceilings noted as the boundaries of how far one cluster stretches.

Decide the permanent things deliberately. The VPC binding and the IPv4-versus-IPv6 call are create-time and forever; put the subnet layout in the template. And wire EKS Pod Identity plus access entries at creation, because identity is the classic thing teams promise to “add later” across a fleet and never cheaply do.

Do that, and you have a cluster you would be comfortable cloning three hundred times — which is the only kind of cluster worth putting in a golden-path template. Part 8 takes the networking layer this article deliberately reserved and goes deep: IP planning, secondary CIDRs, and the connectivity that turns a well-designed cluster into a well-connected fleet.