Managing Amazon EKS at Fleet Scale · Part 5 of 18

Golden Paths for an EKS Fleet: The Self-Service Platform Layer

Parts 1–4 built the fleet: a reference architecture, a cluster-count decision, IaC provisioning, and the GitOps layer that keeps every cluster identical. This is where that machinery becomes a product a developer can consume — a paved road where the compliant way to get a cluster, a namespace, or an app is also the fastest way, with no ticket in the middle.

Why the platform only pays off when the compliant path needs no ticket

Here is the verdict up front: everything Parts 1–4 built — the reference architecture, the many-vs-few decision, IaC provisioning, and the GitOps layer — is plumbing until a developer can consume it without asking you. The platform earns its keep the day a team gets a compliant cluster, namespace, or app on their own.

The trap is subtle. Most platforms make the compliant path the slow one: fill in a request, wait for a review, wait for a human to wire up identity and policy and observability by hand. So teams route around it. They spin up their own cluster, skip the baseline, and now you have a fleet where every cluster is a little different — which is the exact problem the fleet was supposed to solve.

A golden path fixes the incentive. It is the well-lit, fully-paved default route to production: the fastest way to get a cluster is also the way that comes pre-wired with everything security and platform require. Compliance stops being a gate you pass and becomes a property you inherit.

The word that matters is paved road, not cage. A cage says “you may only do this.” A paved road says “this is so easy and complete that you’ll want to.” The road is optional in principle; it wins because taking it is less work than not. When the golden path is genuinely the least-effort option, standardization happens by choice, and that is the only kind that lasts.

This is the first article of the series’ second wave. It turns the spine into a product; Part 6 hardens it with policy-as-code, and Part 7 uses it as the default for a greenfield build.

What a new tenant gets for free — the golden-path template

The golden path is not a document. It is a template — a concrete bundle that a new cluster or tenant comes pre-wired with, delivered automatically. Think of it as the house default from Part 1, packaged so a team never assembles it by hand.

Five things ride in every template, and they map to the decisions the rest of this series makes in depth:

• Compute — Karpenter, pre-configured. A default NodePool and EC2NodeClass so the tenant gets best-fit, Spot-aware, self-consolidating nodes without writing a single autoscaling config.⁵ Karpenter is the fleet compute standard (Part 12); the tenant just gets it.
• Identity — EKS Pod Identity plus access entries. Workloads get AWS permissions through EKS Pod Identity, the recommended default over IRSA for in-cloud clusters — no per-cluster OIDC provider, reusable roles across the fleet.³ Human and controller access comes through access entries, which replace the deprecated aws-auth ConfigMap.⁴ Both are baked in (Part 10 goes deep).
• Policy — a baseline, in audit then enforce. A pod-security baseline — no privileged containers, no host namespaces, no privilege escalation — ships with the template, delivered through the same GitOps pipeline and rolled out in audit mode before it enforces.⁶ This is the teaser to Part 6; here it matters that the baseline is part of the default, not a later add-on.
• Observability — the agents are already running. The telemetry agents that ship metrics and logs off the cluster are part of the template, so a tenant is visible from the moment it exists rather than after someone remembers to instrument it (Part 13 covers the backend).
• Add-ons — the standard set, wired. The core networking and DNS add-ons plus the house standard set, provisioned and configured consistently across every tenant — the subject of the next two sections.

The delivery mechanism is the GitOps layer from Part 4. AWS’s current guidance is a stable core Terraform module for the cluster itself, with EKS Blueprints patterns bootstrapping the add-ons on top — and explicitly not using Blueprints to provision the core cluster in production.¹ The template is the shape of that add-on layer, and Argo CD ApplicationSets are what stamp it onto each cluster (Figure 1).

How the template reaches every cluster — ApplicationSet generators

A template is only a product if it deploys itself. The delivery workhorse is the Argo CD ApplicationSet, the same construct Part 4 leaned on for fleet rollout, now carrying the golden path instead of a single app.

Instead of hand-writing one Argo Application per component per cluster — the N×M problem that sinks naive setups — you write one ApplicationSet whose cluster generator enumerates the clusters Argo knows about, filters them by label, and stamps out the right set of Applications for each match.² “Every tenant tagged tier=standard gets the golden-path stack” becomes a few lines of YAML.

The join that makes this automatic is the GitOps bridge from Part 3. IaC writes each new cluster’s metadata — its labels, endpoint, and the IAM role ARNs its add-ons should assume — into the Argo CD cluster secret.¹ Registering the cluster is what enrolls it: on the next reconcile, the generator sees a new match and deploys the entire template. Cluster number 141 gets the same baseline as cluster number 1, with no human wiring per cluster.

That is what turns “golden path” from a wiki page into an operational fact. The path is not a checklist someone follows; it is what the generator does by default. Drift is designed out because the same manifests render onto every matching cluster, and self-heal reverts anyone who edits a live resource by hand back to what Git says.

Who should own each add-on — let AWS run it, or run it yourself

Every add-on in the template arrives one of two ways, and choosing between them is the most consequential decision in this article. An EKS managed add-on is one AWS installs and operates: AWS owns its lifecycle, its version compatibility with the control plane, and its upgrade path. A self-managed add-on is one you install yourself — typically a Helm chart delivered through GitOps — and own end to end.

The default is managed, and the reasoning is ownership cost. A managed add-on is one less thing your team upgrades, tests for version skew, and debugs at 2 a.m. AWS validates its compatibility with each Kubernetes version, and its health and available versions surface in EKS Cluster Insights, the same built-in scanner that flags upgrade blockers across the fleet.⁷ At a hundred clusters, “AWS tracks this for me and warns me before it breaks” is worth a great deal.

Self-managed earns its place only when managed can’t do the job. Reach for it when you need a version AWS hasn’t published yet, a configuration flag the managed add-on doesn’t expose, or a non-standard build — a custom CNI configuration, a patched image, a bleeding-edge feature. The price is that you now own the upgrade cadence, the compatibility testing, and the failure modes for that component, on every cluster it runs on (Figure 2).

Why the three core add-ons decide your upgrade order

Three add-ons are special: kube-proxy, the VPC CNI, and CoreDNS. Every EKS cluster runs them — they are the networking and DNS substrate the rest of the cluster stands on — and they are the canonical case for letting AWS own the lifecycle, because their compatibility is tied directly to the Kubernetes version.

That coupling shows up as an upgrade order you don’t get to reorder. When you move a cluster to a new version, the sequence is control plane first, then the add-ons in the order kube-proxy → VPC CNI → CoreDNS, then the node groups.⁸ Managed add-ons make this tractable: AWS knows which add-on version pairs with which control-plane version, and Cluster Insights warns you before a mismatch bites.

This is a teaser, not the full treatment — Part 14 owns day-2 upgrades, where this order and the version-skew rules (a node can trail the control plane, but only within a bounded window) get the depth they deserve. The reason it belongs here is that the golden-path template is where you decide these three are managed, and that decision is what makes the eventual fleet-wide upgrade a routine operation instead of a research project.

How a developer gets a compliant tenant without filing a ticket

Now the self-service mechanics — the part that makes this a platform rather than a well-organized ops team. The goal is a request flow where a developer expresses what they want, and a compliant tenant appears, with no human standing in the middle wiring things up.

The cheapest version that actually works is a pull request. The developer doesn’t need a bespoke portal; they open a PR against the fleet repo — a small manifest that says “team-c, standard tier, these environments.” That PR is the request. On merge, the ApplicationSet generator picks up the new tenant and renders the whole golden-path template onto it (Figure 3).

For larger organizations, that PR flow often sits behind a thin platform API or a portal — a form that generates the same manifest, so a developer never edits YAML. That is a nicety, not the substance. The substance is underneath: whether the form calls an API or a human opens the PR, the request resolves to a manifest that the template renders and GitOps syncs. Start with the PR flow; add the portal when the audience outgrows it.

The scaffolding side matters too. A golden path includes starter templates for the things a team actually builds — a service repo pre-wired with a Dockerfile, a Helm chart, a CI pipeline, and the manifests that register it with the platform — so “new service” is a scaffold command, not a week of copy-paste from a sibling team’s repo. The tenant template and the app scaffold are the same idea at two levels: a compliant default you get by asking, not by assembling.

How you make the compliant way the only easy way — guardrails and the exit

The last piece is what separates a paved road from a suggestion. The golden path works because the guardrails make non-compliant configuration impossible to ship on the fast path, not merely frowned upon in a review.

That is the policy gate in Figure 3. The pod-security baseline from the template — no privileged containers, no host namespaces, no privilege escalation — is enforced in the pipeline, so a manifest that violates it is rejected before it can sync, whether it came from a self-service PR or a platform engineer.⁶ Roll it out in audit mode first to see what would break, then flip to enforce.

The difference between “we recommend” and “the pipeline won’t let you” is the difference between a guideline and a guardrail. Part 6 is the full treatment of policy-as-code — Kyverno versus OPA/Gatekeeper, baseline versus restricted — but the design decision starts here: the baseline lives in the template and the gate enforces it.

Guardrails that only say no, though, breed the workarounds they were meant to prevent. The honest complement is a documented exit. Some team, someday, will have a legitimate need the golden path doesn’t cover — a workload that needs a privileged container, a build the standard add-ons can’t run. The answer is not “never,” and it is not “quietly go around us.” It is a reviewed, deliberate off-ramp: request the exception, justify it, get a scoped approval, and carry the extra ownership that comes with leaving the paved road.

A good exit protects the golden path in both directions. It keeps the 95% on the road because the road is genuinely the easiest route, and it gives the 5% a sanctioned way off that you can see, review, and revisit — instead of a shadow fleet you find out about during an incident. The exit being rare and visible is what keeps the default meaningful.

So the decision thresholds, plainly:

• Golden path as the default for every team, no exceptions asked.
• Managed add-ons unless you need a version or build AWS won’t expose.
• The baseline enforced in the pipeline, not suggested in a doc.
• A documented, reviewed exit for the genuine off-road case.

Get those right and “compliant” and “fast” stop being a trade-off. That is the whole point of a platform: the right way is the easy way. Part 6 hardens the policy layer this platform enforces, and Part 7 builds a greenfield cluster with this golden path as its default.