Managing Amazon EKS at Fleet Scale · Part 6 of 18

Policy-as-Code Across an EKS Fleet: Kyverno, Gatekeeper, and the Audit-to-Enforce Path

The guardrail layer of the golden path. Once GitOps (Part 4) keeps hundreds of clusters identical, policy-as-code is what stops any one of them from running a privileged pod. Here is how to choose between Kyverno and OPA/Gatekeeper, the baseline every fleet should enforce, and the audit-then-enforce discipline that rolls a rule out to the whole fleet without taking it down.

What policy-as-code actually guards — the gate every pod passes through

Start with the verdict: at fleet scale you cannot rely on every team to configure their pods safely, so you move the check off the humans and into the cluster. Policy-as-code is a set of rules, stored in Git, that an engine evaluates on every request to the API server — and rejects the ones that break the rules before they ever schedule.

The mechanism is Kubernetes admission control. Both Kyverno and OPA/Gatekeeper register as a validating admission webhook: when someone applies a Deployment, the API server calls the engine, the engine checks the spec against your policies, and it returns admit or reject.¹² A rejected pod never runs. That is the whole point — the guardrail sits in the request path, not in a nightly scan that finds the privileged pod after it has been live for a week.

This is distinct from identity, and the distinction matters. Part 10 handles authentication: access entries decide which IAM principal can talk to the cluster, and EKS Pod Identity decides what AWS APIs a pod’s workload can call.⁶⁷ Policy is the other half. Identity is who you are; admission is what you are allowed to run once you are in. A pod can be perfectly authenticated and still be rejected for asking to be privileged.

Why this belongs at the platform layer and not in each team’s pipeline: a CI check that a team owns is a check a team can skip. An admission policy the platform owns applies to every pod on every cluster, whether it arrived through the golden path, a GitOps sync, or someone’s kubectl apply at 2am. That is the property you are buying — enforcement that does not depend on anyone remembering.

Which engine to reach for — Kyverno’s YAML vs Gatekeeper’s Rego

The two mature choices are Kyverno and OPA/Gatekeeper, and the honest answer is that they solve the same problem with different ergonomics. Pick by how your team thinks and what your policies need to express (Figure 1).

Kyverno is the Kubernetes-native option. Policies are plain YAML resources, so a team that already writes manifests writes policies the same way — no new language. It can also mutate (inject a missing securityContext, add a label) and generate (create a default NetworkPolicy in every new namespace), which turns a policy from “reject the bad” into “also supply the good.” It is CNCF incubating.¹

Gatekeeper is the expressiveness option. It builds on the Open Policy Agent and uses Rego, a purpose-built policy language. Rego costs you a learning curve, and it buys you rules that are awkward or impossible to express in YAML — cross-field conditionals, set logic, policies that reason over the whole object. Because it is OPA underneath, the same policy language reaches beyond Kubernetes into CI, Terraform, and API authorization. It is CNCF graduated.²

My default for a fleet whose first job is the security baseline is Kyverno — the policies read like the manifests the platform team already maintains, and generation lets you hand each new namespace a safe default instead of only punishing the unsafe one. Reach for Gatekeeper when a real policy needs Rego, or when you already run OPA elsewhere and want one language everywhere. Running both is not indecision; it is using each where it fits.

The baseline every fleet should enforce — no privileged, no host access, no escalation

Before you argue engines, agree on what the policies should say. The community already did the hard work here: the Kubernetes Pod Security Standards define three tiers — privileged (no restrictions), baseline (block the known-dangerous), and restricted (hardened best practice).³ The baseline is the floor every fleet should hold, and it is small enough to state plainly.

• No privileged containers. A privileged container can do almost anything the host can; it is a container escape waiting for a bug.
• No host namespaces. Deny hostNetwork, hostPID, and hostIPC — sharing the node’s network or process space breaks the isolation the pod is supposed to give you.
• No privilege escalation. Block allowPrivilegeEscalation: true, so a process can’t gain more privileges than its parent.
• No dangerous host mounts and added capabilities. Keep pods off hostPath mounts into sensitive paths and off capabilities like SYS_ADMIN.

Encode that once as policy and it becomes the property you actually want: no pod on any cluster in the fleet can be privileged, no matter who wrote its manifest. Figure 2 shows the gate doing its job — a compliant pod admitted, a privileged one rejected with a reason the author can read.

Where you land on baseline versus restricted is a judgment call. Baseline is the non-negotiable floor. Restricted — forcing non-root users, read-only root filesystems, dropped capabilities — is stronger, but it breaks more existing workloads, which is exactly why the rollout discipline in the next section matters. You aim for restricted; you get there without an outage by moving in audit mode first.

How you roll a rule out without an outage — audit first, watch, then enforce

This is the part that separates a safe policy program from a self-inflicted incident. A new baseline flipped straight to enforce on a live fleet will reject the next deploy — and quite possibly block a rescheduling pod during a node failure — for every workload that was quietly non-compliant. On a fleet you have no idea how many that is. So you never enforce cold.

Both engines support a report-only mode: Kyverno policies run in Audit, Gatekeeper constraints use an enforcementAction of dryrun or warn.¹² In that mode the engine evaluates every pod and records what would have been rejected, while admitting all of it. That report is your map of the blast radius before you arm anything. The rollout is four steps (Figure 3).

Two details make the difference in practice.

• Watch the whole fleet, not one cluster. The point of the audit window is to find the workloads you forgot about, and those live on the clusters you are not looking at. Both engines surface violations as reports you can roll up centrally.
• Use scoped exceptions, not a weakened policy. When a workload genuinely needs an exception, exempt that namespace explicitly rather than watering the rule down for everyone; the exemption is itself in Git, reviewed and visible.

Then flip to enforce — and, because you have watched, you flip knowing what it will catch. If it still surprises you, the revert is one commit, which is the whole reason the next section insists this ships through GitOps.

Ship policy the way you ship everything — one reviewed, revertible commit

A policy is just another Kubernetes resource, so it belongs in the same GitOps pipeline as the rest of the golden path (Part 4). You do not kubectl apply a Kyverno policy onto 200 clusters by hand — you commit it to the fleet repo and let an Argo CD ApplicationSet fan it out to every matching cluster.⁵

That gives policy the same three properties GitOps gives everything else, and they matter more here than almost anywhere.

• Reviewed. A change to the fleet-wide security baseline is a pull request a human approves — not a live edit someone makes under pressure. The audit-to-enforce flip is a one-line diff on the record.
• Fleet-wide and consistent. The cluster generator applies the policy to every cluster that matches its label selector, so a brand-new cluster inherits the full baseline the moment it is registered. No cluster is quietly missing a rule.
• Revertible. If a freshly-enforced policy breaks deploys, you revert the commit and the controllers roll every cluster back. That single revert is the safety net that makes it reasonable to enforce at all.

This is also how you stage the rollout from the previous section: point the ApplicationSet at env=dev clusters first, promote by label to staging, then prod — the same policy, gated only by which clusters match at each step. Audit-to-enforce and dev-to-prod are the same machinery, driven from Git.

Where policy keeps soft multi-tenancy safe — quotas, network policy, per-namespace rules

Policy is also what makes a shared cluster safe to sell as a product. The cheap tenancy model is namespace-as-a-service: many teams share one cluster and one control plane, each in its own namespace.⁴ It is cost-efficient — it avoids paying the per-cluster control-plane fee for every team — but a namespace alone is a soft boundary, and admission policy is a large part of what firms it up.

Three controls, per tenant, do most of the work:

• Resource quotas and limit ranges cap what a namespace can consume, so one noisy tenant can’t starve the others of CPU, memory, or object count.
• Network policy is deny-by-default between namespaces, so a tenant’s pods can’t reach another tenant’s services unless you allow it — and Kyverno’s generation can create that default-deny policy in every new namespace automatically.
• Per-namespace admission rules pin each tenant to the baseline (and let you apply a stricter profile to the tenants that need one).

Stacked together, these turn “a namespace” into “a bounded, policed tenant.” That is soft multi-tenancy done properly, and for a large fraction of internal platforms it is enough.

When policy isn’t enough — the jump to a separate cluster or account

Now the honest limit, because pretending policy solves everything is how teams get burned. A namespace with quotas, network policy, and admission rules is a strong administrative boundary. It is not a hard security boundary against a hostile or compromised workload — tenants still share one API server, one set of nodes, and one kernel. If a container escapes, the namespace line does not stop it.

So the threshold is this: soft multi-tenancy holds when your tenants are mutually-trusting teams inside one organization. Stop trying to make policy do isolation’s job and give the tenant its own cluster, or its own account,⁴ the moment any one of these is true:

AWS’s own scalability guidance points the same way for environments: use separate clusters for dev, test, and prod rather than leaning on namespaces for hard environment isolation.

That decision — namespace versus cluster versus account — is the many-vs-few-clusters tradeoff from Part 2, and the greenfield reference architecture in Part 7 designs the tenancy model end to end. The rule of thumb to carry out of this article: policy makes a shared cluster safe enough for trusting tenants; a separate cluster or account is what you reach for when “safe enough” isn’t.

Put the whole chain together and the guardrail layer reads as one decision. Pick the engine that fits your team — Kyverno for K8s-native YAML and generation, Gatekeeper for Rego and reuse, both when it pays. Enforce the Pod Security Standards baseline. Roll every rule out audit-first, watch the fleet, then flip to enforce as a reviewed Git commit. Use quotas, network policy, and admission rules to keep soft tenancy safe — and know the line past which you hand a tenant its own cluster. Do that, and the golden path is safe to give to hundreds of teams.