Managing Amazon EKS at Fleet Scale · Part 11 of 18

EKS Secrets and Supply Chain at Fleet Scale: KMS, ESO, CSI, and the GitOps Question

This closes the security triad. Identity decides who you are; policy-as-code decides what you may run; secrets and supply chain decide what you may read and what you ship. Here is how to encrypt Secrets at rest without ever deleting the KMS key that bricks the cluster, how to choose between External Secrets Operator, the Secrets Store CSI Driver, and Sealed Secrets/SOPS, and how to make all of it fit a GitOps fleet.

Why secrets are a different problem at fleet scale — three things you have to get right

At one cluster, secrets are a chore you can do by hand. At hundreds, they become a design decision, because every shortcut you take multiplies by the size of the fleet. This is the third leg of the security triad the series has been building: identity (Part 10) decides who you are, policy-as-code (Part 6) decides what you may run, and this article decides what you may read and what you ship.

There are two distinct problems hiding under the word “secret,” and most confusion comes from mixing them up.

• At rest. A Kubernetes Secret is stored in etcd, the cluster’s backing database. Left alone it is only base64-encoded, which is encoding, not encryption. The question here is how those bytes are encrypted on disk.
• In delivery. A workload needs the secret’s value at runtime. The question here is how it gets there — synced from an external manager, mounted into memory, or decrypted from Git — and what each route exposes along the way.

KMS answers the first. External Secrets Operator, the Secrets Store CSI Driver, and Sealed Secrets/SOPS answer the second, each with a different tradeoff. On top of both sits the supply chain: the images your pods run and the provenance you can prove. Get all three consistent across the fleet and you have a real secrets posture; get any one of them wrong on a few clusters and you have a quiet liability you will find during an audit. The rest of this article takes them in that order — at rest, then delivery, then the GitOps question, then the supply chain.

Locking secrets at rest — envelope encryption and the KMS key you must never delete

Start with the good news: on recent clusters this is largely handled for you. EKS version 1.28 and later apply default envelope encryption to API data at rest, Secrets included, so the bytes in etcd are encrypted without you configuring anything.¹ On any cluster older than 1.28, that is not automatic, and you should add a customer-managed KMS key yourself. This is current as of 2026-06-30 — re-verify the version boundary at publish, because default-behaviour facts like this roll forward.

Envelope encryption means the Secret is not encrypted directly by the KMS key. Instead a local data key encrypts each Secret, and the KMS key encrypts — wraps — that data key (Figure 1). It is efficient, and it means the KMS key never has to touch the plaintext. But it also creates a hard dependency: without the KMS key, the wrapped data key cannot be unwrapped, and the Secret cannot be read.

Adding a customer-managed key to an older cluster is a small IAM exercise with an outsized failure mode. The cluster’s role needs kms:DescribeKey and kms:CreateGrant on the key, the change is a one-way door — you can enable KMS encryption but not turn it back off — and the key becomes part of the cluster’s critical path.¹ The headline caveat is worth stating bluntly: if you ever schedule that key for deletion, you have bricked the cluster. AWS enforces a waiting period before a KMS key is destroyed, which is your one chance to cancel; treat any deletion request against a cluster’s key as an incident.

One more thing that trips people up: encryption at rest is not access control. It protects the bytes if someone gets at etcd or a backup; it does nothing about who can run kubectl get secret and read the value in plaintext. RBAC is the primary control for that — least-privilege roles that keep Secret access down to the workloads and people who need it. Envelope encryption is defence in depth layered underneath, not a substitute.²

How a secret reaches a pod — three delivery patterns, three blast radii

Now the runtime question: a native Kubernetes Secret is fine for a single cluster, but on a fleet you usually keep the source of truth somewhere central and deliver from there. Three patterns dominate, and they differ mostly in where the secret ends up living — which is the same as asking what an attacker gets if they compromise that place (Figure 2).

External Secrets Operator (ESO) keeps the source of truth in a real secrets manager. A controller watches an ExternalSecret resource, reads the value from AWS Secrets Manager (or Parameter Store, Vault, and others), and writes a normal Kubernetes Secret into the cluster.³ Your workloads consume a plain Secret and don’t know the difference, and rotation in the manager syncs down automatically. The tradeoff is that the value does land in etcd — which is fine, because that is exactly what KMS envelope encryption from the previous section protects. This is the default I reach for when a central manager is already the system of record.

The Secrets Store CSI Driver keeps the secret out of etcd altogether. It mounts secrets into the pod as a volume backed by tmpfs — memory, not disk — so the value is present only for the pods that mount it and is never written to the cluster datastore.⁴ That is the pattern to reach for when a compliance rule or threat model says the secret must never be persisted in etcd. The cost is a little more moving machinery (the driver plus a provider) and the fact that a mounted file is a slightly different consumption model than an environment variable or a native Secret.

Sealed Secrets and SOPS keep an encrypted copy in Git. Sealed Secrets encrypts a Secret against a controller’s public key so only the in-cluster controller can decrypt it;⁵ SOPS encrypts the values in a YAML/JSON file (commonly against a KMS key) so the file is safe to commit and gets decrypted at deploy time.⁶ Both let you put the whole desired state, secrets included, in the repo without ever committing plaintext — which is precisely the property a pure GitOps workflow wants, and the subject of the next section.

The GitOps question — secrets in Git without plaintext

Here is the tension the fleet’s golden path forces. The whole point of GitOps (Part 4) is that Git is the single source of truth: Argo CD or Flux reconcile every cluster to what the repo says, so a change is a reviewed commit and a rollback is a revert.⁷ That model wants everything in Git — and a secret is part of a workload’s desired state. But a plaintext secret in Git is a plaintext secret in every clone, every fork, and every backup of that repo, forever. So you cannot just commit it, and you cannot leave it out without breaking the “Git is the truth” promise.

There are two honest ways to resolve it, and they map onto the patterns you just saw.

• Put the secret in Git, encrypted. Sealed Secrets and SOPS make the ciphertext safe to commit, so the repo really does hold the complete desired state and a new cluster bootstraps from Git alone. This is the cleanest fit for a pure GitOps model, and the natural choice for edge or air-gapped clusters where you don’t want a runtime dependency on a central manager.
• Put a reference in Git, keep the secret in a manager. With ESO or CSI, what lives in Git is an ExternalSecret or a SecretProviderClass — a pointer that names the secret, not its value. Git stays the source of truth for configuration; the secrets manager stays the source of truth for values. This is the better fit when you already run a central manager, want one place to rotate and audit, and are comfortable that clusters depend on it at sync time.

The deciding question is where you want the system of record for secret values to be. If it is Git, reach for Sealed/SOPS. If it is a real secrets manager, reach for ESO or CSI and keep only the reference in Git. Both are legitimate; what is not legitimate is a plaintext Secret manifest committed to satisfy the tooling. That is the one move this section exists to rule out.

Guarding the supply chain — scan the image, enforce the baseline, verify provenance

Secrets protect what a workload can read. The supply chain protects what a workload is — the image it runs and whether you can trust where it came from. At fleet scale this has to be enforcement, not guidance, and it rides the same admission-policy layer as everything else in Part 6. Three controls carry most of the weight.

• Scan images for known vulnerabilities. Amazon ECR can scan on push and continuously re-scan, so a CVE disclosed after an image was built still surfaces against images already in your registry.⁸ Wire the result into policy: an image with critical findings shouldn’t be admissible on a production cluster.
• Enforce the Pod Security Standards baseline. This is the same baseline from Part 6 — no privileged containers, no host namespaces, no privilege escalation.⁹ A workload that carries a secret and also runs privileged is a much larger blast radius than either problem alone, which is why the baseline belongs on the same fleet-wide gate.
• Prefer signed images with provenance. Signing lets an admission policy require that an image was built by your pipeline and not swapped out, and provenance metadata records how it was built. Even before you enforce it, capturing and verifying signatures narrows “any image that pulls” down to “images we can attest to.”

The connecting idea is that none of this should live in a per-team pipeline that a team can skip. Scanning thresholds, the PSS baseline, and signature requirements are admission policy, expressed once and fanned out to every cluster through GitOps — the same audit-then-enforce discipline Part 6 uses for any new rule. That is what makes a supply-chain guarantee actually hold across hundreds of clusters instead of on the ones that remembered to opt in.

Which pattern to reach for — the decision, and the one key you must never delete

Put the pieces together and the whole article collapses into a short set of decisions (Figure 3). Encryption at rest and delivery are separate choices; make both deliberately.

The delivery choice comes down to one question — where should the secret live? If the answer is a central manager you already trust, use ESO and let it sync into native Secrets. If the answer is “nowhere on disk, ever,” use the CSI driver and keep it in tmpfs. If the answer is “in Git with everything else,” use Sealed Secrets or SOPS so the copy in the repo is ciphertext. There is no single winner; there is a fit for each requirement, and mature fleets often run more than one.

What does not flex are the two constants. On any cluster older than 1.28, add a customer-managed KMS key so Secrets are encrypted at rest — and keep least-privilege RBAC as the primary control, because encryption protects the bytes, not the read path. And put the supply-chain guardrails — image scanning, the PSS baseline, signed provenance — on the same fleet-wide admission gate as your policies, so they hold on every cluster and not just the diligent ones.

If you carry one sentence out of this article, make it this: choose a delivery pattern that fits your GitOps model, and never, under any circumstances, put a cluster’s KMS key on a deletion path — that mistake has no undo. Do both, and secrets stop being the quiet liability in a fleet and become just another part of the golden path you can hand to hundreds of teams.