Amazon EKS
Running Amazon EKS at scale — standalone field guides for the failures that actually bite in production, plus a structured 15-part fleet-scale curriculum at the end. Opinionated golden path, honest tradeoffs, every claim sourced. Jump to the topic you're fighting today, or read the full series for the whole picture.
Networking & DNS
3 postsIP planning, packet paths, and service discovery — where EKS networking quietly hits a wall.
Your Compute Scaled Out, Now DNS Is Timing Out: Taming the CoreDNS Bottleneck
A traffic surge scales your pods flawlessly, then the logs fill with 'i/o timeout' and CoreDNS pins at 100% CPU. The culprit is rarely CoreDNS itself — it's the default ndots:5 quietly turning one lookup into eight, and UDP resolution racing the kernel's conntrack table into flat 5-second stalls. Here's the amplification math, and the fix: NodeLocal DNSCache plus the cluster-proportional-autoscaler, then cutting the fan-out at the source with ndots:2, FQDNs, and autopath. Every claim sourced.
The Custom Networking Trap: Asymmetric Routing Across a Transit Gateway
You enabled custom networking to stop running out of IPs. Pod-to-pod works, egress works — then a pod tries to reach RDS across a Transit Gateway and the connection just hangs, packets vanishing though the routes look right. Here's how splitting a node into two networks makes replies return on the wrong path, why strict rp_filter silently drops them, and the two fixes: mask the pod IP with default SNAT, or make the round trip symmetric (return routes + TGW appliance mode). Every claim sourced.
Beyond the Defaults: Scaling Amazon EKS Past the VPC CNI Limits
Out of the box, every pod gets a real VPC IP — and that generosity is a hidden ceiling: nodes wedge pods in Pending at 40% CPU because they ran out of IP addresses, not compute. Here's the mechanism (max-pods = ENIs × (IPs per ENI − 1) + 2, so a t3.medium caps at 17 pods), the low-risk first fix (prefix delegation, a /28 of 16 IPs per ENI slot), the enterprise safeguard (custom networking onto non-routable 100.64.0.0/10 to protect scarce corporate IPv4), and when IPv6 deletes the problem outright. Every claim sourced.
Security & Access
2 postsWorkload identity and secure cross-account ingress without static credentials.
Decoupling Network Ingress: Architecting Cross-Account ALBs for Amazon EKS
Your cluster lives in a locked-down Compute account; your internet-facing ALBs, certs, and DNS live in a monitored Edge account. Here's how a developer deploys a plain Kubernetes Ingress and the platform wires up a secure ALB across the account boundary — a credential-free STS chain (EKS Pod Identity role chaining or IRSA), a cross-account TargetGroupBinding so the cluster only registers pod IPs, and an IngressClassParams that hides the whole thing. Not a single static IAM key. Every claim sourced.
Goodbye IRSA, Hello EKS Pod Identity: Why You Should Migrate in 2026
IRSA was the gold standard for zero-trust pod-to-AWS access. But an OIDC provider per cluster and trust-policy sprawl across dozens of clusters is an operational tax no platform team wants to keep paying. Here is what EKS Pod Identity actually changes — a node-local agent talking to the EKS Auth API, identity decoupled from cluster lifecycle, native cross-account — and where IRSA still earns its keep. Every claim sourced.
Compute, Scaling & Performance
2 postsKarpenter, Spot volatility, and the Linux-kernel limits that throttle your pods.
Why Your P99 Latency Spikes on EKS While CPU Sits at 40%
Your service is slow, P99 latency spikes into seconds, yet average CPU is flat at 40% — no restarts, no OOM, no exceptions. It's being silently strangled at the Linux kernel level by the mechanism meant to protect it: the Completely Fair Scheduler. Here's how a CPU limit becomes a per-100ms budget the kernel enforces at a cliff, the throttle-ratio metric that exposes it, and why dropping CPU limits (while keeping requests) is usually the fix. Every claim sourced.
The Spot Storm: Tuning Karpenter to Survive a Region-Wide Capacity Crunch
Running production on EC2 Spot via Karpenter is the ultimate cost cheat code — until a capacity crunch fires a storm of interruption notices, your controller hammers the EC2 API, and Client.RequestLimitExceeded freezes every scale-up. Here's how Karpenter bypasses ASGs to provision via CreateFleet, why narrow NodePools cause the thundering herd, and the resilient design — broad requirements, price-capacity-optimized allocation, and a pause-pod over-provisioning buffer — that keeps the savings without the outage. Every claim sourced.
Storage & Node Resilience
2 postsSurviving AZ blackouts, stuck volumes, and disk-pressure evictions without losing data.
DiskPressure Evicted Your Database to Reclaim a Few Gigabytes of Logs
When a worker node's root disk fills with container logs and image layers, the kubelet flips DiskPressure, fails image garbage collection, and starts killing pods to save itself — and left to defaults it can kill your most critical stateful pod. Here's the eviction algorithm (the nodefs/imagefs signals and thresholds), the hardening fix (isolate /var/lib/containerd and /var/lib/kubelet onto a dedicated volume; Bottlerocket does it by default), and the two levers the ranker sorts on: PriorityClass and ephemeral-storage limits. Every claim sourced.
Surviving an AZ Blackout: The EBS Multi-Attach Lockout on Stateful EKS
When an Availability Zone goes dark, Kubernetes reschedules your stateful pod to a healthy AZ in seconds — but the EBS volume it needs is still exclusively latched to a node that no longer answers, and by design nothing rips it loose for six minutes, so the pod hangs in ContainerCreating. Here's the failure mechanics (single-AZ EBS, the attach/detach controller's data-integrity guard) and the topology-aware binding, node fencing (the out-of-service taint), and per-AZ replication that engineer around it. Every claim sourced.
GitOps & Delivery
3 postsArgo CD at scale, drift control that survives an incident, and how the control plane is built.
GitOps Drift Control That Survives a 2 a.m. Incident
Strict ArgoCD self-heal and live incident response pull in opposite directions — the 2 a.m. kubectl fix that reverts in ten seconds. The answer isn't loosening enforcement; it's shrinking the reconciled surface (secrets to ESO, toggles to a flag store, HPA fields to ignoreDifferences) and adding one scoped break-glass lever, so self-heal can stay strict because there's nothing legitimate left for it to fight. Every claim sourced.
GitOps at Fleet Scale: Hundreds of EKS Clusters Without Losing Control
A deeper, more current companion to the fleet series' GitOps chapter: how to run dozens or hundreds of EKS clusters through GitOps without losing control — Argo CD's single pane versus Flux's distributed pull, the GitOps bridge that wires them to your IaC, the scaling ceilings that actually bite (the controller shards to 10k+ apps; the UI is the softer limit), and the December 2025 managed EKS Capability for Argo CD that changes the build-versus-buy math. Every claim sourced.
Argo CD Core Architecture: How the Control Plane Is Built
A design-level tour of how Argo CD is actually built: the small services behind it — API server, repo-server, application-controller, Redis, Dex, ApplicationSet and notifications controllers, and the v3.x commit-server — the single reconciliation loop that ties them together, and why there is no database. Follows a change from a git push all the way to running Pods, with each component's inputs and outputs drawn out. Real vendor-logo diagrams and numbered flow throughout.
Worked Examples
1 postOne realistic application, drawn end to end onto a fleet platform.
The Fleet-Scale Series
15 postsThe 15-part curriculum on running EKS by the hundred — read in order, from the reference architecture to cost, upgrades, and migration.
Managing Hundreds of EKS Clusters: The Fleet Reference Architecture
The opener to an 18-part series on running Amazon EKS at fleet scale. Once you run clusters by the hundred, the job stops being "operate a cluster" and becomes "operate a platform that emits clusters." This piece lays out the whole platform shape — GitOps-driven hub-and-spoke — and maps where every later article fits. Every claim sourced.
Many Small EKS Clusters or a Few Large Ones? The Fleet Decision
Part 2 of the fleet-scale EKS series, and the first hard architecture decision — the one every later article assumes you've made. The real tradeoff is blast radius and isolation against the per-cluster cost floor and operational overhead (300 clusters is ~$21,900/month in control-plane fees alone). Here's the cost math, the scale ceilings, the tenancy question, and a decision tree you can pick a side with and defend in a review. Every number sourced.
Provisioning an EKS Fleet: Terraform, Blueprints, eksctl, or Cluster API?
Part 3 of the fleet-scale EKS series. This is where clusters come into existence at scale — and where the tool you pick to create a cluster becomes the tool you own for its whole 26-month life. Here is what provisions what, why AWS now says don't build the core cluster with Blueprints, and how the IaC layer hands off cleanly to GitOps. Every claim sourced.
GitOps at Fleet Scale: Argo CD Hub-and-Spoke vs Flux Distributed
Part 4 of the fleet-scale EKS series — the operational heart of the golden path. Once you run dozens or hundreds of EKS clusters, the layer that keeps them identical, and lets you change all of them safely from one place, is GitOps. Here is how to choose between Argo CD's single pane and Flux's distributed pull, how the provisioning bridge from Part 3 feeds it, and when to run both. Every claim sourced.
Golden Paths for an EKS Fleet: The Self-Service Platform Layer
Part 5 of the fleet-scale EKS series. Parts 1–4 built the fleet — a reference architecture, a cluster-count decision, IaC provisioning, and the GitOps layer that keeps every cluster identical. This is where that machinery becomes a product a developer can consume: a paved road where the compliant way to get a cluster, a namespace, or an app is also the fastest way, with no ticket in the middle. Covers the golden-path template and the managed-vs-self-managed add-on decision. Every claim sourced.
Policy-as-Code Across an EKS Fleet: Kyverno, Gatekeeper, and the Audit-to-Enforce Path
Part 6 of the fleet-scale EKS series — the guardrail layer of the golden path. Once GitOps (Part 4) keeps hundreds of clusters identical, policy-as-code is what stops any one of them from running a privileged pod. Here is how to choose between Kyverno and OPA/Gatekeeper, the baseline every fleet should enforce, and the audit-then-enforce discipline that rolls a rule out to the whole fleet without taking it down. Every claim sourced.
Designing a Single EKS Cluster Right: The Greenfield Reference Architecture
Part 7 of the fleet-scale EKS series. Parts 1–6 designed the platform that stamps out clusters; this one designs the thing being stamped — one greenfield cluster you would be happy to replicate three hundred times, and the compute-model choice (managed node groups vs Fargate vs Karpenter vs Auto Mode) you have to defend to do it. Every claim sourced.
EKS Networking at Fleet Scale: IP Planning Before It Bites
Part 8 of the fleet-scale EKS series. At one or two clusters, IP addressing is an afterthought; at hundreds, running out of addresses is the most common avoidable outage there is — and the one you can't fix after the fact, because a cluster is welded to the VPC it was born in. Prefix delegation, secondary CIDRs, IPv6, and how to size the subnets before they bite. Every claim sourced.
Connecting an EKS Fleet: Load Balancers, Pod Security Groups, and Service Mesh
Part 9 of the fleet-scale EKS series. Once you run clusters by the hundred, 'how does traffic reach this service?' splits into four separate decisions — getting traffic in, isolating a single pod, walling off tenants, and identity between clusters. This wires all four (cross-cluster NLB via the Load Balancer Controller, security groups for pods, NetworkPolicy) and argues against reaching for a service mesh before you actually need one. Every claim sourced.
EKS Identity at Fleet Scale: Pod Identity vs IRSA, and Access Entries vs aws-auth
Part 10 of the fleet-scale EKS series. Every cluster inherits two identity decisions the day it is created — how a pod reaches AWS, and how a human or controller reaches the cluster. Parts 3, 5, and 7 kept saying 'wire identity in at creation' and deferred the detail here: pick EKS Pod Identity over IRSA per workload, and move a fleet off the deprecated aws-auth ConfigMap safely. Every claim sourced.
EKS Secrets and Supply Chain at Fleet Scale: KMS, ESO, CSI, and the GitOps Question
Part 11 of the fleet-scale EKS series — closing the security triad. Identity decides who you are; policy-as-code decides what you may run; secrets and supply chain decide what you may read and what you ship. How to encrypt Secrets at rest without ever deleting the KMS key that bricks the cluster, how to choose between External Secrets Operator, the Secrets Store CSI Driver, and Sealed Secrets/SOPS, and how to make all of it fit a GitOps fleet. Every claim sourced.
Autoscaling an EKS Fleet: Karpenter, Consolidation, and the Spot Question
Part 12 of the fleet-scale EKS series. Karpenter is the fleet standard for autoscaling, but it comes with a sentence AWS won't sign: there is no SLA, and you hold the pager. How it schedules best-fit nodes, how you express capacity with NodePools and EC2NodeClasses, how far to push Spot, and when Cluster Autoscaler or Auto Mode is the better call — with every borrowed benchmark labeled as someone else's production, not an AWS promise.
Observing an EKS Fleet: One Pane Over Hundreds of Clusters
Part 13 of the fleet-scale EKS series. How you see across hundreds of clusters without drowning in cost or cardinality: Amazon Managed Prometheus and Grafana as the single pane, promxy to aggregate many workspaces, Container Insights and Network Observability for the network view — and the cardinality discipline that keeps the metrics bill from exploding at fleet scale. Every claim sourced.
Upgrading an EKS Fleet: The Version Treadmill and the Extended-Support Trap
Part 14 of the fleet-scale EKS series. Every cluster is on a support clock, and the clock ends in a bill — standard support lasts 14 months, extended support adds 12 more at six times the price. The version calendar (stamped and re-verifiable), the skew rules you cannot break, the scanners that catch a deprecated API before it breaks a deploy, and the blast-radius-controlled rollout that upgrades a whole fleet without an outage or a surprise invoice. Every claim sourced.
What an EKS Fleet Actually Costs: The Four Billing Streams and Per-Tenant Showback
Part 15 of the fleet-scale EKS series. Every earlier decision — how many clusters, which autoscaler, when to upgrade — shows up here as a number on the invoice. The four independent streams an EKS bill is made of, where the money actually goes at fleet scale (300 clusters is ~$21,900/month in control-plane fees before a single pod runs), and how to attribute every dollar back to the team that spent it. Every claim sourced.
No posts match — try a different keyword or clear the filters.