Managing Hundreds of Amazon EKS Clusters at Fleet Scale · Part 13 of 18

Observing an EKS Fleet: One Pane Over Hundreds of Clusters

A single cluster is easy to watch. Once you run hundreds, "add a dashboard" stops working and observability becomes its own scaling problem — where the answer to "how is prod?" has to come from one pane, and the metrics bill quietly grows faster than the fleet.

Why a fleet breaks the single-cluster dashboard — from one Prometheus to hundreds

Observing one EKS cluster is a solved craft. You run Prometheus, point Grafana at it, and every question about that cluster has an answer in one place. The instinct at scale is to do the same thing many times: a Prometheus per cluster, a Grafana per cluster. That instinct is exactly what breaks.

The moment you operate clusters by the hundred, three things go wrong with the copy-paste approach at once:

• You now run and upgrade hundreds of stateful Prometheus servers, each a pet.
• There is no single place to ask "how is the fleet?" — the view is smeared across hundreds of panes.
• Every one of those Prometheus servers is sized, scaled, and paged on independently — its own day-2 fleet inside your fleet.

So the real problem is not collecting metrics. It's two harder things: how you store metrics without operating a storage tier by hand, and how you see across clusters from one place. Fleet observability is a scaling problem in its own right, and the rest of this piece is about solving those two without the cost curve getting away from you.

The managed default that skips the ops — AMP for storage, AMG for the pane

Start with the managed stack, because it removes the first problem entirely. Amazon Managed Service for Prometheus (AMP) is a serverless, Prometheus-compatible metric store: you keep PromQL and the Prometheus data model, but there is no Prometheus server to size, scale, or upgrade.¹ Clusters send metrics in with ADOT (the AWS Distro for OpenTelemetry) or plain Prometheus remote-write, and AMP holds the time series.

Amazon Managed Grafana (AMG) is the pane on top — managed Grafana with AMP as a first-class data source, so you query with the PromQL you already know.¹² The split is the point: AMP owns storage and query, AMG owns visualization and access. Neither is a server you patch.

For a single cluster this is a pleasant convenience. For a fleet it's the thing that makes the rest tractable — you've turned "operate hundreds of Prometheus servers" into "send metrics to a managed endpoint." What you have not solved yet is the pane. One workspace per cluster still leaves you with hundreds of data sources and no single view. That's the fleet-aggregation problem, and it's where the real design decision lives.

One pane over many workspaces — promxy aggregation vs per-team isolation

Managed storage still leaves you with a choice about topology: how many AMP workspaces, and how they roll up into a view. There are two honest patterns, and they pull in opposite directions.

Aggregate for one pane. Put clusters into a shared set of AMP workspaces and place promxy in front of them — an open-source Prometheus-compatible proxy that fans a single PromQL query out across many backends and merges the results.³ Grafana then points at one promxy data source and sees the whole fleet as if it were one Prometheus. This is what buys you the single pane: "P99 latency across every prod cluster" becomes one query, not a tab per cluster.

Isolate for tenancy. Give each team or each environment its own AMP workspace. Now blast radius is contained, access control is per-workspace, and cost falls out cleanly per tenant — the workspace is the unit of attribution.⁴ The price you pay is the single pane: a viewer has to know which workspace holds the answer, and cross-fleet queries get awkward.

If that tension feels familiar, it should. It's the same shape as the Argo CD versus Flux decision from Part 4: a central hub gives you one place to see and drive everything, at the cost of a shared blast radius; a distributed model contains the blast radius but loses the single pane of glass.⁵ Aggregation is the hub model for metrics; per-team workspaces are the distributed one.

And the per-team-workspace choice is not really an observability decision at all — it's your tenancy model showing up again. If you already run separate clusters or accounts per team for isolation and cost attribution (the many-vs-few call from Part 2, enforced by the guardrails in Part 6), per-team workspaces just extend that boundary into metrics. Let the tenancy model you already committed to decide this, rather than inventing a second, conflicting one for observability.

Seeing the network, not just the pods — Container Insights and Network Observability

Metrics tell you a pod is slow; they rarely tell you why when the cause is the network between pods. Two AWS features fill that gap, and both feed the same pane.

Container Insights collects cluster, node, pod, and container metrics — CPU, memory, disk, and the control-plane signals — as curated CloudWatch metrics, so you get the health of the cluster itself without hand-building every exporter.⁶ It's the "is the cluster healthy" layer beneath your application metrics.

The newer piece is Container Network Observability, which adds a service map, a flow table, and network performance metrics — and crucially exposes them in OpenMetrics / Prometheus format, so the same AMP + AMG stack scrapes and renders them alongside everything else.⁶ That's what lets you answer "which service-to-service path is dropping packets" on the same pane as your latency graphs, instead of in a separate tool. (Feature availability current as of 2026-06-30 — re-verify against the CloudWatch docs before relying on it, as these observability features move quickly.)

The design takeaway is to keep the pane singular. Network visibility that lands in its own console recreates the many-panes problem you just solved. Because these signals are Prometheus-format, they don't have to.

Where the metrics bill actually explodes — cardinality, not cluster count

Here's the part the getting-started guides gloss over, and the reason fleet observability goes wrong in production. At scale, your metrics cost is driven by cardinality — the number of unique time series — far more than by the number of clusters. AMP bills primarily on samples ingested and stored, and every distinct combination of label values is a separate series to ingest, store, and query.⁷

Cardinality explodes multiplicatively, which is why it ambushes teams. One metric with a pod label, a path label, and a status_code label doesn't cost you three series — it costs you every combination that ever appears. Add a high-churn label like pod name or request ID across a fleet where pods are created and destroyed constantly, and a single innocent-looking metric becomes millions of series. The bill grows with your label choices, not your traffic.

The levers that control it are all about limiting series before they're stored, and they compound:

• Label discipline at the source. Drop or relabel high-cardinality labels (pod name, instance ID, raw URL paths, request IDs) in the scrape or remote-write config, before they ever reach AMP. This is the highest-leverage lever because it attacks the multiplication at its root.
• Recording rules. Pre-aggregate the expensive queries into a small set of stored series, so dashboards read cheap rolled-up metrics instead of re-scanning millions of raw ones. You pay to compute once, not on every panel refresh.
• Per-workspace scoping. Splitting workspaces per team or environment (the isolation pattern above) also caps cardinality per workspace — a runaway metric in one team's namespace can't inflate the whole fleet's series count. Isolation and cost control turn out to be the same lever.
• Sampling and retention. Not every metric deserves full-resolution, long-retention storage. Scrape noisy debug metrics less often, and keep raw high-resolution data only as long as you actually query it.

The honest summary: cardinality discipline is the difference between fleet observability that costs a predictable, budgeted amount and one whose bill grows every time a developer adds a label. It's not a cleanup you do once — it's a standing policy, ideally enforced the same way as your other golden-path defaults.

What to actually build — the fleet-observability decision

Pull it together into a default and a small set of thresholds you can act on.

The default: AMP for storage and PromQL, AMG for the pane, ingest via ADOT or remote-write, Container Insights and Network Observability feeding the same stack. Start here for any fleet; it removes the operate-your-own-storage problem before you tune anything else.

The one rule to carry out of this piece: let your existing decisions decide. Your GitOps model and your tenancy model already answered the single-pane-versus-isolation question — align observability with them instead of relitigating it. And whichever topology you pick, budget cardinality from day one, because it's the only part of this bill that grows on its own.

The next article turns to day-2 upgrades — how you keep a fleet inside standard support and off the 6× extended-support price, without a hundred hand-run upgrades.