Why your P99 latency spikes on EKS while CPU sits at 40% — and it isn't your code

Your microservice is slow. P99 latency spikes into whole seconds, yet average CPU sits at a comfortable 40%. There are no restarts, no OOMKilled events, no exceptions in the logs, and no obvious hot loop in the flame graph. You add replicas and it barely helps. You give the pod a bigger CPU limit and it gets worse under the same load. What you're looking at is a workload being silently strangled at the Linux kernel level — by the exact mechanism you added to protect the node: the Completely Fair Scheduler (CFS) and its bandwidth-control quota.

What a CPU limit actually does to your process

Start with the surprising part: limits.cpu: "1" does not mean “this pod may use one core.” It means “this pod may accumulate one core-second of CPU-time per second, rationed in 100-millisecond installments.” Those are very different promises on a multi-core node, and the gap between them is where your latency goes.

When you set a CPU limit, the kubelet writes it into the container's cgroup as two numbers (cgroup v1 naming):

• cpu.cfs_period_us — the length of an accounting window. The default is 100000 µs = 100ms, and Kubernetes never changes it.
• cpu.cfs_quota_us — how much CPU-time the cgroup may run inside each window, computed as limit × 100000 µs.

So a limit of 400m (0.4 CPU) becomes a quota of 0.4 × 100000 = 40000 µs — the container may run for 40ms of CPU-time out of every 100ms. A limit of 1 becomes a quota of 100ms; a limit of 2 becomes 200ms. On cgroup v2 (the EKS AL2023 default) the same thing is expressed in a single file, cpu.max, holding "$QUOTA $PERIOD" — e.g. "100000 100000" for a 1-CPU limit, or "max 100000" when no limit is set.

Here is the crucial detail everyone misses: quota is CPU-time summed across every runnable thread, but it is granted once per period and does not carry over. The kernel refills the bucket to quota at the start of each 100ms window. Your threads drain it. When it hits zero, the scheduler de-schedules the entire cgroup — every thread, on every core — and refuses to run any of them until the next window opens. That enforced idle time is throttling. It is not the OS being unfair; it is the OS doing exactly what you told it to.

Why multi-threaded apps get wrecked

Now put a real runtime on it. A JVM, a Go binary, or a Node.js app with a worker pool doesn't run one thread — it spins up threads sized to the machine it sees. The JVM reads Runtime.availableProcessors(), Go sets GOMAXPROCS, libuv sizes its pool — and on a 64-vCPU node they may all conclude they have 64 cores to play with, regardless of a limits.cpu: "1" the kernel is quietly enforcing.

When a request arrives and 8 of those threads become runnable at once, they land on 8 cores and run in parallel. Parallel execution burns quota 8× faster than wall-clock time: 8 threads consuming 8ms of CPU-time per 1ms of wall time drain a 100ms quota in 12.5ms. The other 87.5ms of that period, all 8 threads sit frozen — no work, no progress, requests queuing — even though seven of the node's cores are idle and available. That is the timeline in Figure 1.

This is why throwing hardware at it backfires. Move the same pod from a 2-vCPU node to a 96-vCPU node and the quota is unchanged, but now more threads run in parallel and exhaust it sooner — so throughput falls and P99 rises on the bigger box. Practitioners have measured exactly this: an NGINX pod capped at 100m served ~1,220 req/s at 195ms P99 on a 2-vCPU node, and only ~455 req/s at 2,500ms P99 on a 96-vCPU node. Same limit, same code, 13× worse tail latency — purely from having more cores to throttle against.

Why your dashboards say everything is fine

The reason this bug survives in production for months is that the metric everyone watches — average CPU utilization — is designed to hide it. Average CPU is total CPU-seconds consumed divided by wall-clock. A throttled pod consumes little CPU (it's frozen most of each period), so its utilization is low. Low CPU reads as “healthy, has headroom.” The truth is the opposite: it's starving. As one widely-cited write-up put it, you can see high throttling and very low CPU utilization at the same time, which is precisely what makes the average-CPU dashboard lie to you.

The three metrics that actually tell the truth

cAdvisor (embedded in the kubelet, scraped at /metrics/cadvisor) exports exactly the counters you need, straight from the cgroup's cpu.stat:

The single most useful number is the throttle ratio — the fraction of periods in which you got throttled. Alert on it, not on the raw counters:

# Fraction of CFS periods a container was throttled, per pod/container.
# Sustained > 0.05–0.10 on a latency-sensitive service = investigate.
sum by (namespace, pod, container) (
  rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m])
)
/
sum by (namespace, pod, container) (
  rate(container_cpu_cfs_periods_total{container!=""}[5m])
)

Two collection gotchas worth knowing. First, filter container!="" — cAdvisor also emits a pod-level roll-up series with an empty container label, and double-counting it skews the ratio. Second, throttled seconds can look scary in isolation (a pod throttled 5ms per 100ms is barely affected), so anchor your alert to the ratio of periods, which normalizes for load. A pod at 90% throttle-ratio is being strangled; a pod at 90% average CPU is merely busy. Learn to tell those two apart on sight.

The fix: keep requests, drop the CPU limit

Once you see that a CPU limit is a budget the kernel enforces cliff-edge, the primary fix almost writes itself: keep requests.cpu, remove limits.cpu. This is the “requests are all you need” philosophy, and it is not reckless — it's a precise split of two jobs that people wrongly assume must travel together.

• requests.cpu is for scheduling and fairness. The scheduler uses it to bin-pack pods onto nodes, and the kernel converts it into a CPU share weight (cgroup v1 cpu.shares, v2 cpu.weight). Under contention, shares decide who gets the core — your request is a guaranteed floor, proportional to what you asked for. Removing the limit does not remove this protection.
• limits.cpu is only for a ceiling. It's the CFS quota — and on a shared, bursty service it mostly manufactures the throttling above while providing protection you rarely actually need, because shares already stop any one pod from starving its neighbors under load.

Drop the limit and the pod keeps its guaranteed share and becomes free to use idle node capacity when it's there. No quota, no cliff, no artificial cap — the tail latency that CFS was adding simply disappears. Figure 3 shows the before/after.

apiVersion: v1
kind: Pod
metadata:
  name: latency-sensitive-api
spec:
  containers:
  - name: api
    image: my-api:1.0
    resources:
      requests:
        cpu: "500m"      # scheduling floor + CPU-share weight (kept)
        memory: "512Mi"
      limits:
        memory: "512Mi"  # keep a MEMORY limit (OOM is a real, hard boundary)
        # NOTE: no cpu limit -> no CFS quota -> no artificial throttling.

apiVersion: v1
kind: Pod
metadata:
  name: latency-sensitive-api
spec:
  containers:
  - name: api
    image: my-api:1.0
    resources:
      requests:
        cpu: "500m"
        memory: "512Mi"
      limits:
        cpu: "1"         # <-- becomes cpu.max "100000 100000":
        memory: "512Mi"  #     100ms quota per 100ms, throttled cliff-edge.

When a CPU limit still earns its place

“No CPU limits” is a strong default for latency-sensitive services on trusted, first-party workloads — it is not a universal law. Keep (or add) a CPU limit when:

• Hard multi-tenancy / untrusted code. If you can't trust a pod to behave, a limit is a blunt but real cap on how much it can steal, independent of shares.
• Batch / cost predictability. Throughput jobs where you want deterministic cost and don't care about tail latency are fine capped — and capping them protects the online services sharing the node.
• Chargeback or noisy-neighbor SLAs. When a contract says a tenant gets “exactly N cores, no more,” a limit is how you enforce it.
• Preventing a runaway from evicting others. Shares only arbitrate under contention; on an otherwise-idle node a bug can consume everything until real contention starts. A limit bounds that.

The nuance: the risk of no limit is that a misbehaving pod can consume idle capacity — but it can never drop another pod below its guaranteed request, because shares kick in the moment there's contention. For most in-house microservices that trade is strongly worth it. Reserve limits for the cases above, and set them generously (well above steady-state) so they cap catastrophe, not normal bursting.

Cgroups, pinning, and the kernel bug you may still be carrying

Removing limits is the 90% answer. The remaining 10% — strict isolation, and the historical kernel behavior that made throttling worse than the math — is worth understanding so you don't misdiagnose it.

cgroup v1 vs v2: same idea, different files

Modern EKS nodes (AL2023) run cgroup v2, where the whole quota lives in one file. When you're on a node debugging a throttled pod, read the truth directly:

# The quota+period, one file: "$QUOTA $PERIOD" (microseconds).
# "100000 100000" = 1 CPU limit;  "max 100000" = no limit.
$ cat /sys/fs/cgroup/kubepods.slice/…/cpu.max
100000 100000

# The verdict: is this cgroup living against its wall?
$ cat /sys/fs/cgroup/kubepods.slice/…/cpu.stat
nr_periods 48211
nr_throttled 41880       # 87% of periods throttled -> strangled
throttled_usec 3517840000

# cgroup v1 splits the same values across separate files.
$ cat /sys/fs/cgroup/cpu/…/cpu.cfs_period_us   # 100000  (100ms)
$ cat /sys/fs/cgroup/cpu/…/cpu.cfs_quota_us    # 100000  (= 1 CPU)

$ cat /sys/fs/cgroup/cpu/…/cpu.stat
nr_periods 48211
nr_throttled 41880
throttled_time 3517840000   # nanoseconds (v2 renames this throttled_usec)

The kernel bug: when throttling was worse than the math

For years, pods throttled more than the quota arithmetic predicted, because of a genuine scheduler bug (kernel.org #198197). CFS distributed quota to per-CPU run-queues in slices and let unused per-CPU slices expire, so a well-behaved, lightly-threaded app could be throttled while nowhere near its aggregate quota. Dave Chiluk's fix — commit de53fd7aedb1, “removing expiration of CPU-local slices” — landed in Linux 5.4 and was backported by distros. If you're on a kernel older than 5.4 (some long-lived AMIs, older Ubuntu 18.04 images), you inherit the worse behavior; the practical takeaway is check uname -r before assuming your throttling is purely the quota design.

Two follow-ons matter. The earlier 4.18 fix (512ac999) addressed a related timer-drift issue, and CFS burst (cpu.cfs_burst_us, Linux 5.14) lets a cgroup bank a bounded amount of unused quota to absorb spikes — useful, but Kubernetes does not configure it by default, so don't count on it. On current EKS (kernel 6.1, cgroup v2) the bug is gone; the design-level throttling from spending a 100ms budget too fast is not, which is why the requests-only fix still matters.

Strict isolation without a quota cliff: the static CPU Manager policy

Some workloads — low-latency trading, high-throughput caches, DPDK-style packet processors — genuinely need dedicated cores with no scheduler contention and no cache-thrash from being bounced between CPUs. Don't reach for a tighter CPU limit for this; reach for CPU pinning. The kubelet's static CPU Manager policy gives Guaranteed pods that request integer CPUs exclusive, pinned cores via cpusets — isolation by ownership, not by throttling.

# /etc/kubernetes/kubelet/kubelet-config.yaml  (per node)
# Enable via EKS AL2023 NodeConfig or a self-managed launch template.
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cpuManagerPolicy: static          # default is "none" (CFS quota only)
# static policy REQUIRES a non-zero reservation for system/k8s daemons:
kubeReserved:
  cpu: "500m"
systemReserved:
  cpu: "500m"
cpuManagerReconcilePeriod: 10s

Two operational notes. First, changing the policy on a live node means draining it, stopping the kubelet, deleting /var/lib/kubelet/cpu_manager_state, then restarting — the state file is not auto-migrated. Second, only pods that are Guaranteed QoS with whole-number CPU (requests.cpu == limits.cpu, an integer) get exclusive cores; everything else shares the leftover pool. This is the one place you deliberately do set a CPU limit — because here the limit equals the request and buys you a dedicated core rather than a throttling wall.

Pitfalls & field notes

If you remember one thing: a CPU limit is a per-100ms budget the kernel enforces at a cliff, not a rate limiter. For most microservices, requests guarantee the floor and shares protect the neighbors — the limit only adds tail latency. Watch the throttle ratio, keep memory limits, and pin (don't cap) the workloads that need real isolation.