The Spot Storm: tuning Karpenter to survive a region-wide capacity crunch

The pattern starts perfect. You point Karpenter at a broad pool of Spot capacity, watch your compute bill fall 60–70%, and forget the autoscaler exists. Then, on a busy afternoon, EC2 gets tight in your region. A single Spot pool your fleet leans on starts reclaiming instances, and dozens of nodes receive a two-minute interruption notice within the same minute. Karpenter does exactly what you told it to: it reacts to every one of them, launching replacements and terminating the doomed nodes.

And then the logs fill with this:

operation error EC2: CreateFleet, https response error
StatusCode: 503, RequestID: ..., api error RequestLimitExceeded:
Request limit exceeded.

Your controller is now throttled by the very API it depends on to recover. Replacements stop launching, drained pods pile up in Pending, and your "self-healing" cluster is paralyzed at the exact moment it needs to heal. One reported incident saw CreateFleet fail for roughly 70 minutes and clear only after manual intervention.⁶ This is the failure mode nobody tunes for until it happens — so let's tune for it.

Why Karpenter is fast: it bypasses the ASG entirely

To understand the failure, you have to understand the mechanism. The legacy Cluster Autoscaler never talked to EC2 directly. It watched for Pending pods, decided which Auto Scaling Group should grow, and then bumped that ASG's desiredCapacity. The ASG — a separate AWS control plane — was what actually called RunInstances. Every node was a clone of the group's single launch template, so "flexibility" meant maintaining a sprawl of ASGs, one per instance shape.

Karpenter throws that layer away. It watches unschedulable pods, bin-packs them against real instance-type data to find an efficient shape, and calls the EC2 Fleet CreateFleet API itself — no ASG, no launch-template sprawl. A single NodePool can span dozens of instance types across families, sizes, and Availability Zones, and Karpenter picks per launch. That directness is the whole point: it provisions right-sized nodes in seconds.

Interruption handling rides the same directness. Karpenter watches an SQS interruption queue that Amazon EventBridge feeds with EC2 events. The moment a Spot interruption warning arrives — the two-minute notice before EC2 reclaims the instance — Karpenter cordons and drains that node and launches a replacement so the evicted pods have somewhere to land.⁴ In steady state this is elegant. During a storm, "launch a replacement for every interrupted node, right now" is precisely what overwhelms the API.

The failure domain: how a storm becomes a self-inflicted outage

Three things have to line up. Understanding each is what lets you defuse it.

1. The EC2 API is a small token bucket

EC2 throttles per account, per Region, using a token bucket: your bucket holds a fixed number of tokens, each API call spends one, and it refills at a set rate.¹ Exceed it and you get RequestLimitExceeded. The buckets that matter for a scaling controller are the mutating ones, and they are deliberately tight:

Read those refill rates again. Sustained, you get about 5 mutating calls per second once the burst is gone. CreateFleet is a mutating action; each new node is a call. Karpenter typically requests fleets of size 1, and it can fire many concurrently, which the AWS team itself flags as raising throttling risk.⁵ Add the DescribeInstances, CreateTags, and TerminateInstances calls that accompany each node's lifecycle, and a few dozen simultaneous replacements empty the bucket in the first second.

2. Narrow NodePools make interruptions both more likely and more synchronized

Here's the trap that feels like good hygiene. Teams pin a NodePool to one or two instance types — "we standardized on m5.large" — for predictability. But a Spot pool is a specific (instance type × AZ) combination, and Spot interruption risk is a function of how deep that pool is. Constrain to one type and you're drawing from one shallow pool. When EC2 needs that capacity back, every one of your nodes is in the same pool, so they all get reclaimed together. Narrow constraints simultaneously raise interruption odds and guarantee the interruptions arrive as a synchronized herd — the worst possible input to a per-node reconcile loop.

3. The reaction is synchronized and expensive

Now compose them. A pool-wide reclamation drops dozens of interruption events into the queue in the same window. Karpenter reacts to each: drain, CreateFleet a replacement, tag, describe, terminate. The mutating bucket drains, RequestLimitExceeded starts coming back, and — because the SDK retries with backoff but the events keep flowing — the controller can stay throttled far longer than the two-minute notice. Replacements don't launch, drained pods sit Pending, and the cluster is stuck. This is the 70-minute incident, not a hypothetical.⁶

The fix, part one: one broad NodePool, price-capacity-optimized

The single highest-leverage change is counterintuitive if you came from ASGs: stop constraining, start diversifying. The AWS guidance is explicit — "diversification and flexibility are important when using Spot instances: instance types, instance sizes, Availability Zones... Being as flexible as possible enables Karpenter to have a wider choice of spare-capacity pools to choose from."⁴ A broad NodePool draws from many independent Spot pools, so a crunch in any one of them reclaims only a slice of your fleet, not all of it — and Karpenter has somewhere else to go.

Karpenter already asks EC2 Fleet for the smart answer. It uses the price-capacity-optimized (PCO) allocation strategy, which "identifies pools with the highest capacity availability... believed to have the lowest chance of interruption in the near term, and then requests Spot Instances from the lowest priced of these pools."² AWS measured PCO at roughly 6× fewer interruptions than the old lowest-price strategy for about 1% more cost. But PCO can only choose from the pools you allow — starve it with narrow requirements and it has nothing to optimize over.

So write requirements that are wide on purpose, and let on-demand sit in the same pool as a fallback when Spot is genuinely exhausted:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        # Prefer spot; fall back to on-demand when the spot pool is exhausted.
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        # BREADTH: three families across every modern generation = a deep,
        # diverse set of spot pools for PCO to choose from.
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["4"]            # 5th-gen and newer
        # Exclude only the tiniest sizes; keep everything else in play.
        - key: karpenter.k8s.aws/instance-cpu
          operator: Gt
          values: ["3"]
        # Spread across all AZs your subnets cover (pool = type x AZ).
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a", "us-east-1b", "us-east-1c"]
      # Recycle nodes on a ceiling so AMIs/config don't drift forever.
      expireAfter: 336h            # 14 days
  limits:
    cpu: "2000"
  disruption:
    # Reclaim empty/underused nodes to keep the bill honest.
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
    budgets:
      # Cap *voluntary* churn so consolidation never becomes its own storm.
      - nodes: "10%"
  weight: 50

The matching EC2NodeClass stays deliberately boring — it defines how a node is built (AMI, role, subnets, security groups), while the NodePool defines what is allowed:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  role: "KarpenterNodeRole-prod-cluster"
  amiFamily: AL2023
  amiSelectorTerms:
    - alias: al2023@latest
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "prod-cluster"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "prod-cluster"
  metadataOptions:
    httpTokens: required        # IMDSv2 only
  tags:
    managed-by: karpenter

If you do nothing else, do this: collapse your narrow, per-shape NodePools into one broad pool with an on-demand fallback. It shrinks both the frequency and the synchronization of interruptions — the two inputs to the storm.

The fix, part two: a pause-pod buffer so apps never wait for compute

Breadth reduces the storm. It does not make node provisioning instant — even a healthy CreateFleet plus boot plus kubelet-ready is tens of seconds to a couple of minutes, and during a crunch it's longer. If your app pods have to wait for that, an interruption is a latency spike or a capacity dip. The way to hide it is to keep warm headroom on the cluster at all times and let real workloads take it the instant they need it.

The mechanism is a Kubernetes-native trick: run low-value "pause" pods under a negative-priority PriorityClass. They request real CPU and memory, so Karpenter provisions nodes to hold them — that's your buffer. But because their priority is negative, any ordinary pod outranks them. When an interruption evicts a real pod and it needs a home, the scheduler preempts a pause pod and places the real pod on that already-warm node in seconds. The evicted pause pod goes Pending, Karpenter notices, and quietly rebuilds the headroom in the background — off the critical path.⁷

Here's the buffer. The PriorityClass value is negative so these pods are always the first thing preempted; the pause image is a near-zero-footprint container that does nothing but hold the reservation:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: overprovisioning
value: -10                     # negative: every real pod outranks it
globalDefault: false
description: "Warm-headroom placeholders; preempted by any real workload."
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: overprovisioning
  namespace: kube-system
spec:
  replicas: 6                  # size this to ~1 node's worth of headroom
  selector:
    matchLabels:
      app: overprovisioning
  template:
    metadata:
      labels:
        app: overprovisioning
    spec:
      priorityClassName: overprovisioning
      terminationGracePeriodSeconds: 0   # evict instantly, no drain wait
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.10
          resources:
            requests:            # this is the reservation Karpenter provisions for
              cpu: "1"
              memory: 2Gi

Wire the interruption queue (and don't forget backoff)

None of the graceful handling above happens unless Karpenter is actually watching for interruptions. That requires an SQS queue plus EventBridge rules that forward the relevant EC2 events to it, and Karpenter configured with the queue name. Miss this and your first sign of an interruption is a node vanishing mid-request — no cordon, no drain, no head start on a replacement.

# Helm values for the Karpenter controller
settings:
  clusterName: "prod-cluster"
  # The SQS queue EventBridge feeds interruption events into.
  interruptionQueue: "prod-cluster-karpenter-interruption"

# EventBridge event patterns Karpenter expects on the queue:
#   - "EC2 Spot Instance Interruption Warning"   (the 2-minute notice)
#   - "EC2 Instance Rebalance Recommendation"    (early heads-up)
#   - "EC2 Instance State-change Notification"   (stopping/terminating)
#   - "AWS Health Event"                         (scheduled maintenance)
source:      ["aws.ec2", "aws.health"]
detail-type:
  - "EC2 Spot Instance Interruption Warning"
  - "EC2 Instance Rebalance Recommendation"
  - "EC2 Instance State-change Notification"
  - "AWS Health Event"

resource "aws_sqs_queue" "interruption" {
  name                      = "prod-cluster-karpenter-interruption"
  message_retention_seconds = 300
  sqs_managed_sse_enabled   = true
}

resource "aws_cloudwatch_event_rule" "spot_interruption" {
  name          = "prod-cluster-spot-interruption"
  event_pattern = jsonencode({
    source      = ["aws.ec2"]
    detail-type = ["EC2 Spot Instance Interruption Warning"]
  })
}

resource "aws_cloudwatch_event_target" "to_sqs" {
  rule = aws_cloudwatch_event_rule.spot_interruption.name
  arn  = aws_sqs_queue.interruption.arn
}

One more knob worth knowing: the AWS SDK Karpenter uses already retries throttled calls with exponential backoff and jitter, which is exactly what AWS recommends for RequestLimitExceeded — progressively longer waits, a maximum retry count, and randomized delay to avoid synchronized retries.¹ That's your safety net, not your strategy. If you regularly see throttling even with a broad pool, the durable answer is a Service Quotas increase on the specific mutating actions (AWS advises raising the refill rate before the bucket maximum, and requesting at most 3× your current limit per request).¹

Pitfalls that quietly re-create the storm

• Confusing disruption budgets with API protection. spec.disruption.budgets caps voluntary disruption — consolidation, drift, expiration. It does not throttle interruption handling, so it won't slow a Spot storm. Use budgets to stop consolidation from becoming its own churn event; use pool breadth and backoff for the storm.
• Many narrow NodePools instead of one broad one. Splitting into a pool-per-team or pool-per-shape re-fragments the Spot pools and reconciles them independently — more concurrent CreateFleet calls, less PCO breadth. Prefer one broad pool with labels/taints for isolation, and add narrow pools only for genuine hardware needs (GPU, local NVMe).
• Turning on SpotToSpotConsolidation without measuring. It can meaningfully raise churn (and thus API traffic and pod disruption) chasing marginal savings. Enable it only after you've watched consolidation behavior on your workloads.³
• No on-demand fallback. A pure-Spot NodePool has nowhere to go when every allowed pool is drained at once. Including on-demand in capacity-type lets Karpenter keep the cluster up during the worst of a crunch and fall back to Spot as it recovers.
• Aggressive consolidateAfter. Very short consolidation windows (e.g. 0s) maximize savings but also maximize node churn — every churn is EC2 API calls. A modest delay (30s–2m) smooths the traffic with minimal cost impact.
• Forgetting terminationGracePeriodSeconds: 0 on pause pods. If a placeholder lingers on a graceful shutdown, preemption isn't instant and the buffer's whole point — immediate reschedule — is lost.