One load spike, one fixed cluster — watch a node get created, then deleted

One trigger, three provisioners — the divergence

The chain from a climbing metric to a Pending pod is identical in all three columns. Everything diverges at one point: who watches that unschedulable pod, and how it turns into an EC2 instance.

HPA algorithm & defaults

The pod-scaling loop. Metrics only ever move the replica count — never a node directly.

Scale-down safety

Pitfalls

• The VPC-CNI IP ceiling can strand pods on a Ready node. Pod IPs come from ENIs; the per-node cap ≈ (maxENIs × (IPsPerENI − 1)) + 2. Hit it and new pods stay Pending / ContainerCreating on a Ready node — Ready is not the finish line. Mitigate with prefix delegation (~16× density).
• A restrictive PDB blocks the drain. Both autoscalers respect PodDisruptionBudgets; too-strict a budget keeps an idle node — and its bill — alive.
• Stabilization thrash. The 0s-up / 300s-down asymmetry scales up fast and down cautiously on purpose; windows tuned too short make replicas flap.
• ASG-per-AZ vs flexible provisioning. A multi-AZ ASG can't control which AZ grows, forcing the one-ASG-per-AZ pattern under CAS; Karpenter and Auto Mode pick AZ and instance type per pending pod.
• Over-provisioning with pause-pods. Low-priority pause pods reserve headroom; a real pod preempts them and schedules instantly, and the now-Pending pause pods trigger node scale-up in the background.
• Auto Mode opacity. No Karpenter pod to see, no SSH/SSM to the node, no AMI customization; managed Bottlerocket instances cycle at a 21-day max lifetime and (from Apr 22 2026) are hidden by default from EC2 list operations.

Under the hood — the questions the loop doesn't answer

Deep dive — skip if you just wanted the loop. The interactive shows the trigger → create → delete cycle; this is the mechanics underneath it.

Where does the Cluster Autoscaler image / code actually live?

The image is registry.k8s.io/autoscaling/cluster-autoscaler:vX.Y.Z, and the tag is pinned to your cluster's Kubernetes minor version — a 1.30 cluster runs the 1.30 image. Kubernetes SIG Autoscaling builds and publishes it from github.com/kubernetes/autoscaler (the cluster-autoscaler/ directory); you pull a prebuilt static Go binary and never compile it. The algorithm is Go in that repo: scale-up simulates scheduling the pending pods (the U3 Unschedulable set) against each node group's template and an expander picks which group to grow; scale-down marks a node unneeded when utilization drops below --scale-down-utilization-threshold (0.5) for --scale-down-unneeded-time (10m) — --scale-down-unready-time is 20m for NotReady nodes — then simulates whether its pods fit elsewhere before removing it. The AWS-specific calls live in cluster-autoscaler/cloudprovider/aws, which controls the DesiredReplicas field of your EC2 Auto Scaling Groups. CAS runs as a single leader-elected replica — it is not itself horizontally scaled.

# tag is pinned to the cluster's K8s minor (1.30 cluster -> the 1.30 image)
registry.k8s.io/autoscaling/cluster-autoscaler:v1.30.0
# prebuilt static Go binary -> github.com/kubernetes/autoscaler (cluster-autoscaler/)
# AWS provider  -> cluster-autoscaler/cloudprovider/aws  -> ASG DesiredReplicas

How is it deployed on EKS — and how does it get permission?

Install via the Helm chart autoscaler/cluster-autoscaler (or the upstream manifest); the image tag must match your K8s minor. Permission flows through IRSA: a ServiceAccount is annotated with eks.amazonaws.com/role-arn, the pod receives a projected service-account token, and the AWS SDK exchanges it at STS via AssumeRoleWithWebIdentity for temporary credentials. The IAM role grants the write actions autoscaling:SetDesiredCapacity and autoscaling:TerminateInstanceInAutoScalingGroup (scoped by ASG-tag conditions) plus a set of read actions. Two flags wire it to AWS: --cloud-provider=aws selects the provider, and --node-group-auto-discovery finds the ASGs it may manage by tag. eksctl and the Terraform EKS module apply those tags for managed node groups.

# the actions the EKS docs grant (writes scoped by ASG-tag conditions)
"Action": [
  "autoscaling:SetDesiredCapacity",
  "autoscaling:TerminateInstanceInAutoScalingGroup",
  "autoscaling:DescribeAutoScalingGroups",
  "autoscaling:DescribeAutoScalingInstances",
  "autoscaling:DescribeLaunchConfigurations",
  "autoscaling:DescribeScalingActivities",
  "autoscaling:DescribeTags",
  "ec2:DescribeInstanceTypes",
  "ec2:DescribeLaunchTemplateVersions",
  "ec2:GetInstanceTypesFromInstanceRequirements",
  "eks:DescribeNodegroup"
]

# IRSA: projected SA token -> STS AssumeRoleWithWebIdentity -> temp creds
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cluster-autoscaler
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<acct>:role/<role>
# discovery flags on the CAS container
--cloud-provider=aws
--node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<cluster-name>

How does CAS scale a group that's at ZERO nodes?

With 0 live nodes there is nothing to inspect, so CAS reads template tags on the ASG to synthesize a fake node for its scheduling simulation — labels, resources, and taints each have their own tag prefix. Without these tags a scaled-to-zero group is invisible to the simulation and never grows.

# scale-from-zero: CAS builds a phantom node from ASG template tags
k8s.io/cluster-autoscaler/node-template/label/<key>         = <value>
k8s.io/cluster-autoscaler/node-template/resources/cpu        = 4
k8s.io/cluster-autoscaler/node-template/resources/memory     = 16Gi
k8s.io/cluster-autoscaler/node-template/taint/<key>         = <value>:NoSchedule

How does a new node get its labels / identity at boot?

The node's user-data bootstrap (bootstrap.sh on AL2, nodeadm on AL2023) starts the kubelet with the node's labels and taints; managed node groups propagate the labels you configured and add the eks.amazonaws.com/* labels. So the identity CAS assumed from the template tags in Q3 is the identity the real node actually boots with at U6.

# user-data (bootstrap.sh / nodeadm) -> kubelet starts with labels + taints
# managed node group also injects the eks.amazonaws.com/* labels
kubelet --node-labels=<your-labels>,eks.amazonaws.com/nodegroup=<ng> ...

Node group vs raw ASG — and what does "managed" actually mean?

A managed node group is an EKS abstraction that provisions your nodes as part of an EC2 Auto Scaling group managed for you by Amazon EKS. The crux: CAS only ever talks to the underlying ASG (SetDesiredCapacity / TerminateInstanceInAutoScalingGroup), never the EKS Nodegroup API; EKS re-syncs its own view via DescribeNodegroup. "Managed" means lifecycle ops: provisioning, AMI and security patching (rolled as new launch-template versions), graceful cordon + drain on updates and scale-down, and node auto-repair. It does not autoscale on its own — the desired count still comes from CAS (or you). A plain managed node group will never add or remove nodes for load by itself. A self-managed node group is a raw ASG you create and own. Terraform's terraform-aws-modules/eks creates managed node groups and can apply the CAS discovery tags. A mixed-instances policy lets one ASG blend instance types and On-Demand/Spot, but CAS needs the types to be "the same shape for CPU, Memory, and GPU" — it simulates using the first type.

Why does Karpenter need none of this?

Karpenter runs as a controller (registry public.ecr.aws/karpenter/, Helm charts karpenter + karpenter-crd) with no ASG and no node group. It watches unschedulable pods, computes a best-fit instance for exactly the pending pods, and calls ec2:CreateFleet directly (also RunInstances / RequestSpotInstances), removing nodes via ec2:TerminateInstances. Configuration is Kubernetes-native CRDs: a NodePool (allowed instance types, zones, capacity types, limits, disruption) and an EC2NodeClass (AMI, IAM role, subnets, security groups); each provisioning decision materializes a NodeClaim. Consolidation actively replaces and removes nodes to pack cheaper (consolidationPolicy: WhenEmpty / WhenEmptyOrUnderutilized, via Delete and Replace) — not just idle scale-down. For Spot it diversifies across many types and pools (Price-Capacity-Optimized) and drains gracefully on the Spot interruption / rebalance notice delivered through an SQS --interruption-queue.

apiVersion: karpenter.sh/v1
kind: NodePool
spec:
  template:
    spec:
      requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c","m","r"]
      nodeClassRef:
        kind: EC2NodeClass
        name: default
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
  amiFamily: AL2023
  role: KarpenterNodeRole
  subnetSelectorTerms:
    - tags: { karpenter.sh/discovery: <cluster-name> }
  securityGroupSelectorTerms:
    - tags: { karpenter.sh/discovery: <cluster-name> }

Who owns which decision — same trigger (an Unschedulable pod at U3), three different levers.