~/ nmadiraju.com Amazon EKS · Pod stuck in Pending
arrow_back All EKS troubleshooting playbooks
Troubleshooting Playbook · Amazon EKS scheduling

Pod Stuck in Pending on Amazon EKS

The event says 0/3 nodes are available: …, or the Pod just sits there. Find the cause from the message, confirm it, and fix it.

1Start here

What Pending means

Pending means Kubernetes accepted the Pod, but its containers aren't running yet. Most often the scheduler can't find a node that fits, and says why in a FailedScheduling event.

Less often the Pod is already on a node and is still pulling images or running init containers. The Events section tells you which, in one command.

Run this first

kubectl describe pod web-7d9f8c6b5-x2k4q -n shop
# Scroll to the bottom: Events
Events:
  Type     Reason            From               Message
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.
  • 0/3 nodes are availableNone of the cluster's 3 nodes passed every check.
  • 3 Insufficient cpuWhy, with a count of nodes per reason. One check can report two reasons, so counts can add up to more than the node total.
  • preemption: …Whether removing lower-priority Pods would make room. No preemption victims found means there were none to remove, typical when all Pods share the default priority; Preemption is not helpful means removing Pods wouldn't help. Either way, fix the reason itself.
  • Node: <none>Near the top of the output: the Pod was never placed. A node name there means it was scheduled: go to cause 8. (Events expire after an hour, so trust Node: over a missing Scheduled event.)

2Decision tree

Find the phrase from your event message on the right, then click it (or Tab to it and press Enter) to jump to that cause; the numbers match the cards below. If the message lists several reasons, start with the nodes you expected the Pod to land on.

Decision tree: from the event message to the cause of a Pending Pod Run kubectl describe pod and read Events. A FailedScheduling event with Node: none means the scheduler can't place the Pod; match a phrase in its message to one of seven causes. A node name under Node: (and a Scheduled event, until events expire after an hour) means the Pod is placed and its containers haven't started yet. Each cause links to its card below. RUN THIS WHICH EVENT DO YOU SEE? FIND YOUR MESSAGE, OPEN THE CAUSE Pod is Pending kubectl describe pod read the Events section 1 Insufficient cpu · Insufficient memory Requests too big, or every node is full card ↓ 2 node(s) had untolerated taint(s) A taint the Pod doesn't tolerate card ↓ 3 didn't match Pod's node affinity/selector No node has the labels the Pod asks for card ↓ 4 pod anti-affinity rules · topology spread constraints Anti-affinity or topology spread can't be met card ↓ 5 unbound immediate PersistentVolumeClaims · PersistentVolume's node affinity Volume not bound, or in another zone card ↓ 6 Too many pods Nodes are at their max Pods card ↓ 7 were unschedulable · no nodes available to schedule pods Nodes are cordoned, or there are none card ↓ FailedScheduling Node: <none> the scheduler can't place it 8 Init:0/1 · ContainerCreating · ErrImagePull Scheduled, but its containers haven't started card ↓ Scheduled Node: ip-10-0-2-41… placed, not started yet
Where you lookThe scheduler can't place the PodPlaced on a node, still starting

3Causes and fixes

One card per cause. Each shows what the message looks like, the commands that confirm it, the fix, how to keep it from coming back, and the walkthrough that explains the mechanism.

Requests too big, or every node is full

Insufficient cpu Insufficient memory

Back to the tree

What you see

Events:
  Type     Reason            From               Message
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.
# Karpenter tried, but its NodePool is at its limit
  Warning  FailedScheduling  karpenter  Failed to schedule pod, all available instance types exceed limits for nodepool (NodePool=default)
# Cluster Autoscaler tried, but the node group is at its max size
  Normal   NotTriggerScaleUp  cluster-autoscaler  pod didn't trigger scale-up: 1 max node group size reached

The scheduler only compares requests with each node's allocatable capacity, not real usage. A node at 20% CPU can still be full on requests.

Confirm

# What does the Pod ask for?
kubectl get pod web-7d9f8c6b5-x2k4q -n shop -o jsonpath='{.spec.containers[*].resources.requests}'
# What is already requested on each node? (cpu and memory: requests, limits)
kubectl describe nodes | grep -E "^Name:|^  (cpu|memory) +[0-9]"
# Did a node autoscaler react, and if not, why?
kubectl get events -n shop --field-selector involvedObject.name=web-7d9f8c6b5-x2k4q

With Karpenter, a Nominated event (Pod should schedule on: nodeclaim/…) means a node is on its way. With Cluster Autoscaler, look for TriggeredScaleUp.

Fix

  • One Pod asks for more than any node can give (say cpu: 6 on 4 vCPU nodes): a new node of the same size won't help. Lower the request, or allow a bigger instance type in the NodePool or node group.
  • Nodes are just full: let the node autoscaler add one. If Karpenter says the NodePool exceed limits, raise spec.limits on the NodePool. If Cluster Autoscaler says max node group size reached, raise the node group's maxSize.
  • No node autoscaler: raise the managed node group's desired size, for example aws eks update-nodegroup-config --cluster-name prod --nodegroup-name general --scaling-config minSize=3,maxSize=8,desiredSize=5.

Prevent

  • Set requests from measured usage (Container Insights, Prometheus, or VPA recommendations), not guesses.
  • Leave headroom in NodePool limits and node group max sizes, and alert when a Pod has been Pending for more than a few minutes.
  • In shared namespaces, use a LimitRange to set default requests and a maximum per container.

Understand it

Requests, limits, QoS and evictionsCluster Autoscaler and Karpenter

A taint the Pod doesn't tolerate

node(s) had untolerated taint(s)

Back to the tree

What you see

Events:
  Type     Reason            From               Message
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 3 node(s) had untolerated taint(s). preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

Kubernetes 1.33 and older name the taint in the message, for example node(s) had untolerated taint {dedicated: gpu}. From 1.34 the message no longer says which one, so check the nodes.

Confirm

# Which taints do the nodes carry?
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
# Which tolerations does the Pod have?
kubectl get pod web-7d9f8c6b5-x2k4q -n shop -o jsonpath='{.spec.tolerations}'

On EKS, taints usually come from a managed node group's taints setting or a Karpenter NodePool's spec.template.spec.taints. Kubernetes adds others itself: node.kubernetes.io/not-ready, unreachable, disk-pressure and memory-pressure mean the node is unhealthy. A eks.amazonaws.com/compute-type=fargate taint means the Pod matched no Fargate profile (see card 7).

Fix

  • The Pod belongs on those nodes (say GPU work on a nvidia.com/gpu pool): add a toleration with the same key, value and effect.
  • It doesn't belong there: don't add a toleration. Give it somewhere untainted to run, such as a general-purpose NodePool or node group with room.
  • The taint is not-ready, unreachable or a *-pressure taint: fix the node, don't tolerate it.

Prevent

  • A toleration only allows a Pod onto a tainted node; it doesn't send it there. Pair each dedicated taint with a label, and give the workloads that belong there both the toleration and a nodeSelector.
  • Keep at least one untainted NodePool or node group for everything else.

Understand it

Taints, tolerations and affinity

No node has the labels the Pod asks for

node(s) didn't match Pod's node affinity/selector

Back to the tree

What you see

Events:
  Type     Reason            From               Message
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

Confirm

# What does the Pod require?
kubectl get pod web-7d9f8c6b5-x2k4q -n shop -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.affinity.nodeAffinity}'
# What labels do the nodes have?
kubectl get nodes -L topology.kubernetes.io/zone,node.kubernetes.io/instance-type,eks.amazonaws.com/nodegroup,karpenter.sh/nodepool

Fix

  • A typo or a stale value, such as eks.amazonaws.com/nodegroup: workers after the group was replaced by workers-v2: fix the selector, or put the label on the nodes (a node group's labels, or a NodePool's spec.template.metadata.labels).
  • A node autoscaler should create a matching node: Karpenter can only launch nodes its NodePool requirements allow, so a selector for an instance type or zone the NodePool excludes never matches. Add it to the NodePool's requirements.
  • Cluster Autoscaler scaling up from zero has no live node to read labels from: it reads a managed node group's labels through the EKS DescribeNodegroup API (its IAM policy needs eks:DescribeNodegroup), or k8s.io/cluster-autoscaler/node-template/label/… tags on the Auto Scaling group.

Prevent

  • Select on well-known labels (topology.kubernetes.io/zone, kubernetes.io/arch) or a label you set yourself, such as workload-tier: general, rather than node group or NodePool names that change when you replace them.
  • Use preferredDuringSchedulingIgnoredDuringExecution when a placement is a preference, not a rule.

Understand it

Taints, tolerations and affinity

Anti-affinity or topology spread can't be met

didn't match pod anti-affinity rules didn't match pod topology spread constraints

Back to the tree

What you see

Events:
  Type     Reason            From               Message
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 3 node(s) didn't match pod anti-affinity rules. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.
  Warning  FailedScheduling  default-scheduler  0/4 nodes are available: 1 Insufficient cpu, 3 node(s) didn't match pod topology spread constraints. preemption: 0/4 nodes are available: 4 No preemption victims found for incoming pod.

The second message is the classic zone spread trap: the Pod must go to the one zone that is behind, and that zone's only node is full.

Confirm

# Which rules does the Pod carry?
kubectl get pod web-7d9f8c6b5-x2k4q -n shop -o yaml | grep -A 12 -E 'podAntiAffinity|topologySpreadConstraints'
# Where are the other replicas, and which zone is each node in?
kubectl get pods -n shop -l app=web -o wide
kubectl get nodes -L topology.kubernetes.io/zone

Fix

  • Required anti-affinity on kubernetes.io/hostname allows one replica per node, so 5 replicas need 5 nodes. Add nodes, lower the replica count, or switch to preferredDuringScheduling… anti-affinity.
  • Zone spread with whenUnsatisfiable: DoNotSchedule waits for room in the zone that's behind. Make sure every zone can get capacity: subnets in every zone in the EC2NodeClass (Karpenter) or the node groups, and room under their limits.
  • If spreading is only nice to have, use whenUnsatisfiable: ScheduleAnyway.

Prevent

  • With Cluster Autoscaler, run one node group per Availability Zone for workloads that spread by zone; a group that spans zones can't be told which zone to grow.
  • With Karpenter, let the NodePool use every zone the workload spreads over.

Understand it

Taints, tolerations and affinity

Volume not bound, or in another zone

pod has unbound immediate PersistentVolumeClaims didn't match PersistentVolume's node affinity

Back to the tree

What you see

Events:
  Type     Reason            From               Message
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 1 Insufficient memory, 2 node(s) didn't match PersistentVolume's node affinity. preemption: 0/3 nodes are available: 1 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling.

An EBS volume lives in one Availability Zone and can only attach to a node in that zone. In the second message the volume is in us-east-1a, and the only node there is full.

Confirm

kubectl get pvc -n shop
# Events show provisioning errors, or that it waits for a Pod
kubectl describe pvc data-db-0 -n shop
# Which zone is the volume in, and which zones have nodes?
kubectl get pv <pv-name> -o jsonpath='{.spec.nodeAffinity}'
kubectl get nodes -L topology.kubernetes.io/zone

Fix

  • Unbound claim: this message means the claim must be bound before the Pod can be scheduled: its StorageClass uses volumeBindingMode: Immediate, doesn't exist, or isn't set and there's no default. Check storageClassName and the claim's events; with Immediate, also check that the EBS CSI driver add-on is installed and its IAM role can create volumes.
  • No default StorageClass: on clusters created at EKS 1.30 or later, no StorageClass is the default (gp2 exists but isn't annotated), so a claim without storageClassName waits. Set a default, for example the EBS CSI add-on's defaultStorageClass.enabled=true, or annotate a gp3 class.
  • Volume in another zone: give the Pod a node in the volume's zone (capacity in that zone in the NodePool or node group). An EBS volume can't move zones; to move the data, snapshot it and restore in the new zone.

Prevent

  • Use StorageClasses with volumeBindingMode: WaitForFirstConsumer, so the volume is created in the zone where the Pod is scheduled. With it, a claim stays Pending until a Pod uses it, which is normal, and a failing EBS CSI driver shows up differently: the Pod's event reads running PreBind plugin "VolumeBinding" and the claim says it's waiting for a volume to be created.
  • For stateful workloads, keep capacity in every zone that holds their volumes, with one node group per zone if you use Cluster Autoscaler.

Understand it

Persistent volumes and the EBS CSI driver

Nodes are at their max Pods

Too many pods

Back to the tree

What you see

Events:
  Type     Reason            From               Message
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 3 Too many pods. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

On EKS the max Pods per node usually comes from how many IP addresses the instance type's network interfaces can hold: 17 on a t3.medium, 29 on an m5.large. Small nodes run out of Pod slots long before CPU.

Confirm

# Max Pods per node, and the instance type
kubectl get nodes -o custom-columns='NAME:.metadata.name,TYPE:.metadata.labels.node\.kubernetes\.io/instance-type,MAXPODS:.status.allocatable.pods'
# How many Pods already run on one node (finished Pods don't count)
kubectl get pods -A --field-selector spec.nodeName=ip-10-0-1-23.ec2.internal,status.phase!=Succeeded,status.phase!=Failed --no-headers | wc -l

Fix

  • Quickest: add nodes, or move to larger instance types with more Pod slots. Node autoscalers count Pod slots too.
  • More Pods per node: turn on prefix delegation in the VPC CNI (ENABLE_PREFIX_DELEGATION=true), which gives each IP slot a /28 prefix of 16 addresses, then raise max Pods on new nodes. Managed node groups without a custom AMI calculate it for you; self-managed groups, custom AMIs and Karpenter (spec.kubelet.maxPods on the EC2NodeClass) need it set. Existing nodes keep their old limit until they're replaced.

Prevent

  • Choose instance types with the Pod count in mind, not only CPU and memory.
  • Turn on prefix delegation when you build a cluster, and keep an eye on free IPs in your subnets.

Understand it

The VPC CNIPrefix delegation and custom networking

Related, but different: a Pod that is scheduled and stuck in ContainerCreating with FailedCreatePodSandBox … failed to assign an IP address to container has a node and a Pod slot, but the VPC CNI couldn't get it an IP, often because the subnet is out of addresses. That's a networking problem, not a scheduling one; see the VPC CNI walkthrough.

Nodes are cordoned, or there are none

node(s) were unschedulable no nodes available to schedule pods

Back to the tree

What you see

Events:
  Type     Reason            From               Message
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 3 node(s) were unschedulable. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  default-scheduler  no nodes available to schedule pods

Confirm

# Cordoned nodes show Ready,SchedulingDisabled
kubectl get nodes
# Managed node group size and health
aws eks describe-nodegroup --cluster-name prod --nodegroup-name general --query 'nodegroup.{status:status,scaling:scalingConfig,health:health}'
# Karpenter: are NodeClaims being created and launched?
kubectl get nodepools,nodeclaims

Fix

  • Cordoned after a drain or an upgrade that stopped halfway: finish the rollout, or kubectl uncordon <node> a healthy node.
  • No nodes at all: the node group may be scaled to 0, or new nodes fail to join (see the node group's health issues, or Karpenter's NodeClaims and logs).
  • Fargate-only cluster: a Pod that matches no Fargate profile goes to the default scheduler. The only nodes are running Fargate Pods, tainted eks.amazonaws.com/compute-type=fargate, so the event usually reads untolerated taint(s); you see no nodes available only when nothing runs on Fargate yet. Add a profile that matches the Pod's namespace and labels.

Prevent

  • Keep a small always-on node group (min size of 1 or more) for cluster add-ons like CoreDNS.
  • Alarm on node group health issues and on Pods Pending for more than a few minutes, especially during upgrades.

Understand it

Managed node groupsFargate and EKS Auto Mode

Scheduled, but its containers haven't started

Scheduled Init:0/1 ContainerCreating ErrImagePull

Back to the tree

What you see

$ kubectl get pod web-7d9f8c6b5-x2k4q -n shop
NAME                  READY   STATUS     RESTARTS   AGE
web-7d9f8c6b5-x2k4q   0/1     Init:0/1   0          6m

Events:
  Type    Reason     From               Message
  Normal  Scheduled  default-scheduler  Successfully assigned shop/web-7d9f8c6b5-x2k4q to ip-10-0-2-41.ec2.internal
  Normal  Pulled     kubelet            Container image "busybox:1.36" already present on machine and can be accessed by the pod
  Normal  Created    kubelet            Created container: wait-for-db
  Normal  Started    kubelet            Started container wait-for-db

The Pod's phase is still Pending: Kubernetes counts a Pod as Pending until its containers have been set up, which includes waiting for a node, pulling images, and running init containers. kubectl get shows a more specific STATUS such as Init:0/1, ContainerCreating or ImagePullBackOff.

Confirm

# Phase and node: Pending plus a node name means it was scheduled
kubectl get pod web-7d9f8c6b5-x2k4q -n shop -o jsonpath='{.status.phase} {.spec.nodeName}{"\n"}'
# Each container's State: Waiting and its Reason, plus the latest events
kubectl describe pod web-7d9f8c6b5-x2k4q -n shop
# An init container that never finishes: read its logs
kubectl logs web-7d9f8c6b5-x2k4q -n shop -c wait-for-db

Fix

  • Init container still running: it's waiting for something (here a database). Fix that dependency or the init script; the app containers start only after every init container exits successfully.
  • Image pull: ErrImagePull or ImagePullBackOff means a wrong name or tag, missing ECR pull permissions on the node role, or no network path to ECR from private subnets (NAT, or VPC endpoints for ECR and S3). A large image may just be slow.
  • ContainerCreating for minutes: read the events. FailedMount or FailedAttachVolume is a volume problem; FailedCreatePodSandBox is usually Pod networking (see card 6).

Prevent

  • Give init containers a timeout and a clear log line, so a stuck dependency is obvious.
  • Keep images small, and give private subnets VPC endpoints for ECR so pulls don't depend on a NAT gateway.

Understand it

PodsLiveness, readiness and startup probes

4Diagnose it

Six Pods stuck in Pending. Read the output and pick the cause. A wrong pick tells you why it doesn't fit; there's no score.

Diagnose it