~/ nmadiraju.com Amazon EKS · ImagePullBackOff
arrow_back All EKS troubleshooting playbooks
Troubleshooting Playbook · Amazon EKS images

ImagePullBackOff on Amazon EKS

The Pod flips between ErrImagePull and ImagePullBackOff, and the event says Failed to pull image "…". Find the cause from the registry's answer, confirm it, and fix it.

1Start here

What ImagePullBackOff means

The Pod has a node, and that node's kubelet couldn't pull a container image. The failed attempt shows as ErrImagePull; the kubelet then waits before trying again, and while it waits the status is ImagePullBackOff. The wait doubles each time, up to five minutes.

Back-off is never the cause. The cause is in the Failed to pull image event just before it, which quotes what the registry answered: not found, refused, throttled, or no answer at all.

Run this first

kubectl describe pod web-7d9f8c6b5-x2k4q -n shop
# Scroll to the bottom: Events
Events:
  Type     Reason   From     Message
  Normal   Pulling  kubelet  Pulling image "111122223333.dkr.ecr.us-east-1.amazonaws.com/shop/web:v2.4.1"
  Warning  Failed   kubelet  Failed to pull image "111122223333.dkr.ecr.us-east-1.amazonaws.com/shop/web:v2.4.1": ... 111122223333.dkr.ecr.us-east-1.amazonaws.com/shop/web:v2.4.1: not found
  Warning  Failed   kubelet  Error: ErrImagePull
  Normal   BackOff  kubelet  Back-off pulling image "111122223333.dkr.ecr.us-east-1.amazonaws.com/shop/web:v2.4.1"
  Warning  Failed   kubelet  Error: ImagePullBackOff
  • Failed to pull image "…"The exact image reference the node tried. Check it first: registry host, repository, tag.
  • : not foundThe end of the message is the registry's answer, and it picks your branch of the tree: not found, 401/403, 429 or i/o timeout.
  • Error: ErrImagePullOne pull attempt failed.
  • Back-off pulling imageThe kubelet is waiting before the next try (10 seconds at first, up to 5 minutes). It keeps retrying, so once you fix the cause the Pod starts on its own.

2Decision tree

Find the phrase from the Failed to pull image event on the right, then click it (or Tab to it and press Enter) to jump to that cause; the numbers match the cards below.

Decision tree: from the Failed event to the cause of ImagePullBackOff Run kubectl describe pod and read the kubelet's Failed event, which quotes the registry's answer. A not found answer points to a wrong name or tag, a missing architecture, or a pull-through cache first pull. 401 or 403 means credentials: the node role for ECR, or a pull secret for other registries. 429 means the Docker Hub rate limit. An i/o timeout or a pull that never finishes means no network path, or a very large image. Each cause links to its card below. RUN THIS WHAT DID THE REGISTRY SAY? FIND YOUR MESSAGE, OPEN THE CAUSE ImagePullBackOff kubectl describe pod read the Failed event 1 <image>:<tag>: not found · manifest unknown The image name or tag doesn't exist card ↓ 2 no match for platform in manifest: not found No image for the node's CPU architecture card ↓ 3 not found · 403, on a …/docker-hub/… cache path Pull-through cache: the first pull fails card ↓ not found manifest unknown no such image for this node 4 *.dkr.ecr.*: no basic auth credentials · 403 The node can't authenticate to ECR, or is denied card ↓ 5 insufficient_scope · 403 (Docker Hub, ghcr.io) A private registry with no pull secret card ↓ 401 · 403 pull access denied the registry refused you 6 429 Too Many Requests · toomanyrequests Docker Hub rate limit card ↓ 429 toomanyrequests the registry is throttling 7 dial tcp …:443: i/o timeout The node has no network path to the registry card ↓ 8 Pulling image … (no Pulled yet) · context canceled The image is large, so the pull is slow or gets cancelled card ↓ i/o timeout or still Pulling no answer, or a slow one
Where you lookThe registry answered with an errorNo answer from the registry, or a slow pull

3Causes and fixes

One card per cause. Each shows what the message looks like, the commands that confirm it, the fix, how to keep it from coming back, and the walkthrough that explains the mechanism.

The image name or tag doesn't exist

not found manifest unknown InvalidImageName

Back to the tree

What you see

Events:
  Type     Reason   From     Message
  Warning  Failed   kubelet  Failed to pull image "111122223333.dkr.ecr.us-east-1.amazonaws.com/shop/web:v2.4.l": rpc error: code = NotFound desc = failed to pull and unpack image "111122223333.dkr.ecr.us-east-1.amazonaws.com/shop/web:v2.4.l": failed to resolve reference "111122223333.dkr.ecr.us-east-1.amazonaws.com/shop/web:v2.4.l": 111122223333.dkr.ecr.us-east-1.amazonaws.com/shop/web:v2.4.l: not found
# A typo in a Docker Hub repository name looks like a login problem
  Warning  Failed   kubelet  Failed to pull image "ngnix:1.27": ... failed to resolve reference "docker.io/library/ngnix:1.27": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Read the image reference in the message character by character: here the tag ends in a lowercase l, not a 1. Docker and other clients word the same thing as manifest unknown. A reference that can't even be parsed (say, upper-case letters in the repository) never reaches the registry; the status is InvalidImageName instead.

Confirm

# Which image does the Pod ask for, exactly?
kubectl get pod web-7d9f8c6b5-x2k4q -n shop -o jsonpath='{.spec.containers[*].image}'
# Does that tag exist in ECR?
aws ecr describe-images --repository-name shop/web --image-ids imageTag=v2.4.l
# An error occurred (ImageNotFoundException) ... does not exist within the repository with name 'shop/web' ...
# Which tags do exist (newest first)?
aws ecr describe-images --repository-name shop/web --query 'reverse(sort_by(imageDetails,&imagePushedAt))[:5].imageTags'

If the tag was there last week, check the repository's lifecycle policy: a rule with "countType": "imageCountMoreThan", "countNumber": 30 deletes old images, and a node replacement or a rollback that needs an old tag then fails.

Fix

  • Typo or wrong tag: fix the image in the Deployment (or its Helm values) and let it roll out. Kubelet retries on its own, but a fixed spec needs new Pods.
  • The tag was never pushed: the CI job failed or pushed to another repository or Region. Push it, then the next retry pulls it.
  • A lifecycle policy expired it: push the image again from your build, and exclude tags that are still deployed from the rule (for example, match only tagPrefixList: ["pr-"]).

Prevent

  • Have CI deploy the exact tag or digest it just pushed (shop/web@sha256:…), never a hand-typed one.
  • Turn on tag immutability in ECR so a tag always means the same image, and write lifecycle rules that only expire build or pull-request tags.

Understand it

PodsRuntime and image security

No image for the node's CPU architecture

no match for platform in manifest

Back to the tree

What you see

$ kubectl get pod web-7d9f8c6b5-x2k4q -n shop -o wide
NAME                  READY   STATUS             NODE
web-7d9f8c6b5-x2k4q   0/1     ErrImagePull       ip-10-0-3-88.ec2.internal

Events:
  Warning  Failed   kubelet  Failed to pull image "acme/report:3.2": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/acme/report:3.2": no match for platform in manifest: not found

The image is a multi-architecture index (manifest list) with no entry for this node's architecture, typically linux/arm64 on a Graviton node. An image built for a single architecture behaves differently: it pulls fine on the wrong node and then fails to start with exec format error and CrashLoopBackOff.

Confirm

# Which architecture is the node?
kubectl get node ip-10-0-3-88.ec2.internal -L kubernetes.io/arch,node.kubernetes.io/instance-type
# Which platforms does the image offer?
docker buildx imagetools inspect acme/report:3.2
#   Platform:  linux/amd64      <- only amd64, no linux/arm64

No Docker on hand? crane manifest acme/report:3.2 or skopeo inspect --raw docker://acme/report:3.2 print the same list.

Fix

  • Build for both: docker buildx build --platform linux/amd64,linux/arm64 -t 111122223333.dkr.ecr.us-east-1.amazonaws.com/acme/report:3.2 --push . publishes one tag that works on both kinds of node.
  • Can't rebuild it (a vendor image): keep the Pod on x86 nodes with nodeSelector: {kubernetes.io/arch: amd64}.
  • Karpenter: a NodePool that allows kubernetes.io/arch In [amd64, arm64] may pick Graviton for a Pod that has no arch constraint, because it's cheaper. Pin such workloads to amd64, or make every image multi-arch before you allow arm64.

Prevent

  • Make multi-architecture builds the CI default, and test on both architectures before allowing Graviton in a NodePool or node group.
  • Check third-party images with docker buildx imagetools inspect before you adopt them.

Understand it

Spot and Graviton

Pull-through cache: the first pull fails

not found 403 Forbidden

Back to the tree

What you see

  Warning  Failed   kubelet  Failed to pull image "111122223333.dkr.ecr.us-east-1.amazonaws.com/docker-hub/library/redis:7.4": ... failed to resolve reference "111122223333.dkr.ecr.us-east-1.amazonaws.com/docker-hub/library/redis:7.4": 111122223333.dkr.ecr.us-east-1.amazonaws.com/docker-hub/library/redis:7.4: not found
# A Docker Hub official image without /library/ in the path
  Warning  Failed   kubelet  Failed to pull image "111122223333.dkr.ecr.us-east-1.amazonaws.com/docker-hub/redis:7.4": ... failed to resolve reference "111122223333.dkr.ecr.us-east-1.amazonaws.com/docker-hub/redis:7.4": unexpected status from HEAD request to https://111122223333.dkr.ecr.us-east-1.amazonaws.com/v2/docker-hub/redis/manifests/7.4: 403 Forbidden

An ECR pull-through cache rule copies an upstream image (Docker Hub, Quay, GHCR, ECR Public and others) into your registry on first pull. Images already in the cache pull fine; only the first pull of a new image or repository fails.

Confirm

# Is there a rule for this prefix, and which upstream does it point at?
aws ecr describe-pull-through-cache-rules --query 'pullThroughCacheRules[].[ecrRepositoryPrefix,upstreamRegistryUrl,credentialArn]'
# Has the cache repository been created yet?
aws ecr describe-repositories --repository-names docker-hub/library/redis
# What can the node role do? (policies attached to it)
aws iam list-attached-role-policies --role-name eks-node-role

Fix

  • The pulling role lacks the import permissions: the node role needs ecr:BatchImportUpstreamImage (in AmazonEC2ContainerRegistryPullOnly, not in ReadOnly) and, if the cache repository doesn't exist yet, ecr:CreateRepository. Add them in an IAM policy or the registry permissions policy, or create the repository ahead of time.
  • Docker Hub official images need /library/ in the path: …/docker-hub/library/redis:7.4, not …/docker-hub/redis:7.4.
  • No internet route for the first pull: AWS documents that when you use ECR through interface VPC endpoints, the first pull through a cache rule may need a route to the internet (for example a NAT gateway); later pulls don't. Pull each new image once from a subnet with that route, or keep a NAT route while you warm the cache.
  • Wrong upstream credentials: Docker Hub, GHCR and other authenticated upstreams need a Secrets Manager secret whose name starts with ecr-pullthroughcache/, in the same account and Region as the rule.

Prevent

  • Warm the cache for new images in CI (one pull through the rule) before you deploy them to private clusters.
  • Use a repository creation template so cache repositories get your lifecycle and scanning settings.

Understand it

Private clusters and endpoint access

The node can't authenticate to ECR, or is denied

401 Unauthorized no basic auth credentials 403 Forbidden

Back to the tree

What you see

# The node role has no ECR permissions: kubelet gets no ECR token
  Warning  Failed   kubelet  Failed to pull image "111122223333.dkr.ecr.us-east-1.amazonaws.com/shop/web:v2.4.1": ... failed to resolve reference "111122223333.dkr.ecr.us-east-1.amazonaws.com/shop/web:v2.4.1": pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials
# The node has a token, but the repository's account doesn't allow it (cross-account)
  Warning  Failed   kubelet  Failed to pull image "444455556666.dkr.ecr.us-east-1.amazonaws.com/platform/base:1.8": ... unexpected status from HEAD request to https://444455556666.dkr.ecr.us-east-1.amazonaws.com/v2/platform/base/manifests/1.8: 403 Forbidden

Image pulls use the node's IAM role, through the kubelet's ECR credential provider. A Pod's own role from EKS Pod Identity or IRSA plays no part in pulling its image (unless, on EKS 1.35 or later, you opt in to per-Pod pull roles with the eks.amazonaws.com/ecr-role-arn ServiceAccount annotation). On Fargate, the Pod execution role is used instead.

Confirm

# Which role do the nodes use?
aws eks describe-nodegroup --cluster-name prod --nodegroup-name general --query nodegroup.nodeRole
# Does it have an ECR read policy?
aws iam list-attached-role-policies --role-name eks-node-role --query 'AttachedPolicies[].PolicyName'
# Cross-account: run with credentials in 444455556666, the repository owner
aws ecr get-repository-policy --registry-id 444455556666 --repository-name platform/base

On the node (SSM Session Manager), journalctl -u kubelet | grep -i credential shows whether the credential provider failed to get a token, for example an AccessDeniedException on ecr:GetAuthorizationToken.

Fix

  • Node role without ECR access: attach AmazonEC2ContainerRegistryPullOnly (or the older AmazonEC2ContainerRegistryReadOnly) to the node role. No node replacement is needed: the kubelet asks for a token again on its next retry.
  • Cross-account pull: in the account that owns the repository, add a repository policy statement that allows your node role ARN ecr:BatchGetImage and ecr:GetDownloadUrlForLayer. The node role's own policy must allow those actions too (the managed pull policies use Resource: *).
  • Repository encrypted with a customer managed KMS key: ECR decrypts layers on pull through grants it created on the key with the repository. Don't revoke those grants, or every pull from that repository fails.

Prevent

  • Create node roles from code with the pull policy attached (eksctl does this by default; the EKS docs' IAM console and CLI steps attach AmazonEC2ContainerRegistryPullOnly), and keep cross-account repository policies in the same code as the repository.
  • For many accounts, grant pulls with an organization condition (aws:PrincipalOrgID) in the repository policy instead of listing role ARNs one by one.

Understand it

Managed node groups

A private registry with no pull secret

insufficient_scope 403 Forbidden Unable to retrieve some image pull secrets

Back to the tree

What you see

# Docker Hub private repository, no credentials
  Warning  Failed   kubelet  Failed to pull image "acme/billing:2.0": ... pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
# GitHub Container Registry private package, no credentials
  Warning  Failed   kubelet  Failed to pull image "ghcr.io/acme/billing:2.0": ... failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Aacme%2Fbilling%3Apull&service=ghcr.io: 403 Forbidden
# The Pod names a secret that isn't in its namespace
  Warning  FailedToRetrieveImagePullSecret  kubelet  Unable to retrieve some image pull secrets (regcred); attempting to pull the image may not succeed.

The node's IAM role only helps with ECR. Every other private registry needs credentials in a kubernetes.io/dockerconfigjson Secret, referenced from the Pod's imagePullSecrets or its ServiceAccount, in the Pod's own namespace. The same insufficient_scope message appears for a misspelled public Docker Hub repository, so check the name before adding credentials.

Confirm

# Which pull secrets does the Pod carry (directly or from its ServiceAccount)?
kubectl get pod web-7d9f8c6b5-x2k4q -n shop -o jsonpath='{.spec.imagePullSecrets}'
# Is the secret there, in this namespace, with the right type?
kubectl get secret regcred -n shop -o jsonpath='{.type}'
# Which registry does it hold credentials for?
kubectl get secret regcred -n shop -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq '.auths | keys'

The registry host in the secret must match the image: credentials for https://index.docker.io/v1/ don't help with ghcr.io. An expired token (a GitHub personal access token, say) fails the same way as no token.

Fix

  • Create the secret in the Pod's namespace: kubectl create secret docker-registry regcred -n shop --docker-server=ghcr.io --docker-username=acme-bot --docker-password="$GHCR_TOKEN".
  • Reference it: add imagePullSecrets: [{name: regcred}] to the Pod template, or patch it onto the namespace's ServiceAccount so every Pod using it gets it.
  • Rather not hand out registry tokens? Cache the registry in ECR with a pull-through cache rule (Docker Hub, GHCR, GitLab, Quay and others), so nodes pull with their IAM role.

Prevent

  • Sync registry credentials from AWS Secrets Manager with External Secrets, so a rotated token reaches every namespace.
  • Use a robot or machine account with read-only scope and an expiry you track, not a person's token.

Understand it

PodsSecrets from AWS

Docker Hub rate limit

429 Too Many Requests toomanyrequests

Back to the tree

What you see

  Warning  Failed   kubelet  Failed to pull image "redis:7.4": ... failed to pull and unpack image "docker.io/library/redis:7.4": failed to copy: httpReadSeeker: failed open: unexpected status from GET request to https://registry-1.docker.io/v2/library/redis/manifests/sha256:...: 429 Too Many Requests
toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit

Docker Hub limits unauthenticated pulls per source IP address. Nodes in private subnets share their NAT gateway's public IP, so the whole cluster (or the whole VPC) draws on one allowance; it often shows up during a scale-out or a node rollover, when many nodes pull at once.

Confirm

# Ask Docker Hub for the remaining allowance, from a node or a Pod (it is per source IP)
TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
curl -s --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest | grep -i ratelimit
# ratelimit-remaining: 0;w=... means this IP has used its allowance for the window

Docker sets the limits and changes them from time to time; the response headers show the limit and window that apply to you right now.

Fix

  • Cache Docker Hub in ECR: create a pull-through cache rule for Docker Hub (with your Docker Hub login in a Secrets Manager secret named ecr-pullthroughcache/…) and pull 111122223333.dkr.ecr.us-east-1.amazonaws.com/docker-hub/library/redis:7.4. ECR pulls upstream once; nodes pull from ECR with their IAM role.
  • Official images: many are mirrored on Amazon ECR Public, for example public.ecr.aws/docker/library/redis:7.4.
  • Quick relief: add an imagePullSecrets entry with a Docker Hub account; authenticated pulls get a higher limit.

Prevent

  • Don't pull from Docker Hub directly in production: mirror or cache every third-party image in ECR, and point Helm chart values at the ECR copy.
  • Use imagePullPolicy: IfNotPresent with pinned tags, so nodes that already have the image don't pull it again.

Understand it

Private clusters and endpoint access

The node has no network path to the registry

dial tcp …:443: i/o timeout

Back to the tree

What you see

# No route to ECR at all (private subnet, no NAT, no ECR endpoints)
  Warning  Failed   kubelet  Failed to pull image "111122223333.dkr.ecr.us-east-1.amazonaws.com/shop/web:v2.4.1": ... failed to resolve reference "111122223333.dkr.ecr.us-east-1.amazonaws.com/shop/web:v2.4.1": failed to do request: Head "https://111122223333.dkr.ecr.us-east-1.amazonaws.com/v2/shop/web/manifests/v2.4.1": dial tcp 3.215.103.21:443: i/o timeout
# ECR endpoints exist, but no S3 gateway endpoint: the manifest works, the layers don't
  Warning  Failed   kubelet  Failed to pull image "111122223333.dkr.ecr.us-east-1.amazonaws.com/shop/web:v2.4.1": ... failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://prod-us-east-1-starport-layer-bucket.s3.us-east-1.amazonaws.com/...": dial tcp 52.217.12.4:443: i/o timeout

ECR needs three things from a node without internet access: the ecr.api endpoint (for the auth token), the ecr.dkr endpoint (for manifests) and S3 (for the layers, which ECR stores in S3). A public IP for the ECR host means private DNS isn't resolving it to the interface endpoint. S3 always resolves to public IPs, even with a gateway endpoint, because that endpoint works through the route table: check the routes.

Confirm

# Which endpoints does the VPC have?
aws ec2 describe-vpc-endpoints --filters Name=vpc-id,Values=vpc-0a1b2c3d --query 'VpcEndpoints[].[ServiceName,VpcEndpointType,PrivateDnsEnabled]' --output table
# Does the node's subnet route to a NAT gateway or the S3 gateway endpoint?
aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=subnet-0f1e2d3c --query 'RouteTables[].Routes[].[DestinationCidrBlock,DestinationPrefixListId,NatGatewayId,GatewayId]'
# From the node (SSM Session Manager): does ECR resolve to private IPs?
nslookup api.ecr.us-east-1.amazonaws.com
nslookup 111122223333.dkr.ecr.us-east-1.amazonaws.com

Fix

  • Add the endpoints: interface endpoints com.amazonaws.us-east-1.ecr.api and com.amazonaws.us-east-1.ecr.dkr with private DNS turned on, and a gateway endpoint com.amazonaws.us-east-1.s3 associated with the nodes' route tables.
  • Endpoints exist but still time out: the endpoints' security group must allow inbound TCP 443 from the nodes (their security group or the VPC CIDR). If the S3 endpoint has a restrictive policy, allow arn:aws:s3:::prod-us-east-1-starport-layer-bucket/*.
  • Images from outside AWS (Docker Hub, GHCR, Quay) need a NAT route, or a pull-through cache in ECR (the first pull of each image still needs an internet route; see card 3).

Prevent

  • Before removing a NAT route, make sure every image the cluster runs comes from ECR, and test a fresh node pull.
  • Manage the VPC endpoints in the same code as the cluster, and include sts, ec2 and the other endpoints nodes and add-ons need.

Understand it

Private clusters and endpoint access

The image is large, so the pull is slow or gets cancelled

Pulling image ContainerCreating context canceled

Back to the tree

What you see

$ kubectl get pod trainer-0 -n ml
NAME        READY   STATUS              RESTARTS   AGE
trainer-0   0/1     ContainerCreating   0          9m

Events:
  Normal   Pulling  9m   kubelet  Pulling image "111122223333.dkr.ecr.us-east-1.amazonaws.com/ml/trainer:2.1"
# A pull with no progress for 5 minutes is cancelled by containerd
  Warning  Failed   kubelet  Failed to pull image "111122223333.dkr.ecr.us-east-1.amazonaws.com/ml/trainer:2.1": rpc error: code = Canceled desc = failed to pull and unpack image "111122223333.dkr.ecr.us-east-1.amazonaws.com/ml/trainer:2.1": ...: context canceled
# The slow pull that did finish
  Normal   Pulled   kubelet  Successfully pulled image "111122223333.dkr.ecr.us-east-1.amazonaws.com/ml/trainer:2.1" in 7m42.1s (7m42.1s including waiting). Image size: 11834212345 bytes.

Here nothing failed yet: a Pulling event with no Pulled after it means the pull is still running. Multi-gigabyte images (CUDA, ML frameworks) can take minutes per node, and longer when many Pods pull at once.

Confirm

# How big is the image? (compressed size, in bytes)
aws ecr describe-images --repository-name ml/trainer --image-ids imageTag=2.1 --query 'imageDetails[0].imageSizeInBytes'
# How long did earlier pulls take on this node type?
kubectl get events -A --field-selector reason=Pulled | grep trainer
# Did containerd cancel a stalled pull? (on the node, via SSM Session Manager)
journalctl -u containerd | grep "no progress"
# Is the node short on disk for unpacking it?
kubectl describe node ip-10-0-2-41.ec2.internal | grep -A6 Conditions

Fix

  • Make the image smaller: multi-stage builds, a slim base image, and model weights loaded from S3 or a volume at start instead of baked into the image.
  • Pull faster: SOCI parallel pull mode, built into the EKS-optimized AL2023 AMI and Bottlerocket (on Bottlerocket, settings.container-runtime.snapshotter = "soci" with pull-mode = "parallel-pull-unpack"), downloads layers in chunks over several connections and unpacks them in parallel. It trades CPU, network and disk I/O for shorter pulls.
  • Pre-pull: cache big images on the node's data volume (a Bottlerocket data volume snapshot or a custom AMI), or run a DaemonSet that pulls the image on every matching node ahead of the workload.
  • Disk: give nodes a big enough root or data volume; a large image that doesn't fit fails with no space left on device.

Prevent

  • Track image size in CI and fail builds that grow past a budget.
  • For GPU and ML node pools, plan the image cache and disk size with the node design, not after the first slow rollout.

Understand it

GPU nodes and model servingPods

4Diagnose it

Six Pods stuck pulling their image. Read the output and pick the cause. A wrong pick tells you why it doesn't fit; there's no score.

Diagnose it