CrashLoopBackOff and OOMKilled on Amazon EKS: Fix "Back-off restarting failed container"

One card per cause. Each shows what the output looks like, the commands that confirm it, the fix, how to keep it from coming back, and the walkthrough that explains the mechanism.

Killed for running out of memory

OOMKilled Exit Code: 137

Back to the tree

What you see

$ kubectl describe pod orders-api-6b8f9c7d4-q7m2x -n shop
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
    Ready:          False
    Restart Count:  4
    Limits:
      memory:  512Mi
    Requests:
      cpu:     250m
      memory:  512Mi

The kernel's OOM killer stopped the container: usually because it reached its own memory limit, sometimes because the whole node ran out of memory. 137 is 128 + 9, the SIGKILL signal. The app gets no warning, so logs --previous usually just stops mid-stream.

Confirm

# How the last run ended, and the memory limit
kubectl get pod orders-api-6b8f9c7d4-q7m2x -n shop -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}{" "}{.spec.containers[0].resources.limits.memory}{"\n"}'
# Live usage per container (needs metrics-server); watch it climb before the kill
kubectl top pod -n shop -l app=orders-api --containers

Exit code 137 without OOMKilled is a SIGKILL from something else, most often the kubelet stopping a container that failed its liveness probe and didn't exit on SIGTERM in time (card 2). If the container was well below its limit, check kubectl describe node for a SystemOOM event: the node as a whole ran out of memory, and the fix is node sizing and requests that reflect real usage, not a higher limit.

Fix

The limit is below what the app really needs: raise resources.limits.memory to the measured peak plus headroom. For important workloads set the memory request equal to the limit, so the node reserves all of it.
The runtime is sized past the limit: a JVM with -Xmx equal to the limit goes over once threads, metaspace and buffers are added. Size the heap below the limit with -XX:MaxRAMPercentage=75, or cap Node.js with --max-old-space-size; the app then fails with its own out-of-memory error instead of a silent kill. A JVM that can't read cgroup v2 limits (JDK 8 older than 8u372, on Amazon Linux 2023 or Bottlerocket) sizes its heap from the node's memory, so upgrade it.
Memory grows until it dies: that's a leak, and a bigger limit only delays the next kill. Profile it or take a heap dump.

Prevent

Set limits from measured usage (Container Insights, Prometheus container_memory_working_set_bytes, or VPA recommendations), not guesses.
Alert on kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} from kube-state-metrics, so you see OOM kills before they become a crash loop.

Understand it

Requests, limits, QoS and evictions

Whole container, not one process: on cgroup v2 nodes (Amazon Linux 2023, Bottlerocket), Kubernetes 1.28 and later kill every process in the container when it hits its limit. Earlier, only the largest process died, and a container whose worker process was killed could keep running half broken. Since 1.32 the kubelet setting singleProcessOOMKill: true brings the old behaviour back.

The liveness probe kills a slow or busy app

Liveness probe failed failed liveness probe, will be restarted

Back to the tree

What you see

Events:
  Type     Reason     From     Message
  Warning  Unhealthy  kubelet  Liveness probe failed: Get "http://10.0.2.41:8080/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   Killing    kubelet  Container app failed liveness probe, will be restarted
  Warning  BackOff    kubelet  Back-off restarting failed container app in pod orders-api-6b8f9c7d4-q7m2x_shop(8d2e4c1a-5b7f-4e3a-9c6d-2f1b0a7e3c94)

Under Last State you see Reason: Error with Exit Code: 143 (the app exited on SIGTERM) or 137 (it didn't stop within the grace period and got SIGKILL), or Completed with 0 if the app shuts down cleanly on SIGTERM. The logs often look healthy right up to the shutdown.

Confirm

# The probe settings: delay, timeout, period, failure threshold
kubectl describe pod orders-api-6b8f9c7d4-q7m2x -n shop | grep -E "Liveness|Readiness|Startup"
# Probe failures for this Pod
kubectl get events -n shop --field-selector involvedObject.name=orders-api-6b8f9c7d4-q7m2x,reason=Unhealthy
# How long does the endpoint really take? (from a throwaway Pod)
kubectl run curl -n shop --rm -it --image=curlimages/curl --restart=Never -- curl -sS -m 5 -o /dev/null -w '%{http_code} %{time_total}s\n' http://10.0.2.41:8080/healthz

Fix

The app starts slowly (migrations, cache warm-up, a big JVM): add a startupProbe, for example the same endpoint with periodSeconds: 10 and failureThreshold: 30 for up to 5 minutes. Liveness and readiness checks don't start until it succeeds.
The app is busy and the check times out: raise timeoutSeconds (default 1 second) and failureThreshold (default 3), and make the liveness endpoint cheap: answered by the process itself, without calling the database or other services.
Liveness checks a dependency: then a database blip restarts every replica at once. Move dependency checks to the readiness probe, which only takes the Pod out of its Services.
A low CPU limit can throttle the health handler past its timeout under load. Raise the limit, or remove the CPU limit and keep the request.

Prevent

Keep the liveness probe to one question: is this process stuck beyond repair? Answer it locally and fast.
Load-test with the probe settings you ship. Timeouts come from GC pauses and CPU throttling under load, not from an idle test Pod.

Understand it

Liveness, readiness and startup probes CPU throttling and latency

The app exits with an error, or just ends

Reason: Error Exit Code: 1 Reason: Completed

Back to the tree

What you see

    Last State:     Terminated
      Reason:       Error
      Exit Code:    2

$ kubectl logs orders-api-6b8f9c7d4-q7m2x -n shop --previous
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6f3a2b]

goroutine 1 [running]:
main.loadPricing(...)
        /src/pricing.go:42

    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0

A Go panic exits with code 2; most other runtimes exit 1 on an unhandled error. The second block also loops: Pods of a Deployment have restartPolicy: Always, so a main process that finishes cleanly (a one-off script, or a server that forks into the background) is restarted too.

Confirm

# The previous run's output (the current run may not have printed anything yet)
kubectl logs orders-api-6b8f9c7d4-q7m2x -n shop -c app --previous
# Reason and exit code of the last run, per container
kubectl get pod orders-api-6b8f9c7d4-q7m2x -n shop -o jsonpath='{range .status.containerStatuses[*]}{.name}{" "}{.lastState.terminated.reason}{" "}{.lastState.terminated.exitCode}{"\n"}{end}'
# Did it start with a new image or config? Compare revisions
kubectl rollout history deployment/orders-api -n shop

An app can write its exit reason to /dev/termination-log (the container's terminationMessagePath), and kubectl describe pod shows it as Message under Last State. With terminationMessagePolicy: FallbackToLogsOnError, an error exit with an empty termination file uses the last log lines instead (up to 80 lines or 2 KB).

Fix

A bug in the new version: fix what the stack trace points at. Meanwhile kubectl rollout undo deployment/orders-api -n shop goes back to the previous ReplicaSet.
Exit 0, Completed: run one-off work as a Job, not a Deployment, and run servers in the foreground as the container's main process.
Exit codes above 128 are signals: 137 is SIGKILL, 143 SIGTERM, 139 SIGSEGV (a crash in native code).

Prevent

Roll out with maxUnavailable: 0: new Pods that never become Ready then don't replace the old ones, and the rollout reports failure after progressDeadlineSeconds.
Log the reason to stdout or stderr before exiting, so logs --previous has it.

Understand it

Pods Deployments and ReplicaSets

Missing or wrong config, Secret or environment variable

environment variable is not set no such file or directory

Back to the tree

What you see

$ kubectl logs orders-api-6b8f9c7d4-q7m2x -n shop --previous
Error: required environment variable DATABASE_URL is not set

$ kubectl logs orders-api-6b8f9c7d4-q7m2x -n shop --previous
2026/11/02 09:41:09 open /etc/orders/config.yaml: no such file or directory

These are the app's own words, so they vary. The pattern is the same: the app starts, reads its config, and quits within a second or two.

Confirm

# What the container is given: env, envFrom and volume mounts
kubectl get pod orders-api-6b8f9c7d4-q7m2x -n shop -o jsonpath='{.spec.containers[0].env}{"\n"}{.spec.containers[0].envFrom}{"\n"}{.spec.containers[0].volumeMounts}{"\n"}'
# Which keys exist? (describe shows key names and sizes, not values)
kubectl describe configmap orders-config -n shop
kubectl describe secret orders-db -n shop
# Start a copy of the Pod with a shell instead of the app, and look around
kubectl debug orders-api-6b8f9c7d4-q7m2x -n shop -it --copy-to=orders-debug --container=app -- sh
kubectl delete pod orders-debug -n shop

Inside the copy, env | sort and ls /etc/orders show exactly what the app saw. It needs a shell in the image. For a distroless image, add a debug container to a copy (kubectl debug POD -it --image=busybox:1.36 --share-processes --copy-to=orders-debug) and read /proc/<app-pid>/environ and /proc/<app-pid>/root/etc/orders while the app process is still alive.

Fix

A key is missing or misnamed: add it to the ConfigMap or Secret, or fix the name the app or the manifest uses. envFrom silently skips keys that aren't there, so a renamed key shows up only as the app's error.
A file the app can't find: check the volume's mountPath and subPath, and that the ConfigMap key matches the file name the app opens. Files mounted with subPath don't update when the ConfigMap changes.
Values from AWS Secrets Manager (External Secrets or the Secrets Store CSI driver): check the ExternalSecret's status or the SecretProviderClass, and that the Pod's IAM role may read the secret.
Environment variables are read only when the container starts. After fixing them, kubectl rollout restart deployment/orders-api -n shop rolls out fresh Pods.

Prevent

Validate config at startup and print exactly which key or file is missing before exiting.
Leave required references required (no optional: true), so a missing ConfigMap or Secret stops the container before it starts, with a clear message, instead of a crash later.

Understand it

ConfigMaps and Secrets Secrets from AWS

Not a crash loop: CreateContainerConfigError. If a ConfigMap or Secret referenced in env or envFrom doesn't exist, or a valueFrom key is missing, the container is never created: STATUS reads CreateContainerConfigError, the event says Error: secret "orders-db" not found or couldn't find key DB_PASSWORD in Secret shop/orders-db, and there are no logs. runAsNonRoot: true on an image that runs as root fails the same way (container has runAsNonRoot and image will run as root). A missing ConfigMap mounted as a volume shows as ContainerCreating with a FailedMount event instead. Fix the reference; the ConfigMaps and Secrets walkthrough shows how each one is wired.

A dependency isn't reachable when the app starts

connection refused no such host i/o timeout

Back to the tree

What you see

$ kubectl logs orders-api-6b8f9c7d4-q7m2x -n shop --previous
2026-11-02T09:41:08Z INFO  connecting to postgres at orders-db.shop.svc.cluster.local:5432
2026-11-02T09:41:09Z FATAL dial tcp 10.100.37.12:5432: connect: connection refused

FATAL dial tcp: lookup orders-db.shop.svc.cluster.local on 10.100.0.10:53: no such host
FATAL dial tcp 10.0.48.200:5432: i/o timeout

The app quits because it can't reach a database, cache or API in its first seconds. Typical when a whole namespace is deployed at once, or when an RDS security group doesn't allow the Pods in.

Confirm

# Is the Service there, and are the Pods behind it Ready?
kubectl get service orders-db -n shop
kubectl get endpointslices -n shop -l kubernetes.io/service-name=orders-db -o wide
kubectl get pods -n shop -l app=orders-db
# Test the path from a throwaway Pod in the same namespace
kubectl run nettest -n shop --rm -it --image=nicolaka/netshoot --restart=Never -- nc -zv -w 3 orders-db 5432

connection refused: the address answered but nothing listens there (database still starting or wrong port), or a Service with no ready endpoints, where kube-proxy rejects the connection itself. no such host: a wrong name, or the Service doesn't exist. i/o timeout: packets are dropped, usually by a security group or a NetworkPolicy.

Fix

Retry instead of exiting: make the app retry its connections with backoff at startup. It's the fix that also survives the dependency restarting later.
Or wait before starting: an init container that loops until the dependency answers, for example until pg_isready -h orders-db -p 5432; do sleep 2; done in a postgres image. The app container starts only after it exits.
i/o timeout to RDS: allow the nodes' security group (or the Pod's, with security groups for Pods) on the database port in the RDS security group. Inside the cluster, check NetworkPolicies on both namespaces.
Kubernetes has no "depends on" between Deployments. Ordering exists only inside a Pod: init containers, then app containers.

Prevent

Retry with backoff in every client, and let the readiness probe (not liveness) reflect whether dependencies are reachable.
Log the host, port and error on every failed attempt, so the cause is in logs --previous.

Understand it

Pods (init containers)Services and kube-proxy

Read-only root filesystem, or a non-root user can't write

Read-only file system Permission denied

Back to the tree

What you see

$ kubectl logs web-5d7c9b8f6-2kq4r -n shop --previous
nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (30: Read-only file system)

$ kubectl logs web-5d7c9b8f6-2kq4r -n shop --previous
nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)

Usually right after hardening: readOnlyRootFilesystem: true was added, or runAsNonRoot and runAsUser were set so the Pod passes the Pod Security Admission restricted level. Error 30 means nobody can write there; error 13 means this user can't.

Confirm

# Pod-level and container-level security settings
kubectl get pod web-5d7c9b8f6-2kq4r -n shop -o jsonpath='{.spec.securityContext}{"\n"}{.spec.containers[0].securityContext}{"\n"}'
# Which user runs, and who owns the path? (in a copy of the Pod with a shell instead of nginx)
kubectl debug web-5d7c9b8f6-2kq4r -n shop -it --copy-to=web-debug --container=nginx -- sh -c 'id; ls -ld /var/cache/nginx'
kubectl delete pod web-debug -n shop

Fix

Read-only root filesystem: keep it, and mount an emptyDir volume at each path the app writes: here /var/cache/nginx, plus /var/run for the pid file, or /tmp.
Non-root user: use an image built to run unprivileged (for nginx, nginxinc/nginx-unprivileged, which listens on 8080), or give the paths to that UID in the Dockerfile. For mounted volumes, fsGroup in the Pod's securityContext makes them writable by the group.
A port below 1024: on containerd 2.x nodes (current EKS Amazon Linux 2023 AMIs) a non-root process may bind it. On containerd 1.x nodes, or with hostNetwork: true, it fails with bind() to 0.0.0.0:80 failed (13: Permission denied). Listening on 8080 and pointing the Service's targetPort at it works everywhere.

Prevent

Test images with the security settings you'll ship, for example docker run --read-only --user 101 ….
Give your base Helm chart emptyDir mounts for the usual writable paths, so every service starts with them.

Understand it

Pod Security Admission Runtime and image security

Wrong command, entrypoint or CPU architecture

StartError executable file not found in $PATH exec format error

Back to the tree

What you see

    Last State:     Terminated
      Reason:       StartError
      Message:      failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "orders-server": executable file not found in $PATH: unknown
      Exit Code:    128

$ kubectl logs orders-api-6b8f9c7d4-q7m2x -n shop --previous
exec /usr/local/bin/orders-server: exec format error
$ kubectl get node ip-10-0-3-17.ec2.internal -L kubernetes.io/arch
NAME                        STATUS   ROLES    AGE   VERSION               ARCH
ip-10-0-3-17.ec2.internal   Ready    <none>   2d    v1.34.1-eks-113cf36   arm64

In the first block the process never started: the command named in the image or in command: doesn't exist in the image. In the second the Pod landed on an arm64 (Graviton) node, and the image was built only for amd64 (x86), so the kernel can't run the binary; Last State shows Reason: Error with Exit Code: 255.

Confirm

# Which command does the Pod run? (empty means the image's ENTRYPOINT and CMD)
kubectl get pod orders-api-6b8f9c7d4-q7m2x -n shop -o jsonpath='{.spec.containers[0].command}{" "}{.spec.containers[0].args}{"\n"}'
# Which node, and which architecture?
kubectl get pod orders-api-6b8f9c7d4-q7m2x -n shop -o wide
kubectl get nodes -L kubernetes.io/arch,node.kubernetes.io/instance-type
# Which platforms does the image support?
docker buildx imagetools inspect 111122223333.dkr.ecr.us-east-1.amazonaws.com/orders-api:2.14.0

A multi-arch image lists a manifest per platform, such as linux/amd64 and linux/arm64. If the image is an index (multi-arch, or a docker buildx image with its default attestations) with no entry for the node's platform, the pull fails instead (no match for platform in manifest) and the Pod shows ErrImagePull, not a crash loop. Only a plain single-platform manifest pulls fine and then fails with exec format error.

Fix

Wrong command: in Kubernetes, command replaces the image's ENTRYPOINT and args replaces its CMD. Use the binary's full path, or drop command to keep the image's own.
A shell wrapper that can't find the binary exits with 127 (sh: orders-server: not found) and Reason: Error rather than StartError; the fix is the same.
Wrong architecture: build multi-arch images (docker buildx build --platform linux/amd64,linux/arm64 --push …), or keep amd64-only workloads off Graviton with nodeSelector: kubernetes.io/arch: amd64, or a kubernetes.io/arch requirement in the Karpenter NodePool.

Prevent

Build every image for both architectures in CI before a NodePool allows arm64. Karpenter often picks Graviton when it's allowed, because it's cheaper.
Set the entrypoint in the Dockerfile in exec form with a full path, for example ENTRYPOINT ["/usr/local/bin/orders-server"].

Understand it

Spot and Graviton Pods

CrashLoopBackOff and OOMKilled on Amazon EKS

1Start here

Run this first

2Decision tree

3Causes and fixes

Killed for running out of memory

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

The liveness probe kills a slow or busy app

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

The app exits with an error, or just ends

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

Missing or wrong config, Secret or environment variable

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

A dependency isn't reachable when the app starts

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

Read-only root filesystem, or a non-root user can't write

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

Wrong command, entrypoint or CPU architecture

visibilityWhat you see

fact_checkConfirm

buildFix

shieldPrevent

schoolUnderstand it

4Diagnose it

Related

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it

What you see

Confirm

Fix

Prevent

Understand it