DiskPressure evicted your database to reclaim a few gigabytes of logs — here's how to stop it

The page goes off at 3 a.m. Your primary Postgres pod is Terminating, then gone, and a fresh one is stuck ContainerCreating on a different node. Nothing OOM-killed. CPU was fine. When you describe the pod you find the real story in one line:

Status:   Failed
Reason:   Evicted
Message:  The node was low on resource: ephemeral-storage.
          Container db was using 1.4Gi, which exceeds its
          request of 0.

Events:
  Warning  EvictionThresholded   node/ip-10-0-3-88   attempting
           to reclaim ephemeral-storage
  Warning  ImageGCFailed         node/ip-10-0-3-88   failed to
           garbage collect required amount of images
  Normal   NodeHasDiskPressure   node/ip-10-0-3-88   status is
           now: NodeHasDiskPressure

The chain is brutal and completely mechanical. Some noisy neighbor — a chatty sidecar, a log shipper that stopped shipping, an image the node kept pulling — filled the node's root filesystem. The kubelet noticed free space cross a threshold, raised DiskPressure, tried to reclaim space by deleting images, couldn't free enough, and then did the only other thing it knows how to do: it started evicting pods. Your database happened to sit near the front of that line, because you never told the kubelet it was special.

This is a host-level systems problem, and it wants a host-level fix. You cannot YAML your way out of a full disk from inside a pod. Below: exactly how the kubelet computes the eviction decision, then how to make the node's volatile directories physically incapable of taking your workloads down with them, and finally how to make sure that if something is going to die, it is never the pod that matters.

How the kubelet decides who dies

The kubelet's eviction manager is a small control loop, evaluated every housekeeping-interval (default 10s). On each pass it reads a set of eviction signals from the node's cAdvisor/CRI stats and compares them against configured thresholds. For disk, the signals that matter are computed straight from the filesystem:

Two things about that table decide most incidents. First, on a default EKS node there is one EBS volume: /, holding the OS, /var/lib/containerd (images), and /var/lib/kubelet (emptyDir + logs) all together. So nodefs and imagefs are the same physical disk — anything that fills it trips everything at once. Second, these are percentages of free space, so a small root volume trips far sooner than operators expect: on a 20 GiB root, nodefs.available<10% means eviction starts with 2 GiB still free.

Hard vs soft: the grace period you probably don't have

Thresholds come in two flavors. A hard threshold (the ones above) triggers immediate eviction with a 0s grace period — the kubelet does not honor terminationGracePeriodSeconds and does not consult PodDisruptionBudgets. A soft threshold adds an eviction-soft-grace-period: the signal must stay below the line for that whole window before the kubelet acts, and it then honors up to eviction-max-pod-grace-period for a clean shutdown. Out of the box, EKS ships hard thresholds only. That means the first time you learn about disk pressure, a stateful pod is already being killed without the graceful-shutdown window your application assumes it has. Adding a soft threshold slightly above the hard one buys you a warning band and a chance to drain cleanly.

Image garbage collection runs first — until it can't

Before evicting user pods, the kubelet tries the cheap fix: delete images nobody is using. Image GC is governed by two knobs, imageGCHighThresholdPercent (default 85) and imageGCLowThresholdPercent (default 80). When image-disk usage climbs above 85%, the kubelet deletes images ordered by last-used time, oldest first, until usage drops back to 80%. It also sweeps dead containers and images on a schedule — unused images every five minutes, unused containers every minute.

The trap is ImageGCFailed. If every remaining image is currently in use (running pods reference them), the kubelet has nothing safe to delete and cannot reach the low threshold. It records ImageGCFailed and, because disk is still over the eviction line, escalates to killing pods. So ImageGCFailed in your events is rarely the disease — it is the kubelet announcing that the polite option failed and the impolite one is next. Note the ordering interaction: if imageGCHighThresholdPercent is set higher than the point where imagefs.available eviction triggers, eviction can start before GC ever runs, which is its own footgun. Keep the GC high threshold below the eviction line.

The ordering: usage over request, then Priority

When the kubelet must evict, it does not pick at random. It builds a ranked list of pods and kills from the top. The ranking is, in order:

1. Whether the pod's usage of the starved resource exceeds its request. Pods using more ephemeral storage than they requested are sorted ahead of pods using less than (or within) their request. A pod with no request has an effective request of zero, so any usage counts as "over request."
2. Pod Priority. Among pods that are over request, lower-spec.priority pods are ranked to die first; higher-priority pods are evicted last.
3. How far over request the pod is — the biggest overspender in the group goes first.

Read that carefully against the default state of most manifests: no ephemeral-storage request (so everything is "over request") and no PriorityClass (so everything is priority 0). In that world the tiebreaker collapses to raw usage, and a busy database writing WAL and temp files to an emptyDir can easily be the single largest disk user on the node — which puts it at the front of the kill list. The two levers you actually control are exactly the two the kubelet sorts on: set a request so the pod isn't automatically "over," and set a priority so it sorts to the back. We do both in the last section.

Eviction is not OOMKill — don't confuse the two

One clarification that saves hours of misdiagnosis. Node-pressure eviction is the kubelet reacting to a node-level resource signal (disk here, or memory.available) by terminating whole pods; the pod shows Reason: Evicted. OOMKill is the Linux kernel killing a single process when a container blows past its memory limit (or the node truly runs out of RAM before the kubelet can act); you see OOMKilled and exit code 137. For memory pressure the kernel also weighs in via oom_score_adj — Guaranteed pods get -997 (near-invincible), BestEffort get 1000 (first to die), Burstable get a value scaled by how small their memory request is relative to node capacity. Disk pressure has no kernel equivalent: there is no "OOM killer for disk." The kubelet is the only actor, which is precisely why tuning it — and the disk underneath it — is the whole game.

The platform fix: give the volatile directories their own disk

Every manifest-level mitigation is downstream of one physical fact: on a stock node the OS root and the churning container directories share a single volume. The moment you split them, a runaway logger or image storm can fill its disk to 100% and the kubelet never raises DiskPressure against the volume the OS and system daemons live on. You have converted a node-killing event into a localized, self-correcting one. This is a node-image / launch-template change, and it is the highest-leverage thing in this article.

The directories to isolate are the two that actually churn:

• /var/lib/containerd — the container image content store and snapshots. This is your imagefs. Image bloat lives here.
• /var/lib/kubelet — emptyDir volumes, pod logs, and (when imagefs isn't separate) the writable container layer. This is your nodefs. Runaway application writes live here.

Bottlerocket does this by default — that's the point

Bottlerocket ships a two-volume design out of the box: a small, immutable OS volume (active/passive partitions, dm-verity, the API datastore) and a separate data volume that holds container images, orchestration state, and ephemeral storage. On boot Bottlerocket grows the data partition to fill the device, and if you enlarge the EBS volume you reboot to extend it. You don't script anything — you just size the data volume in the launch template. That is a large part of why Bottlerocket nodes are calmer under disk pressure: the OS physically cannot be starved by container churn.

## Launch template block device mapping: root + a dedicated data EBS
##   /dev/xvda -> 20 GiB   (OS root)
##   /dev/xvdb -> 200 GiB gp3, 500 MB/s, 6000 IOPS  (containerd + kubelet)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -euo pipefail
DEV=/dev/nvme1n1                      # the second EBS (xvdb) as NVMe

# Stop the writers before we move their state
systemctl stop kubelet containerd || true

mkfs.xfs -f "$DEV"
mkdir -p /mnt/data
mount "$DEV" /mnt/data

# IMPORTANT (AL2023): copy, don't rm. Newer AL2023 AMIs pre-cache the
# pause image under /var/lib/containerd; deleting it makes nodes fail
# to join. rsync preserves the cached content.
rsync -aXS /var/lib/containerd/ /mnt/data/containerd/
rsync -aXS /var/lib/kubelet/    /mnt/data/kubelet/
umount /mnt/data

# Mount both volatile dirs onto the data volume via bind-friendly layout
mount "$DEV" /mnt/data
mkdir -p /mnt/data/containerd /mnt/data/kubelet
mount --bind /mnt/data/containerd /var/lib/containerd
mount --bind /mnt/data/kubelet    /var/lib/kubelet

# Persist across reboots
echo "$DEV /mnt/data xfs defaults,noatime 0 2" >> /etc/fstab
echo "/mnt/data/containerd /var/lib/containerd none bind 0 0" >> /etc/fstab
echo "/mnt/data/kubelet    /var/lib/kubelet    none bind 0 0" >> /etc/fstab

systemctl start containerd kubelet
--//--

# Bottlerocket: no script needed. Attach a data volume in the launch
# template and Bottlerocket uses it for /var/lib automatically.

# --- Launch template blockDeviceMappings (Terraform) ---
block_device_mappings {
  device_name = "/dev/xvda"           # OS volume (keep small)
  ebs { volume_size = 4  volume_type = "gp3" }
}
block_device_mappings {
  device_name = "/dev/xvdb"           # DATA volume (containers live here)
  ebs {
    volume_size = 200
    volume_type = "gp3"
    throughput  = 500
    iops        = 6000
  }
}

# --- Bottlerocket user data (TOML): tune the kubelet in one place ---
[settings.kubernetes]
"image-gc-high-threshold-percent" = "80"
"image-gc-low-threshold-percent"  = "70"

[settings.kubernetes.eviction-hard]
"nodefs.available"  = "12%"
"imagefs.available" = "15%"

[settings.kubernetes.eviction-soft]
"nodefs.available"  = "20%"

[settings.kubernetes.eviction-soft-grace-period]
"nodefs.available"  = "2m"

Workload insurance: make sure the right pod dies last

Filesystem isolation stops most incidents, but disks can still fill — a data volume is smaller than infinity. So set the two levers the eviction ranker actually sorts on, and cap the leaky writers before they matter.

1. PriorityClass so critical pods are evicted last

A PriorityClass is a cluster-scoped object mapping a name to an integer value (user-defined range up to 1,000,000,000; the built-ins system-cluster-critical and system-node-critical sit at 2000000000 and 2000001000 and are reserved for control-plane components). Give your critical stateful pods a high value and the kubelet sorts them to the back of the eviction list within the "over request" tier. Don't overreach — reserve the system-critical classes for what they're for; a high app-level value is plenty.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: critical-stateful
value: 1000000                 # high, but below the reserved system tiers
globalDefault: false
description: "Primary datastores. Evicted only after everything else."
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-db
spec:
  replicas: 1
  selector: { matchLabels: { app: orders-db } }
  template:
    metadata:
      labels: { app: orders-db }
    spec:
      priorityClassName: critical-stateful      # <-- the insurance
      containers:
        - name: db
          image: postgres:16
          resources:
            requests:
              cpu: "1"
              memory: 2Gi
              ephemeral-storage: 2Gi            # not "over request" for 2Gi
            limits:
              memory: 2Gi                        # Guaranteed-ish for memory
              ephemeral-storage: 4Gi             # hard cap on local disk

2. ephemeral-storage requests and limits on everything

Ephemeral storage is the writable container layer, container logs, and emptyDir volumes combined. Two reasons to declare it on every pod. A request keeps the pod from being auto-classified as "over request" (the first sort key) the instant it writes a byte, and it lets the scheduler avoid packing a node past its ephemeral capacity. A limit is a scalpel: if a single container exceeds its own ephemeral-storage limit, the kubelet evicts that pod specifically — for its own overspend — rather than waiting for the whole node to hit DiskPressure and taking a bystander with it. Cap emptyDir volumes independently with sizeLimit; a pod whose emptyDir exceeds its sizeLimit is evicted on the spot.

apiVersion: v1
kind: Pod
metadata:
  name: log-shipper
spec:
  priorityClassName: low-priority         # define separately, e.g. value: 100
  containers:
    - name: shipper
      image: fluent/fluent-bit:3
      resources:
        requests:
          ephemeral-storage: 256Mi
        limits:
          ephemeral-storage: 1Gi          # blows past 1Gi -> only THIS pod dies
      volumeMounts:
        - name: scratch
          mountPath: /var/log/buffer
  volumes:
    - name: scratch
      emptyDir:
        sizeLimit: 512Mi                   # emptyDir cap, enforced independently

3. Tune the kubelet: soft band + sane image GC

On AL2023 you deliver kubelet configuration through nodeadm NodeConfig (a drop-in under /etc/eks/nodeadm.d/ or the managed node group's NodeConfig). Add a soft threshold above the hard one for a graceful-shutdown window, and keep image GC's high threshold below the eviction line so GC always runs before eviction.

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  kubelet:
    config:
      # Image GC fires before eviction (85/80 defaults -> tighten a little)
      imageGCHighThresholdPercent: 80
      imageGCLowThresholdPercent: 70
      # Hard thresholds: immediate, 0s grace (the last line of defense)
      evictionHard:
        nodefs.available: "10%"
        nodefs.inodesFree: "5%"
        imagefs.available: "15%"
      # Soft thresholds: a warning band with a real grace period
      evictionSoft:
        nodefs.available: "15%"
        imagefs.available: "20%"
      evictionSoftGracePeriod:
        nodefs.available: "2m"
        imagefs.available: "2m"
      evictionMaxPodGracePeriod: 60
      # Reserve headroom so system daemons never compete with pods
      systemReserved:
        ephemeral-storage: "2Gi"
      kubeReserved:
        ephemeral-storage: "2Gi"

4. Rotate logs at the source

None of the above helps if a container writes an unbounded log to its own filesystem. Container stdout/stderr is rotated by the runtime (containerd caps and rotates via the kubelet's containerLogMaxSize / containerLogMaxFiles, defaulting to 10Mi × 5), but application logs written to files inside the container — or to an emptyDir — are not. Rotate them at the sidecar (logrotate, or the log agent's own buffer caps) and give that emptyDir a sizeLimit. A log shipper that stops shipping is the single most common trigger for the whole cascade in Figure 1; treat its buffer as hostile by default.

Pitfalls that bite in production