When an AWS Availability Zone goes dark, your stateful pods hang in ContainerCreating — here's how to design around it

Your monitoring lights up: an entire Availability Zone has gone dark — power event, network partition, the kind of blast radius AWS designs zones to contain. The good news arrives fast. Kubernetes notices the node is gone, and within a minute the scheduler places your payments-0 pod onto a healthy node in another AZ. The bad news arrives slower, and it doesn't leave: the new pod sits in ContainerCreating, minute after minute, and kubectl describe pod tells you exactly why:

Warning  FailedAttachVolume  attachdetach-controller
  Multi-Attach error for volume "pvc-9f3c..." Volume is already
  exclusively attached to one node and can't be attached to another

The volume isn't actually attached to anything you can see — pull up the EC2 console and it shows available. But Kubernetes still believes it belongs to the ghost node in the dead AZ, and it will not force the issue. This is not a bug. It is the storage layer protecting your data, and understanding why is the difference between an outage that self-heals in seven minutes and one that corrupts a database.

Why the rescheduled pod is stuck: the exclusive-attachment lockout

Start with the mechanics, because every design decision later hangs off them. An EBS volume claimed through a PersistentVolumeClaim is, by default, ReadWriteOnce (RWO): it can be attached to a single node, read-write, at any moment. That is not a Kubernetes limitation invented for convenience — it mirrors the physical reality that a standard block device with an XFS or EXT4 filesystem is designed for exactly one operating system mounting it. Two nodes writing the same EXT4 filesystem at once is not "degraded"; it is guaranteed corruption.

So when the node in the dead AZ was healthy, three facts were true and consistent: the volume was attached to that node, a VolumeAttachment object recorded that binding in the API server, and the pod ran happily on top. When the AZ blacked out, the node stopped answering the kubelet heartbeat, the node object flipped to NotReady, and after the eviction timeout Kubernetes rescheduled the pod. But here is the trap: the VolumeAttachment object still says the volume belongs to the ghost node. Nobody was able to run the clean detach handshake, because the node that owned the mount is unreachable.

The new pod's attach request therefore collides with a still-live attachment record for a different node. The attach/detach controller inside kube-controller-manager emits the Multi-Attach error and waits. It is deliberately refusing to yank the volume away, because from the control plane's point of view it cannot prove the old node has stopped writing. The only thing it knows for certain is that forcing a detach from a node that might still be alive is how you destroy a filesystem.

Why the volume can't just follow the pod — and why Multi-Attach won't save you

It's tempting to look at the diagram and think: fine, just let the volume move to AZ-b. It can't, and the reason is physical, not a policy you can flip. An EBS volume is a block device that physically lives inside one Availability Zone. AWS will not attach a volume in us-east-1a to an EC2 instance running in us-east-1b — there is no wire for it. When the pod lands in a healthy AZ, the disk it wants is still sitting in the smoking one. The best the system can do is wait for the dead AZ to come back and reattach there, which is exactly what the six-minute path eventually does — assuming the AZ returns.

The other tempting shortcut is EBS Multi-Attach: "if the volume could attach to two nodes, there'd be no exclusive lock." This is a trap worth spelling out, because it looks like a fix and is not one. Multi-Attach carries a stack of constraints that disqualify it as a general availability mechanism:

Read that "same AZ" row again. Multi-Attach is a single-AZ high-availability trick for two instances that share a clustered filesystem — think a legacy active/passive cluster manager. It is orthogonal to zone resilience. If your goal is to survive a zone going dark, Multi-Attach is not on the table. That leaves two real levers, and you generally want both: keep pod and volume co-located so you never trip a different failure (topology), and recover fast when a zone dies (fencing) or don't depend on the volume at all (replication).

Step one: keep the pod and its volume in the same AZ

Before you can recover gracefully, you have to stop creating avoidable mismatches. The default, naive behavior — volumeBindingMode: Immediate — provisions the EBS volume the moment the PersistentVolumeClaim appears, picking an AZ with no idea where the pod will actually run. On a busy multi-AZ cluster that's a coin flip you'll lose often enough to page someone.

The fix is to invert the order: let the scheduler choose the node first, then provision the volume in that node's AZ. That's exactly what WaitForFirstConsumer does. Binding is delayed until a pod consuming the claim is scheduled; only then does the EBS CSI driver create the volume, in the zone the pod landed in. The AWS EKS Best Practices guidance is blunt about this: WaitForFirstConsumer should always be used for zone-restricted cloud storage like EBS.

Here is the StorageClass that encodes both halves: gp3 for cost, WaitForFirstConsumer so the volume follows the pod's node, and allowedTopologies to constrain provisioning to the zones your node groups actually span. The topology key is the EBS CSI driver's own zone label, topology.ebs.csi.aws.com/zone.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3-topology
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer   # bind AFTER the pod is scheduled
reclaimPolicy: Delete
allowVolumeExpansion: true
parameters:
  type: gp3
  encrypted: "true"
  iops: "3000"
  throughput: "125"
allowedTopologies:
- matchLabelExpressions:
  - key: topology.ebs.csi.aws.com/zone
    values:
    - us-east-1a
    - us-east-1b
    - us-east-1c

Step two: fence the dead node to force the detach in seconds

The six-minute wait exists because the control plane won't assume a silent node is dead — a network partition looks identical to a blackout, and force-detaching from a node that's actually alive and still writing is how you corrupt a filesystem. So the safe way to recover faster is not to shorten the timeout blindly; it's to give Kubernetes a human (or an automated, well-guarded controller) asserting: I have confirmed this node is gone. Detach now.

That assertion is the node.kubernetes.io/out-of-service taint, delivered by the Non-Graceful Node Shutdown feature that went GA in Kubernetes 1.28 (feature gate NodeOutOfServiceVolumeDetach, now locked on). Apply it and Kubernetes stops waiting: pods without a matching toleration are force-deleted, and the volumes attached to that node are force-detached immediately, freeing them to reattach in the healthy AZ.

The command is short, but the discipline around it is the whole point. Never apply this taint until you have confirmed the instance is actually powered off — a node that's merely network-partitioned may still be running your process and writing to the disk, and forcing a detach there invites exactly the two-writer corruption the six-minute wait was protecting you from.

# 1. CONFIRM the instance is truly gone (stopped/terminated, not just unreachable)
aws ec2 describe-instances --instance-ids i-0abc123def456 \
  --query 'Reservations[].Instances[].State.Name' --output text
#   -> must read: stopped   (or terminated)

# 2. Fence it: force-delete pods + force-detach EBS volumes immediately
kubectl taint nodes ip-10-0-1-23.ec2.internal \
  node.kubernetes.io/out-of-service=nodeshutdown:NoExecute

# 3. AFTER the AZ recovers and the node rejoins, REMOVE the taint (note trailing '-')
kubectl taint nodes ip-10-0-1-23.ec2.internal \
  node.kubernetes.io/out-of-service=nodeshutdown:NoExecute-

Doing this by hand at 3 a.m. is not a plan. You want the confirm-and-taint loop automated, guarded by the same safety check. The AWS Node Termination Handler in Queue Processor mode watches an SQS queue fed by EventBridge — ASG lifecycle hooks, EC2 status-change events, Spot interruption and rebalance notices — cordons and drains the node, and can then stamp the out-of-service taint so RWO volumes detach without the wait. It's the automation layer for interruptions AWS actually tells you about.

# aws-node-termination-handler — Queue Processor mode
enableSqsTerminationDraining: true
queueURL: "https://sqs.us-east-1.amazonaws.com/111122223333/nth-queue"

# Cordon AND drain (don't stop at cordon)
cordonOnly: false

# After drain, stamp the out-of-service taint so RWO/EBS volumes
# detach immediately instead of waiting out the ~6-minute window
enableOutOfServiceTaint: true

enableSqsTerminationDraining  # queue-driven mode; reacts to ASG /
                              # EC2 status / Spot events via SQS+EventBridge
queueURL                      # the SQS queue NTH polls
cordonOnly: false             # cordon THEN drain (evict the pods)
enableOutOfServiceTaint: true # add node.kubernetes.io/out-of-service
                              # after drain -> force-detach volumes now

Step three: stop depending on one volume at all

Fencing makes recovery fast, but it's still recovery from a dependency on a single disk in a single zone. The architecturally honest answer to "survive an AZ blackout without data loss" is to not have your data live in one place to begin with. Run the workload as a StatefulSet with a replica in every AZ, each owning its own EBS volume in its own zone, and let the application replicate data between them. Lose a zone and you lose one replica — the others hold quorum, keep serving, and no single stuck attachment can wedge the service.

Two scheduling constraints make this real. topologySpreadConstraints with topologyKey: topology.kubernetes.io/zone and maxSkew: 1 forces the replicas to spread evenly across zones rather than clumping. A podAntiAffinity on the hostname keeps two replicas off the same node. Combined with the topology-aware StorageClass from earlier, each pod's volumeClaimTemplate provisions a fresh volume in the zone where that replica lands.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cache
spec:
  serviceName: cache
  replicas: 3
  selector:
    matchLabels: { app: cache }
  template:
    metadata:
      labels: { app: cache }
    spec:
      # 1) Spread one replica into each AZ
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels: { app: cache }
      # 2) Never place two replicas on the same node
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels: { app: cache }
            topologyKey: kubernetes.io/hostname
      containers:
      - name: cache
        image: redis:7
        volumeMounts:
        - { name: data, mountPath: /data }
  # 3) Each replica gets its OWN volume, in its OWN zone
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: ebs-gp3-topology
      resources:
        requests:
          storage: 20Gi

The critical mindset shift: EBS gives you durability (your bytes survive a disk failure within the zone) but not availability across zones. A three-replica StatefulSet spread across three AZs, replicating at the data layer — Redis replication, Postgres streaming replicas, a Raft-based store — turns a zone blackout into the loss of one-third of your capacity instead of a full-stop outage waiting on a six-minute detach. The single-volume patterns above are how you make the surviving replicas fail over cleanly; replication is how you make sure there's always a surviving replica.

Pitfalls that turn a graceful failover into an outage

• Tainting a partitioned node instead of a dead one. The out-of-service taint skips every safety wait. If the "dead" node is only network-isolated and still writing, you've just authorized a second writer. Always verify the EC2 instance state (stopped/terminated) before you taint — make that check the first line of the runbook, not an afterthought.
• Forgetting to remove the taint after recovery. The taint is a break-glass state, not a permanent setting. Leave node.kubernetes.io/out-of-service on a node that has rejoined and it will keep force-evicting pods. Removing it (the trailing - on the taint command) is a mandatory recovery step.
• Leaving StorageClass on Immediate binding. Without WaitForFirstConsumer, the CSI driver picks a zone before the scheduler does, and you'll periodically wedge pods with volume node affinity conflict — an availability bug you created, unrelated to any AZ actually failing.
• Reaching for EBS Multi-Attach as an HA fix. It's io1/io2-only, single-AZ, and demands a cluster-aware filesystem. It solves shared-block access inside one zone; it does nothing for a zone going dark, and running EXT4/XFS on it corrupts data.
• Assuming EKS managed node groups need nothing. Managed node groups drain on normal lifecycle events, but a sudden AZ blackout produces silent, unreachable nodes — no clean signal. That's the exact scenario where you still hit the six-minute wait and need the fencing path.
• Confusing durability with availability. EBS replicates within an AZ, so your data survives a disk failure. It does not survive the AZ. If your SLO can't absorb a multi-minute stall, one volume in one zone is never enough — replicate.