~/ nmadiraju.com Amazon EKS · Node NotReady or won't join
arrow_back All EKS troubleshooting playbooks
Troubleshooting Playbook · Amazon EKS nodes

Node NotReady or Won't Join on Amazon EKS

The node shows NotReady, or the new instance never appears in kubectl get nodes, and Pods wait with 0/3 nodes are available: 3 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. Find the cause from the condition or the log, confirm it, and fix it.

1Start here

What NotReady means

Every node runs a kubelet that registers the node with the API server, then keeps reporting its health. NotReady means the kubelet reported a problem (Ready: False) or stopped reporting at all (Ready: Unknown). A node that won't join never got that far: the EC2 instance runs, but no Node object appears.

Either way, the node is tainted so new Pods stop landing there, and Pods on a node that stays unreachable are evicted after five minutes by default. The node's Ready condition tells you which case you're in.

Run this first

kubectl get nodes
kubectl describe node ip-10-0-2-41.ec2.internal
NAME                        STATUS     ROLES    AGE   VERSION
ip-10-0-1-23.ec2.internal   Ready      <none>   12d   v1.33.5-eks-113cf36
ip-10-0-2-41.ec2.internal   NotReady   <none>   12d   v1.33.5-eks-113cf36

# describe node, Conditions (time columns trimmed)
  Type            Status  Reason                      Message
  MemoryPressure  False   KubeletHasSufficientMemory  kubelet has sufficient memory available
  DiskPressure    False   KubeletHasNoDiskPressure    kubelet has no disk pressure
  PIDPressure     False   KubeletHasSufficientPID     kubelet has sufficient PID available
  Ready           False   KubeletNotReady             container runtime network not ready: NetworkReady=false ... cni plugin not initialized
  • not in the listThe instance is running but never registered: IAM, the network path to the API, or the launch itself (causes 1 to 3). Read the kubelet log on the instance, or the node group's health issues.
  • Ready: FalseThe kubelet is running and says what's wrong in the message: the network plugin or the container runtime (causes 4 and 5).
  • Ready: UnknownKubelet stopped posting node status. The control plane hasn't heard from the kubelet: the node is starved, or the instance is down (causes 6 and 7).
  • *Pressure: TrueMemory, disk or PIDs are short. On its own this taints the node and evicts Pods rather than making it NotReady; see cause 6.

2Decision tree

Start from what kubectl get nodes and the node's Ready condition say, find the phrase from your logs or conditions on the right, then click it (or Tab to it and press Enter) to jump to that cause. The numbers match the cards below.

Decision tree: from kubectl get nodes to the cause of a missing or NotReady node Run kubectl get nodes, then kubectl describe node and read the Ready condition. A running instance that never appears in the list never registered: an IAM mapping problem (Unauthorized), a network path problem (timeout), or a launch or bootstrap problem. Ready False means the kubelet is running and names the problem: the VPC CNI isn't running, or containerd is down. Ready Unknown means the kubelet stopped posting status: the node ran out of memory, disk or PIDs, or the EC2 instance is stopped or impaired. Each cause links to its card below. RUN THIS WHAT DO YOU SEE? FIND YOUR MESSAGE, OPEN THE CAUSE Missing or NotReady kubectl get nodes then describe node: Ready 1 Unauthorized · NodeCreationFailure The node's IAM role isn't allowed into the cluster card ↓ 2 i/o timeout · TLS handshake timeout · ClusterUnreachable The node can't reach the cluster's API endpoint card ↓ 3 AsgInstanceLaunchFailures · InsufficientFreeAddresses · x509 No instance launched, or its bootstrap config is wrong card ↓ Not in the list instance up, or none it never registered 4 NetworkReady=false · cni plugin not initialized The VPC CNI (aws-node) isn't running on the node card ↓ 5 PLEG is not healthy · container runtime is down containerd is down or stuck card ↓ Ready: False KubeletNotReady the kubelet says why 6 Kubelet stopped posting · Memory/Disk/PIDPressure The node ran out of memory, disk or PIDs card ↓ 7 status check impaired · instance stopped The EC2 instance is stopped or failing its status checks card ↓ Ready: Unknown NodeStatusUnknown the kubelet went quiet
Where you lookNever joined, or the kubelet reports a faultThe kubelet stopped reporting

3Causes and fixes

One card per cause. Each shows what the condition or log looks like, the commands that confirm it, the fix, how to keep it from coming back, and the walkthrough that explains the mechanism.

The node's IAM role isn't allowed into the cluster

Unauthorized NodeCreationFailure

Back to the tree

What you see

$ kubectl get nodes
NAME                        STATUS   ROLES    AGE   VERSION
ip-10-0-1-23.ec2.internal   Ready    <none>   12d   v1.33.5-eks-113cf36
# the new instance is running in EC2, but never shows up here
# on the instance: journalctl -u kubelet
kubelet[2041]: E1102 09:14:07 "Unable to register node with API server" err="Unauthorized" node="ip-10-0-2-41.ec2.internal"
kubelet[2041]: E1102 09:14:09 "Failed to watch" err="failed to list *v1.Service: Unauthorized"
$ aws eks describe-nodegroup --cluster-name prod --nodegroup-name general --query 'nodegroup.health.issues'
[ { "code": "NodeCreationFailure", "message": "Instances failed to join the kubernetes cluster", "resourceIds": ["i-0b2c3d4e5f6a78901"] } ]

Unauthorized means the request reached the API server and the node's identity was rejected. The network path is fine; the cluster just doesn't know this IAM role. A managed node group reports NodeCreationFailure when its instances haven't joined within 15 minutes, for this or a network cause (card 2).

Confirm

# Which IAM role does the instance actually use?
aws ec2 describe-instances --instance-ids i-0b2c3d4e5f6a78901 --query 'Reservations[].Instances[].IamInstanceProfile.Arn'
aws iam get-instance-profile --instance-profile-name <profile-name> --query 'InstanceProfile.Roles[].Arn'
# How does the cluster map IAM to Kubernetes? API, API_AND_CONFIG_MAP or CONFIG_MAP
aws eks describe-cluster --name prod --query 'cluster.accessConfig.authenticationMode'
# Is there an access entry for that role, of type EC2_LINUX?
aws eks describe-access-entry --cluster-name prod --principal-arn arn:aws:iam::111122223333:role/eks-node-role --query 'accessEntry.type'
# Or, with CONFIG_MAP: is the role mapped in aws-auth?
kubectl get configmap aws-auth -n kube-system -o yaml

The principal must be the role ARN (:role/eks-node-role), not the instance profile ARN. In aws-auth, a role with a path (:role/eks/nodes/eks-node-role) must be written without the path.

Fix

  • Access entries (API or API_AND_CONFIG_MAP): create one for the node role: aws eks create-access-entry --cluster-name prod --principal-arn arn:aws:iam::111122223333:role/eks-node-role --type EC2_LINUX. An EC2_LINUX entry puts the node in system:nodes for you; don't attach an access policy to it. Use type EC2 instead for Auto Mode custom NodeClasses and for self-managed AL2023 nodes that are named by instance ID.
  • aws-auth only (CONFIG_MAP): map the role with eksctl create iamidentitymapping --cluster prod --arn arn:aws:iam::111122223333:role/eks-node-role --group system:bootstrappers,system:nodes --username 'system:node:{{EC2PrivateDNSName}}', or restore the entry someone removed.
  • Managed node groups create their own access entry (or aws-auth entry) when the group is created. If it was deleted, for example while migrating from aws-auth to access entries, recreate it as above; the role stays the same.
  • Mapping is right, still Unauthorized: check the node's clock. Its token is a signed AWS STS request that is valid only for a short window, so a clock that is far off is rejected. EKS AMIs sync time with chrony and the Amazon Time Sync Service, so this is rare.

Prevent

  • Move to access entries and manage them in the same IaC as the node groups, so a node role and its entry are created and deleted together.
  • Don't hand-edit aws-auth; a YAML mistake there can lock out every node at once.

Understand it

Access entriesManaged node groups

Let AWS check it for you: the AWSSupport-TroubleshootEKSWorkerNode Systems Manager runbook takes the cluster name and the instance ID, and checks the common reasons a node can't join: the node role's policies, security groups, network ACLs, routes and NAT gateways, VPC endpoints, DHCP options and the AMI. Its on-node checks run through Systems Manager, so the instance must be running with the SSM agent. It doesn't check access entries or aws-auth; do that with the commands above.

The node can't reach the cluster's API endpoint

i/o timeout TLS handshake timeout ClusterUnreachable

Back to the tree

What you see

# on the instance: journalctl -u kubelet
kubelet[2041]: E1102 09:14:37 "Unable to register node with API server" err="Post \"https://9F3A1C2B7D5E4F60A8B9C0D1E2F3A4B5.gr7.us-east-1.eks.amazonaws.com/api/v1/nodes\": dial tcp 10.0.12.34:443: i/o timeout" node="ip-10-0-2-41.ec2.internal"
# public endpoint only, and the NAT gateway's IP isn't in publicAccessCidrs
err="Post \"https://9F3A...eks.amazonaws.com/api/v1/nodes\": dial tcp 54.210.17.92:443: i/o timeout"
# a node that can't reach the public endpoint at all can also log
err="... net/http: TLS handshake timeout"

A timeout means the packets never got an answer: no route, a security group or network ACL in the way, or a public endpoint that only allows other source IPs. Compare card 1, where the API server answers and says Unauthorized.

Confirm

# How is the endpoint exposed, and to which source CIDRs?
aws eks describe-cluster --name prod --query 'cluster.resourcesVpcConfig.{private:endpointPrivateAccess,public:endpointPublicAccess,cidrs:publicAccessCidrs,clusterSG:clusterSecurityGroupId}'
# From the node (SSM Session Manager): any HTTP status, even 401 or 403, proves the path works
curl -sk -o /dev/null -w '%{http_code}\n' --max-time 5 https://9F3A1C2B7D5E4F60A8B9C0D1E2F3A4B5.gr7.us-east-1.eks.amazonaws.com/livez
# Which security groups does the node have, and does its subnet route to a NAT gateway?
aws ec2 describe-instances --instance-ids i-0b2c3d4e5f6a78901 --query 'Reservations[].Instances[].{sg:SecurityGroups[].GroupId,subnet:SubnetId}'

For a node that can't open a session, the VPC Reachability Analyzer can trace TCP 443 from the instance to one of the API server's private IPs (kubectl get endpointslices -l kubernetes.io/service-name=kubernetes lists them) and names the hop that blocks it.

Fix

  • Public endpoint with an allow list: nodes in private subnets reach it from the NAT gateway's Elastic IP. Add that IP to publicAccessCidrs, or better, turn on endpointPrivateAccess so nodes use the private endpoint inside the VPC.
  • Private endpoint enabled: nodes need DNS to resolve it to private IPs (with public access also on, they would otherwise get the public IPs), so the VPC needs enableDnsHostnames and enableDnsSupport, and a custom DHCP option set must still point at the Amazon-provided DNS (or forward to it).
  • Security groups: the cluster security group lets the control plane and nodes that carry it talk to each other. If you gave nodes your own security group instead, it must allow outbound TCP 443 to the cluster security group, and the cluster security group must allow inbound 443 from it.
  • No internet route and no private endpoint: a node in a private subnet with no NAT gateway can only reach the API through the private endpoint, and needs VPC endpoints for the AWS APIs it calls (card 4 covers what that breaks after joining).

Prevent

  • Enable private endpoint access on every cluster, even when you keep the public endpoint for people.
  • Keep the cluster security group attached to nodes, and add your own groups next to it rather than in place of it.

Understand it

Private clusters and endpoint accessControl plane vs data plane

Private clusters: with no outbound internet access, nodes also need VPC endpoints for ec2, ecr.api, ecr.dkr and an s3 gateway endpoint, plus sts for IRSA or eks-auth for Pod Identity. The endpoints' security group must allow 443 from the node subnets.

No instance launched, or its bootstrap config is wrong

AsgInstanceLaunchFailures InsufficientFreeAddresses x509: certificate signed by unknown authority

Back to the tree

What you see

$ aws eks describe-nodegroup --cluster-name prod --nodegroup-name general --query 'nodegroup.health.issues'
[ { "code": "AsgInstanceLaunchFailures", "message": "...", "resourceIds": ["eks-general-58c9b0e4-..."] },
  { "code": "InsufficientFreeAddresses", "message": "...", "resourceIds": ["subnet-0a1b2c3d4e5f60789"] } ]
# self-managed node whose NodeConfig has another cluster's CA: journalctl -u kubelet
kubelet[2041]: E1102 09:20:11 "Unable to register node with API server" err="Post \"https://9F3A...eks.amazonaws.com/api/v1/nodes\": tls: failed to verify certificate: x509: certificate signed by unknown authority"

Two different failures with the same result. Either no EC2 instance starts (capacity, quota, subnet IPs, a missing launch template, AMI or instance profile), or one starts with the wrong cluster name, endpoint or CA in its user data, so the kubelet never talks to the right cluster.

Confirm

# Does the Auto Scaling group even launch instances, and why not?
aws autoscaling describe-scaling-activities --auto-scaling-group-name eks-general-58c9b0e4-... --max-items 5 --query 'Activities[].StatusMessage'
# Free IPs in the node group's subnets
aws ec2 describe-subnets --subnet-ids subnet-0a1b2c3d4e5f60789 --query 'Subnets[].AvailableIpAddressCount'
# Self-managed: what did the instance boot with? (decode the user data, check name, endpoint, CA)
aws ec2 describe-instance-attribute --instance-id i-0b2c3d4e5f6a78901 --attribute userData --query 'UserData.Value' --output text | base64 -d

On the node, sudo bash /etc/eks/log-collector-script/eks-log-collector.sh bundles the kubelet, containerd, nodeadm, cloud-init and CNI logs into one archive under /var/log/.

Fix

  • InsufficientFreeAddresses: the subnets are out of IPs (nodes and Pods share them with the VPC CNI). Add subnets or a secondary CIDR, or free addresses; see the VPC CNI walkthrough.
  • AsgInstanceLaunchFailures or InstanceLimitExceeded: read the scaling activity message: capacity in that Availability Zone, an EC2 quota, or a launch template that points at a deleted AMI or security group. Allow more instance types or zones, or raise the quota.
  • Wrong bootstrap (self-managed nodes): on Amazon Linux 2023, the NodeConfig in user data must have the exact cluster name, apiServerEndpoint, certificateAuthority and cidr. Copy them from aws eks describe-cluster and launch new instances; fixing a running node by hand doesn't last.
  • Node has no private DNS name: the kubelet logs node "" not found. The VPC's DHCP option set needs a domain-name and Amazon-provided DNS.

Prevent

  • Prefer managed node groups, Karpenter or Auto Mode, which write the bootstrap config for you.
  • Size node subnets for nodes plus Pods, and alarm on AvailableIpAddressCount.

Understand it

Managed node groupsControl plane vs data plane

The VPC CNI (aws-node) isn't running on the node

NetworkReady=false cni plugin not initialized

Back to the tree

What you see

$ kubectl describe node ip-10-0-2-41.ec2.internal
Conditions:
  Type    Status  Reason           Message
  Ready   False   KubeletNotReady  container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
$ kubectl get pods -n kube-system -l k8s-app=aws-node -o wide --field-selector spec.nodeName=ip-10-0-2-41.ec2.internal
NAME             READY   STATUS             RESTARTS   NODE
aws-node-7xk2p   1/2     CrashLoopBackOff   6          ip-10-0-2-41.ec2.internal

The node joined, so IAM and the API path work. But the kubelet won't report Ready until a CNI plugin has written its config, and on EKS that's the aws-node DaemonSet. Until then the node keeps the node.kubernetes.io/not-ready taint.

Confirm

# Why is aws-node failing? Read the aws-node container, and the init container if it's stuck in Init
kubectl logs -n kube-system aws-node-7xk2p -c aws-node --tail=50
kubectl describe pod -n kube-system aws-node-7xk2p
# On the node, IPAMD's own log
sudo tail -n 50 /var/log/aws-routed-eni/ipamd.log
# Is kube-proxy running there? IPAMD reaches the API server through the kubernetes Service IP, which kube-proxy programs
kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide --field-selector spec.nodeName=ip-10-0-2-41.ec2.internal

Fix

  • UnauthorizedOperation in the logs: IPAMD can't call EC2 (AssignPrivateIpAddresses, DescribeNetworkInterfaces, …). Give aws-node the AmazonEKS_CNI_Policy (IPv6 clusters need a custom policy instead): on the node role, or, as AWS recommends, on its own role through IRSA or a Pod Identity association for the aws-node service account in kube-system. Check that role's trust policy if you moved it.
  • Failed to check API server connectivity: IPAMD can't reach the API through the kubernetes Service IP, often because kube-proxy isn't running on the node. Reinstall or update the kube-proxy add-on. Since VPC CNI 1.19.4, IPAMD only stops on this when Security Groups for Pods or custom networking is on; otherwise it logs Proceeding without API server connectivity and starts anyway.
  • Init:ImagePullBackOff: the node can't pull the CNI images from ECR. In private subnets, add the ecr.api and ecr.dkr interface endpoints and the s3 gateway endpoint (and ec2 for IPAMD itself).
  • No aws-node Pod at all: the VPC CNI add-on was deleted, or the DaemonSet's node affinity excludes this node. Reinstall it with aws eks create-addon --cluster-name prod --addon-name vpc-cni.

Prevent

  • Run the VPC CNI, kube-proxy and CoreDNS as EKS add-ons, and keep them on versions supported by your cluster version.
  • Give aws-node its own IAM role (IRSA or Pod Identity) in the same IaC as the add-on, so the permission can't be removed with an unrelated node role change.

Understand it

The VPC CNIPod Identity and IRSA

containerd is down or stuck

PLEG is not healthy container runtime is down

Back to the tree

What you see

$ kubectl describe node ip-10-0-2-41.ec2.internal
Conditions:
  Type    Status  Reason           Message
  Ready   False   KubeletNotReady  PLEG is not healthy: pleg was last seen active 3m12s ago; threshold is 3m0s
  Ready   False   KubeletNotReady  container runtime is down

The kubelet is alive and posting status, but it can't get answers from containerd. PLEG (the Pod lifecycle event generator) relists containers from the runtime; when a relist hasn't finished for 3 minutes, the kubelet reports the node not ready. Common triggers: containerd crashed or hung, a full disk under /var/lib/containerd, or a node so overloaded that the runtime can't keep up.

Confirm

# Open a shell on the node (SSM Session Manager), then:
aws ssm start-session --target i-0b2c3d4e5f6a78901
sudo systemctl status containerd kubelet
sudo journalctl -u containerd --since "30 min ago" | tail -n 50
# Does the runtime answer at all?
sudo crictl ps | head
df -h /var/lib/containerd

Fix

  • containerd crashed or hung: sudo systemctl restart containerd often brings the node back, but treat it as a stopgap and find out why from its logs.
  • Disk full: free space (unused images: sudo crictl rmi --prune) and give the node group a bigger root volume.
  • Overloaded node or a fault you can't explain: cordon, drain and replace the node (kubectl drain, then terminate the instance; the node group or Karpenter launches a new one).

Prevent

  • Install the EKS node monitoring agent: it reports runtime problems as the ContainerRuntimeReady condition, and node auto repair can replace the node.
  • Keep requests honest so the scheduler doesn't pack more onto a node than it and its runtime can handle.

Understand it

Control plane vs data plane

The node ran out of memory, disk or PIDs

Kubelet stopped posting node status. MemoryPressure DiskPressure PIDPressure

Back to the tree

What you see

$ kubectl describe node ip-10-0-2-41.ec2.internal
Conditions:
  Type            Status   Reason             Message
  MemoryPressure  Unknown  NodeStatusUnknown  Kubelet stopped posting node status.
  Ready           Unknown  NodeStatusUnknown  Kubelet stopped posting node status.
Events:
  Warning  EvictionThresholdMet  kubelet      Attempting to reclaim memory
  Warning  SystemOOM             kubelet      System OOM encountered, victim process: java, pid: 18342
  Normal   NodeNotReady          node-controller  Node ip-10-0-2-41.ec2.internal status is now: NodeNotReady

A pressure condition on its own doesn't make a node NotReady: the kubelet reports it and evicts Pods to recover, and the control plane taints the node (node.kubernetes.io/memory-pressure:NoSchedule and so on). The node goes NotReady when the shortage gets so bad that the kubelet or containerd is killed or starved and stops renewing its heartbeat; then the control plane marks every condition Unknown.

Confirm

# The last thing the node said before it went quiet: events about memory, disk or PIDs
kubectl get events -A --field-selector involvedObject.kind=Node,involvedObject.name=ip-10-0-2-41.ec2.internal --sort-by=.lastTimestamp
# What ran there, and with which requests and limits
kubectl describe node ip-10-0-2-41.ec2.internal | sed -n '/Non-terminated Pods/,/Events/p'
# The instance's own console: OOM killer lines, a full disk
aws ec2 get-console-output --instance-id i-0b2c3d4e5f6a78901 --latest --output text | tail -n 40

If the node comes back, the conditions show which resource ran short: MemoryPressure, DiskPressure or PIDPressure True, with reasons such as KubeletHasInsufficientMemory.

Fix

  • Memory: set memory limits on the Pods that grew (a limit makes the kernel kill one container, not starve the node), and requests that match real use so the scheduler doesn't overcommit the node.
  • Disk: give the node group a bigger root volume (managed node groups without a launch template default to 20 GiB on Linux) and keep container logs and images in check; the kubelet's image garbage collection only frees unused images.
  • PIDs: a fork bomb or a thread leak in one Pod can use every PID. Set podPidsLimit in the kubelet config.
  • Get the node back: once the heavy Pods are gone it often recovers on its own; if it doesn't within a few minutes, replace it.

Prevent

  • Reserve room for the system: EKS AMIs set kubeReserved from the instance size; add systemReserved and keep the kubelet's eviction thresholds so it evicts before the kernel OOM killer acts.
  • Alarm on node memory and disk use (Container Insights or Prometheus) well before the eviction thresholds.

Understand it

Requests, limits, QoS and evictions

The EC2 instance is stopped or failing its status checks

NodeStatusUnknown node.kubernetes.io/unreachable impaired

Back to the tree

What you see

$ kubectl get node ip-10-0-2-41.ec2.internal -o jsonpath='{.spec.taints}'
[{"effect":"NoSchedule","key":"node.kubernetes.io/unreachable"},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable"}]

$ aws ec2 describe-instance-status --instance-ids i-0b2c3d4e5f6a78901 --include-all-instances \
    --query 'InstanceStatuses[].{state:InstanceState.Name,system:SystemStatus.Status,instance:InstanceStatus.Status}'
[ { "state": "running", "system": "ok", "instance": "impaired" } ]

The kubelet went quiet because the machine under it did: stopped, hung, or failing EC2 status checks. The control plane taints the node unreachable; Pods on it are evicted after their toleration for that taint runs out, 300 seconds by default. Those Pods then show Terminating until the node comes back or its Node object is deleted. A Deployment starts replacements elsewhere; a StatefulSet waits.

Confirm

# Which instance is the node? providerID ends with the instance ID
kubectl get node ip-10-0-2-41.ec2.internal -o jsonpath='{.spec.providerID}{"\n"}'
# Instance state and status checks
aws ec2 describe-instance-status --instance-ids i-0b2c3d4e5f6a78901 --include-all-instances
# Was it a Spot interruption, or did someone stop it? (scheduled maintenance shows under Events in describe-instance-status)
aws ec2 describe-instances --instance-ids i-0b2c3d4e5f6a78901 --query 'Reservations[].Instances[].{state:State.Name,reason:StateTransitionReason,lifecycle:InstanceLifecycle}'

When an instance is terminated, the cloud controller removes its Node object, so a terminated node disappears rather than staying NotReady. A stopped instance keeps its Node object, tainted node.cloudprovider.kubernetes.io/shutdown.

Fix

  • Impaired or hung: reboot it from EC2, or terminate it and let the node group, Karpenter or Auto Mode launch a replacement. A failed system status check is an AWS host problem; for an EBS-backed instance, stop and start (not reboot) usually moves it to new hardware.
  • Stopped by someone: start it, or delete it. Don't stop nodes to save money; scale the node group down instead, so Pods are drained first.
  • Stateful Pods stuck Terminating on a node that's gone for good: if you're sure it's down, add the node.kubernetes.io/out-of-service=nodeshutdown:NoExecute taint (non-graceful node shutdown) so its volumes detach and the Pods move.

Prevent

  • Turn on node auto repair for managed node groups (aws eks update-nodegroup-config --cluster-name prod --nodegroup-name general --node-repair-config enabled=true). It replaces a node whose Ready condition stays bad for 30 minutes, and by default pauses when a group of more than five nodes has more than 20% unhealthy (you can change the threshold). Auto Mode does this by default.
  • Spread replicas across nodes and zones, and use PodDisruptionBudgets, so one lost node never takes a whole service down.

Understand it

Managed node groupsFargate and EKS Auto Mode

Auto Mode nodes: you can't SSH or SSM into them. Read their logs with a NodeDiagnostic resource (or the kubectl ekslogs plugin), kubectl debug node/<instance-id> -it --profile=sysadmin --image=public.ecr.aws/amazonlinux/amazonlinux:2023, or aws ec2 get-console-output; and check kubectl get nodeclaims for nodes that never joined. The node monitoring agent and auto repair are on by default there.

4Diagnose it

Six nodes that are missing or NotReady. Read the output and pick the cause. A wrong pick tells you why it doesn't fit; there's no score.

Diagnose it