~/ nmadiraju.com Amazon EKS · Spark and EMR on EKS
arrow_back Back to nmadiraju
Visual Walkthrough · Data on Amazon EKS

How Spark and EMR on EKS Work on Amazon EKS

Follow the nightly sales-etl Spark job onto a shared EKS cluster, then check yourself with 8 questions.

1Visual Walkthrough

A Spark job on EKS: driver and executor Pods, EMR on EKS, Karpenter Spot nodes, S3 and logs The EKS cluster holds the Kubernetes API, a node group or Karpenter, an On-Demand node with EMR's job-runner and the Spark driver, four executor node slots, and a tray for Pending executors or Apache Celeborn. AWS services hold EMR on EKS (virtual cluster sales-vc), the IAM job execution role, the S3 bucket and the logs. The walkthrough covers RBAC for the driver, EMR on EKS job submission and roles, Pending executors and Karpenter, Spot and Graviton, dynamic allocation, and shuffle data lost and kept through Spot reclaims. Data team nightly sales-etl EKS CLUSTER prod-east NAMESPACE analytics Kubernetes API Managed node group analytics-ng 4 × m6i · On-Demand · fixed size nothing adds nodes when Pods are Pending Karpenter · 2 NodePools job-runner Pod runs spark-submit for EMR driver Pod · sales-etl Apache Celeborn · remote shuffle service 3 workers on On-Demand nodes executors push shuffle data here AWS SERVICES Amazon EMR on EKS virtual cluster sales-vc → namespace analytics IAM role sales-etl-exec job execution role S3 bucket acme-sales raw/2026-11-01/ in curated/sales/ out Logs and Spark UI s3://acme-spark-logs/ CloudWatch /emr/sales-etl persistent app UI

    2Quiz to check understanding

    Check yourself