~/ nmadiraju.com Amazon EKS · GPU nodes and model serving
arrow_back Back to nmadiraju
Visual Walkthrough · AI/ML on EKS

How GPU Nodes and Model Serving Work on Amazon EKS

Follow llm, vLLM serving Llama 3.1 8B, from Pending to scaled to zero, then check yourself with 8 questions.

1Visual Walkthrough

Two GPU nodes serving Llama 3.1 8B with vLLM, with the device plugin, model sources, KEDA and Karpenter A VPC with two g5.2xlarge GPU nodes, each running the NVIDIA device plugin and an llm Pod (vLLM) with a GPU memory bar, and node-1 running chat-api, KEDA and Karpenter. Left: Hugging Face, Docker Hub, an S3 bucket and ECR. Right: the EKS control plane, NodePool gpu, ScaledObject llm and a CPU HPA. The walkthrough shows a Pod Pending without the device plugin, a KV cache too small for the full context, continuous batching, a slow cold start and its fix, scaling on queue depth, and scaling to zero at night. Hugging Face model hub · internet Docker Hub vllm/vllm-openai S3 bucket weights · same Region Amazon ECR vllm-openai mirror YOUR VPC · us-east-1 gpu-1 · g5.2xlarge · 1× A10G AL2023 NVIDIA AMI TERMINATED nvidia-device-plugin GPU memory · 24 GB weights 16 GB KV cache too small Llama-3.1-8B-Instruct limits nvidia.com/gpu: 1 · :8000 gpu-2 · g5.2xlarge · 1× A10G AL2023 NVIDIA AMI TERMINATED nvidia-device-plugin GPU memory · 24 GB weights 16 GB · KV cache ~4 GB weights 16 GB KV cache llm-5f2kq · vLLM Llama-3.1-8B-Instruct limits nvidia.com/gpu: 1 · :8000 node-1 · m6i.large chat-api Pod (caller) POST llm:8000/v1/chat/completions KEDA Karpenter EKS CONTROL PLANE API server and scheduler NodePool gpu karpenter.k8s.aws/instance-family: g5 taint nvidia.com/gpu:NoSchedule limits: nvidia.com/gpu: 4 consolidationPolicy: WhenEmpty ScaledObject llm HPA llm target: CPU 70% of request REMOVED

    2Quiz to check understanding

    Check yourself