Build a hybrid LLM router on AWS for under $1 an hour: vLLM on a spot GPU for routine requests, Claude for the rest

A step-by-step build of a gateway that sends routine LLM requests to a self-hosted model on a spot GPU and everything else to Claude on Amazon Bedrock. Eight phases, each with the commands, the check that proves it worked, and how to undo it. Every number in it was measured on a real build, and every bug it warns about was hit on one.

Most self-hosted LLM tutorials stop at vllm serve on a GPU machine that someone forgets to turn off. Most of the time goes into the parts around it: a gateway that decides which requests the small model gets, a fallback for when the GPU is busy or gone, a GPU node that Karpenter starts on demand and you pause or delete between sessions, and a teardown that leaves nothing billing. This workshop builds all of it in an empty AWS account, in eight phases. We built it once, then deleted everything and rebuilt it twice from the written steps alone. Rebuild 1 hit eight problems the original build never did, plus a typo in our own instructions. Every problem is in the pitfalls table, marked by the run that found it.

1What you will build, and what it costs

You are building a gateway that sends each LLM request to the cheapest model that can handle it. Routine requests, such as extraction and short rewrites, go to a 7-billion-parameter model that you host yourself. Hard requests go to Claude on Amazon Bedrock, a managed model service. Bedrock is also the fallback whenever the self-hosted model is busy, slow or missing.

The small model is the easy part. The work is everything that keeps it cheap, reachable and replaceable.

Every request starts the same way. It reaches an internal Application Load Balancer (ALB) and then the gateway: LiteLLM, an open-source proxy that speaks the OpenAI API, running as two containers on ECS Fargate. A routing plugin in the gateway then decides, in a few milliseconds, with three checks in order:

1. Hard rules. A reasoning tag, or more than about 2,000 tokens, means "complex".
2. A semantic cache of earlier routing decisions. The gateway turns the prompt into an embedding (a list of numbers that captures its meaning) and reuses the decision of the most similar earlier prompt. When nothing similar is cached, a background judge (one Claude call that does not block the request) labels the prompt for next time.
3. A circuit breaker, which reads how many requests are waiting inside vLLM. If the queue is long, or vLLM cannot be reached, it sends the request to Bedrock instead.

A "standard" request goes through an internal Network Load Balancer (NLB) to vLLM, an open-source inference server, on a GPU node in Amazon EKS. A "complex" request, or any request while vLLM is unavailable, goes to Bedrock.

Here is what it costs at us-east-1 list prices on 18 October 2026. Spot prices move during the day, and Karpenter picks whichever GPU type is cheapest at that moment.

Before you start, you need five things. Request the quota first: it is the only one you have to wait for.

• Tools: AWS CLI v2, Terraform 1.5 or later, kubectl, Helm, Docker Desktop, and envsubst (from the gettext package).
• An admin identity in the target account. An IAM Identity Center (SSO) role is the best choice.
• An S3 bucket for Terraform state, with versioning on. Terraform's native S3 locking (use_lockfile) means no DynamoDB table.
• Bedrock access to Claude Haiku 4.5. Anthropic models need a one-time use-case form per account. Send one message from the Bedrock playground to prove it works.
• GPU quota. New accounts have 0 vCPUs for G and VT instances, on-demand and spot. The quota counts vCPUs, not GPUs: one xlarge GPU instance is 4 vCPUs and one GPU.

git clone https://github.com/Nareshcbit/enterprise-ai-production-blueprints.git
cd enterprise-ai-production-blueprints/01-hybrid-model-router

cp terraform/envs/dev/backend.hcl.example      terraform/envs/dev/backend.hcl
cp terraform/envs/dev/terraform.tfvars.example terraform/envs/dev/terraform.tfvars
# both are gitignored: they hold your bucket, account ID and ARNs

make check

# All G and VT Spot Instance Requests, counted in vCPUs (4 = one xlarge GPU node)
aws service-quotas request-service-quota-increase --region us-east-1 \
  --service-code ec2 --quota-code L-3819A6DF --desired-value 4

AWS identity: arn:aws:iam::<ACCOUNT_ID>:role/<your-admin-role>
Region: us-east-1  Name: ai-router-dev  Gateway image: 0.2.2

2Phase 1 — Network

Goal: a VPC across two Availability Zones, with two public and two private subnets, one NAT gateway, and a security group for testing from CloudShell. Everything else in this workshop lives in the private subnets.

Both choices here are about cost. First, one NAT gateway is shared by both zones. That is enough for a lab, but it is not resilient to a zone failure; set single_nat_gateway = false for one per zone.

Second, only the free S3 gateway endpoint is created. Interface endpoints for Bedrock and ECR would keep that traffic on AWS's network, but each costs about $0.01 an hour per zone. Three of them across two zones come to about $43 a month, more than the NAT data charges they save in a lab. So the variable that controls them now defaults to off.

make network

terraform -chdir=terraform/envs/dev init -backend-config=backend.hcl
terraform -chdir=terraform/envs/dev apply -target=module.networking
# Apply complete! Resources: 22 added, 0 changed, 0 destroyed.

variable "enable_vpc_endpoints" {
  description = "Create interface endpoints for Bedrock Runtime and ECR (~$0.01/hr each per AZ). The S3 gateway endpoint is free and always created."
  type        = bool
  default     = false
}

  endpoints = merge(
    local.gateway_endpoints,                                   # S3: free
    var.enable_vpc_endpoints ? local.interface_endpoints : {}, # Bedrock, ECR
  )

You should see Apply complete! Resources: 22 added. Two private subnets carry two tags that later phases depend on. kubernetes.io/role/internal-elb=1 lets the load balancer controller place the vLLM NLB there. karpenter.sh/discovery=<name> lets Karpenter find the subnets for GPU nodes. To undo, run terraform destroy -target=module.networking before anything else is built.

3Phase 2 — The gateway on Fargate

Goal: the LiteLLM gateway runs as two Fargate tasks behind an internal ALB and answers chat requests from Claude on Bedrock. At the end of this phase the routing plugin is live, but there is no self-hosted model yet.

This phase takes three commands, because of the ordering problem. First create only the ECR repository. Then build and push the image. Then create the ALB, the ECS cluster and service, and the master key in Secrets Manager. Build for linux/amd64. The Fargate tasks here run on x86, and a laptop with an Apple chip builds ARM images unless you say otherwise. The image also bakes in a small embedding model for the semantic cache, so tasks never download one when they start.

The gateway's address for vLLM is set to http://vllm.invalid:8000/v1 in this phase. That name never resolves. The circuit breaker cannot read the vLLM queue, so it sends every request to Bedrock. That is on purpose: it is the safe state, and exactly how the system behaves later whenever vLLM is down.

make ecr       # 2 resources: the repository and its lifecycle policy
make image     # build the router image and push it
make gateway   # 24 resources: ALB, ECS cluster + service, master key

REPO=$(terraform -chdir=terraform/envs/dev output -raw gateway_ecr_repository_url)
aws ecr get-login-password --region us-east-1 \
  | docker login --username AWS --password-stdin ${REPO%%/*}
docker build --platform linux/amd64 -t $REPO:0.2.2 app   # Fargate here is x86_64
docker push $REPO:0.2.2

environment = [
  { name = "BEDROCK_MODEL", value = "bedrock/converse/${var.bedrock_inference_profile_id}" },
  { name = "VLLM_API_BASE", value = var.vllm_api_base },       # http://vllm.invalid:8000/v1 for now
  { name = "VLLM_METRICS_URL", value = var.vllm_metrics_url }, # the circuit breaker reads this
  { name = "ROUTER_CACHE_MODE", value = var.router_cache_mode }, # off | shadow | active
]
secrets = [
  { name = "LITELLM_MASTER_KEY", valueFrom = aws_secretsmanager_secret.master_key.arn },
]

# 1. on your laptop, in the project folder: get the two values
terraform -chdir=terraform/envs/dev output -raw gateway_alb_dns_name
terraform -chdir=terraform/envs/dev output -raw gateway_master_key_secret_arn

# 2. in a CloudShell VPC environment (private subnet + the cloudshell security group)
ALB=http://<gateway_alb_dns_name>
KEY=$(aws secretsmanager get-secret-value --region us-east-1 \
  --secret-id <gateway_master_key_secret_arn> --query SecretString --output text)
curl -s $ALB/v1/chat/completions -D - -o /tmp/body \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"model":"auto","max_tokens":60,"messages":[{"role":"user",
       "content":"Extract the invoice number from: INV-4471, due 2026-11-01."}]}' \
  | grep -i x-litellm-model-group
# x-litellm-model-group: bedrock-claude

You should see both ALB targets turn healthy. That took about 80 seconds in both rebuilds. The ALB is internal, so test it from a CloudShell VPC environment: CloudShell's "Create VPC environment" option, with a private subnet and the CloudShell security group from phase 1. The test tab shows how to get the ALB address and the master key onto it. The response header x-litellm-model-group: bedrock-claude shows which backend answered. To undo, delete the images from ECR, then destroy the gateway module. ECR refuses to delete a repository that still holds images.

4Phase 3 — The EKS cluster

Goal: an EKS 1.37 cluster with two small on-demand nodes for system workloads, the core add-ons, and access for the people who use it.

The API endpoint is both public and private. Public lets you run kubectl from a laptop; every call is still authenticated with IAM. Private lets the nodes reach the API inside the VPC. The system nodes are on-demand on purpose. Karpenter runs on them. If AWS took back a spot node running Karpenter, nothing would be left to replace nodes.

Access is declared in code, and the cluster grants nobody access by default. The module option that makes the cluster creator an admin is off, so that access does not depend on who happened to run terraform apply. Each entry in eks_access_entries maps an IAM principal to admin or view, and Terraform turns it into an EKS access entry with a cluster-wide access policy. Include the identity that runs Terraform. It installs the Helm charts in phase 4, and without an entry those installs fail with Unauthorized.

make eks          # 41 resources, ~15 minutes
make kubeconfig
kubectl get nodes

# terraform.tfvars — nobody gets cluster access unless listed here,
# including the identity that runs terraform (Helm installs use it)
eks_access_entries = {
  me = {
    principal_arn = "arn:aws:iam::<ACCOUNT_ID>:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_AdministratorAccess_<suffix>"
    access        = "admin"   # AmazonEKSClusterAdminPolicy
  }
  readonly = {
    principal_arn = "arn:aws:iam::<ACCOUNT_ID>:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_ReadOnlyAccess_<suffix>"
    access        = "view"    # AmazonEKSViewPolicy
  }
}

NAME                         STATUS   ROLES    AGE   VERSION
ip-10-0-1-xx.ec2.internal    Ready    <none>   3m    v1.37.0-eks-...
ip-10-0-2-xx.ec2.internal    Ready    <none>   3m    v1.37.0-eks-...

You should see 41 resources added, two nodes Ready, and the system pods running. The control plane takes about 10 minutes and the node group about 3 more. Keep the laptop awake during this apply. In rebuild 1, the Mac slept for four minutes in the middle of it. Terraform lost track of its own request, retried it, and got "node group already exists". The fix is in the pitfalls table below. To undo, destroy module.gpu_compute (the cluster and its node group live in that module) before phase 4 creates Kubernetes objects.

5Phase 4 — Model weights on EFS

Goal: the weights of Qwen2.5-7B-Instruct-AWQ sit on Amazon EFS, a managed network file system, so any new GPU node can load them inside the VPC in seconds.

The model choice is about the hardware, not the price. The GPU node costs the same whichever model runs on it, so pick the best model that fits the cheapest GPU: a 16 GB NVIDIA T4. Qwen2.5-7B-Instruct, quantized with AWQ (a method that stores the weights in 4 bits instead of 16), is 5.6 GB, uses the Apache 2.0 license, and is not gated, so it needs no Hugging Face token. That leaves room on a T4 for the KV cache: the memory vLLM uses to hold the context of the requests it is serving.

EFS is set to Bursting throughput, which has no throughput charge; a new file system starts with burst credits. Backups are off because the weights can be downloaded again. A StorageClass creates one EFS access point per claim, under /models. The file system ID changes on every rebuild, so the manifest holds ${EFS_FILE_SYSTEM_ID} and envsubst fills it in from Terraform's output.

make addons    # the final full apply: 38 resources, and no changes to phases 1-3
make storage   # vllm namespace, StorageClass, PVC
make model     # one-off Job: hf download onto EFS

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-models
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap               # one EFS access point per claim
  fileSystemId: ${EFS_FILE_SYSTEM_ID}    # filled in by envsubst: it changes every rebuild
  directoryPerms: "755"
  basePath: /models
reclaimPolicy: Retain                    # keep the weights if the claim is deleted

containers:
  - name: download
    image: public.ecr.aws/docker/library/python:3.12-slim
    command: ["sh", "-c"]
    args:
      - >-
        pip install --quiet --no-cache-dir "huggingface_hub[hf_xet]==2.2.0" &&
        hf download "$MODEL_ID" --local-dir "/models/$MODEL_DIR"
    env:
      - name: MODEL_ID
        value: Qwen/Qwen2.5-7B-Instruct-AWQ   # Apache 2.0, ungated: no token
    volumeMounts:
      - name: models
        mountPath: /models

make addons is the full apply. It creates EFS, and it also installs everything in phase 5 and the load balancer controller for phase 6, so those phases have no Terraform step of their own.

You should see:

• 38 resources added, and no changes to phases 1 to 3.
• The claim model-weights is Bound.
• The Job lists the files: 5.6 GB, which ls -h shows as 5.2G because it counts in GiB.

To undo, delete the vllm namespace and the StorageClass; EFS goes with the final destroy.

6Phase 5 — Karpenter, KEDA and Prometheus

Goal: the controllers that scale vLLM are running. No GPU node can start yet, because nothing allows one.

There is nothing to run in this phase. make addons in phase 4 already installed these controllers. This phase explains what each one does and checks that it works.

Each controller has one job:

• Karpenter watches for pods that cannot be scheduled. It starts the cheapest EC2 instance that fits them and removes the instance when it is empty. An SQS queue and five EventBridge rules deliver the two-minute spot interruption notice. Karpenter then moves pods off the node before AWS reclaims it.
• KEDA (Kubernetes Event-driven Autoscaling) changes a Deployment's replica count from an external metric. Here, the metric is how many requests are waiting inside vLLM.
• Prometheus is here only because KEDA's Prometheus scaler runs PromQL queries (Prometheus’s query language) against a server. It is installed as the server alone, with no Alertmanager, no exporters and no disk, and it keeps two hours of data.
• The AWS Load Balancer Controller creates the NLB for vLLM. AWS recommends it over the built-in Kubernetes controller, which now receives only critical fixes.

The versions are pinned for a reason. Karpenter 1.15 is the only release line that supports Kubernetes 1.37. KEDA 2.21 is tested up to 1.36; on 1.37 it logs a warning and works, because the APIs it uses are stable. The Karpenter Helm release depends explicitly on its IAM role and its Pod Identity association, the EKS feature that gives a pod an IAM role. Without that dependency, the controller can start before its permissions exist. It then fails with UnauthorizedOperation and restarts once.

resource "helm_release" "karpenter" {
  repository = "oci://public.ecr.aws/karpenter"
  chart      = "karpenter"
  version    = var.karpenter_version   # 1.15.0: the only line that supports Kubernetes 1.37

  # The controller calls EC2 at start-up, so its IAM role and Pod Identity
  # association must exist first (otherwise: UnauthorizedOperation, restart).
  depends_on = [module.karpenter]

  values = [yamlencode({
    settings     = { clusterName = module.eks.cluster_name,
                     interruptionQueue = module.karpenter.queue_name }
    nodeSelector = { role = "system" }   # never on a node it manages
  })]
}

resource "helm_release" "prometheus" {
  chart   = "prometheus"
  version = var.prometheus_version
  # Server only: KEDA needs PromQL, nothing else.
  values = [yamlencode({
    alertmanager             = { enabled = false }
    kube-state-metrics       = { enabled = false }
    prometheus-node-exporter = { enabled = false }
    prometheus-pushgateway   = { enabled = false }
    server = { retention = "2h", persistentVolume = { enabled = false } }
  })]
}

kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter   # 2 Running
kubectl get pods -n keda                                               # 3 Running
kubectl get pods -n monitoring                                         # 1 Running
kubectl get apiservice v1beta1.external.metrics.k8s.io                 # Available True

You should see two Karpenter pods (one leader, one standby that logs "attempting to acquire leader lease"), three KEDA pods, and one Prometheus pod, all running. The external metrics API should report Available. Two KEDA pods may show one restart each, from a start-up race over a certificate file. That is harmless if the count stays at one. Test Prometheus from a pod inside the cluster. The kubectl get --raw .../proxy path hangs, because the control plane cannot reach the pod's port.

7Phase 6 — vLLM on a spot GPU

Goal: Karpenter starts a spot GPU node because vLLM asks for one. vLLM serves the model from EFS behind an internal NLB, and the gateway sends standard requests to it.

The node pool is where the cost control lives:

• Spot only, from three GPU families (g4dn, g5, g6) in the xlarge size: 4 vCPUs and one GPU. Karpenter picks whichever is cheapest and available. It chose a g4dn.xlarge (T4) in the original build and a g5.xlarge (A10G) in rebuild 2.
• A CPU limit of 4, matching the quota, so the pool never asks for a second node the account cannot run.
• A taint, which keeps every other pod off the GPU.
• Bottlerocket as the node's operating system. Its NVIDIA variant ships the GPU driver and the device plugin, so there is nothing else to install.

vLLM's flags are set for the weakest GPU in the pool. A T4 has no bfloat16 support, so the data type is half. It also cannot run the faster kernels for 4-bit weights, so quantization is plain awq. The A10G runs the same flags.

make gpu        # EC2NodeClass + NodePool: no node yet, nothing needs a GPU
make vllm       # Service (NLB), Deployment, ScaledObject: the pending pod triggers Karpenter
kubectl get nodeclaims -w
# gpu-k6z7z   g5.xlarge   spot   us-east-1b   ...   True

make vllm-url   # prints http://k8s-vllm-vllm-xxxxxxxxxx.elb.us-east-1.amazonaws.com:8000
# edit terraform.tfvars (see the next step), then:
make gateway    # rolls the gateway onto the new URLs

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  template:
    spec:
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: gpu }
      taints:
        - { key: nvidia.com/gpu, value: "true", effect: NoSchedule }
      requirements:
        - { key: karpenter.sh/capacity-type,        operator: In, values: ["spot"] }
        - { key: karpenter.k8s.aws/instance-family, operator: In, values: ["g4dn", "g5", "g6"] }
        - { key: karpenter.k8s.aws/instance-size,   operator: In, values: ["xlarge"] }
  limits:
    cpu: "4"                       # = the 4-vCPU spot quota: at most one GPU node
  disruption:
    consolidationPolicy: WhenEmpty # never move a running vLLM pod
    consolidateAfter: 5m

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu
spec:
  role: ${KARPENTER_NODE_ROLE}
  amiSelectorTerms:
    - alias: bottlerocket@latest   # NVIDIA variant: driver + device plugin included
  subnetSelectorTerms:
    - tags: { karpenter.sh/discovery: ${CLUSTER_NAME} }
  securityGroupSelectorTerms:
    - tags: { karpenter.sh/discovery: ${CLUSTER_NAME} }
  blockDeviceMappings:
    - deviceName: /dev/xvda        # OS volume: the snapshot is 6 GB, so 4 GiB fails
      ebs: { volumeSize: 8Gi, volumeType: gp3, encrypted: true }
    - deviceName: /dev/xvdb        # container images: vLLM is ~20 GB unpacked
      ebs: { volumeSize: 80Gi, volumeType: gp3, encrypted: true }

spec:
  enableServiceLinks: false     # else Service "vllm" injects VLLM_PORT=tcp://... and vLLM crashes
  nodeSelector: { role: gpu }
  tolerations:
    - { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }
  containers:
    - name: vllm
      image: docker.io/vllm/vllm-openai:v0.32.0
      args:
        - /models/qwen2.5-7b-instruct-awq
        - --served-model-name=qwen2.5-7b-awq
        - --quantization=awq        # plain AWQ kernels: the faster ones need a newer GPU than a T4
        - --dtype=half              # a T4 has no bfloat16
        - --max-model-len=8192
        - --enable-chunked-prefill  # long prompts are processed in pieces, between other requests
      resources:
        limits: { nvidia.com/gpu: "1", memory: 14Gi }
      volumeMounts:
        - { name: models, mountPath: /models, readOnly: true }   # the EFS claim
        - { name: shm, mountPath: /dev/shm }

apiVersion: v1
kind: Service
metadata:
  name: vllm
  namespace: vllm
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external   # the LB controller, not the in-tree one
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internal
spec:
  type: LoadBalancer
  loadBalancerSourceRanges: [ "${VPC_CIDR}" ]
  ports: [ { name: http, port: 8000, targetPort: http } ]

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm
  namespace: vllm
spec:
  scaleTargetRef: { name: vllm }
  minReplicaCount: 1
  maxReplicaCount: 1               # raise with the quota
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-server.monitoring.svc.cluster.local
        query: sum(vllm:num_requests_waiting{namespace="vllm"})
        threshold: "2"

Two settings in those files look odd. Both fix real failures. The EC2 node class asks for an 8 GiB system volume, because the current Bottlerocket snapshot is 6 GB and our first 4 GiB setting was rejected. The Deployment sets enableServiceLinks: false. Kubernetes injects environment variables for every Service in the namespace, and a Service named vllm produces VLLM_PORT=tcp://172.20.x.x:8000. vLLM reads that as its own port setting and stops with "appears to be a URI".

You should see a NodeClaim of type g4dn.xlarge, g5.xlarge or g6.xlarge with capacity spot, then vLLM 1/1 Running. The logs show the KV cache it got. A T4 holds 113,744 tokens, which is 13.9 requests at the full 8,192-token context. An A10G holds 239,440 tokens, about 29 such requests.

# g4dn.xlarge (T4, 16 GB)
Loading weights took 20.75 seconds
GPU KV cache size: 113,744 tokens, Maximum concurrency for 8,192 tokens per request: 13.88x

# g5.xlarge (A10G, 24 GB)
Loading weights took 26.94 seconds
GPU KV cache size: 239,440 tokens, Maximum concurrency for 8,192 tokens per request: 29.23x

kubectl port-forward -n vllm svc/vllm 8000:8000 &
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "qwen2.5-7b-awq", "max_tokens": 100,
  "messages": [{"role": "user", "content": "Extract the invoice number from: INV-4471, due 2026-11-01."}]
}' | jq -r '.choices[0].message.content'
# The invoice number is INV-4471.

Last, point the gateway at vLLM. Run make vllm-url, put the address into terraform.tfvars with the two suffixes shown below, and run make gateway. ECS rolls the two tasks onto a new task definition without downtime. vLLM itself has no API key in this setup. The NLB accepts any caller inside the VPC, which is fine for a lab and not for production.

# terraform.tfvars: the address from make vllm-url, with these two suffixes
vllm_api_base    = "http://k8s-vllm-vllm-xxxxxxxxxx.elb.us-east-1.amazonaws.com:8000/v1"
vllm_metrics_url = "http://k8s-vllm-vllm-xxxxxxxxxx.elb.us-east-1.amazonaws.com:8000/metrics"

8Phase 7 — Prove the routing

Goal: show each path working, from a CloudShell VPC environment, using the response header and the router's log lines: the rules, the cache, and the fallback.

ask(){   # $1 prompt, $2 optional tag, $3 optional model (default: auto)
  local extra=""; [ -n "$2" ] && extra=",\"metadata\":{\"tags\":[\"$2\"]}"
  curl -s -D /tmp/h -o /tmp/b $ALB/v1/chat/completions \
    -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
    -d "{\"model\":\"${3:-auto}\",\"max_tokens\":60$extra,\"messages\":[{\"role\":\"user\",\"content\":\"$1\"}]}" \
    -w "%{http_code} "
  grep -i '^x-litellm-model-group' /tmp/h | tr -d '\r'
}
ask "Extract the invoice number from: INV-4471, due 2026-11-01."   # A
ask "Is a verbal contract enforceable?" reasoning                    # B
sleep 5
ask "Extract the invoice number from: INV-9902, due 2026-12-15."   # C

== A. short extraction (rules say standard)
  HTTP/1.1 200 OK | x-litellm-model-group: vllm-local
  -> The invoice number is INV-4471.
== B. reasoning tag (rule says complex)
  HTTP/1.1 200 OK | x-litellm-model-group: bedrock-claude
== C. similar extraction (semantic cache)
  HTTP/1.1 200 OK | x-litellm-model-group: vllm-local

route.decision source=rule mode=active cache=miss
route.chosen model=vllm-local
route.chosen model=bedrock-claude
route.judged tier=standard cached=1
route.decision source=cache mode=active cache=hit cached_tier=standard rule_tier=standard similarity=0.941
route.chosen model=vllm-local

Request C hit the cache in the original build and missed it in rebuild 2, although both times it went to vLLM, because the rules also say "standard". The gateway runs two tasks, each with its own in-memory cache, and the load balancer sent C to the task that had not seen A. Each task learns on its own, so a new kind of prompt can miss once on each task before both hit.

The fallback is the path that matters most when something breaks. Pause vLLM and send another extraction. In rebuild 2, the cache said "standard" (a hit at 0.952), the circuit breaker could not read vLLM's queue, and the request went to Bedrock. The caller still got the right answer.

# pause vLLM: KEDA scales it to 0 replicas
kubectl annotate -n vllm scaledobject vllm autoscaling.keda.sh/paused-replicas=0 --overwrite

ask "Extract the invoice number from: INV-5530, due 2027-01-10."
# 200 x-litellm-model-group: bedrock-claude
# -> The invoice number is: INV-5530

# router log: the cache said standard, the circuit breaker overruled it
# route.decision source=cache mode=active cache=hit cached_tier=standard similarity=0.952
# WARNING plugins.circuit_breaker vLLM metrics unavailable at http://k8s-vllm-vllm-xxxx...:8000/metrics
# route.chosen model=bedrock-claude

# resume vLLM (a new GPU node: about 9 more minutes)
kubectl annotate -n vllm scaledobject vllm autoscaling.keda.sh/paused-replicas-

9Phase 8 — Tear it all down

Goal: nothing from this workshop is left running or billing.

terraform destroy alone is not enough. Some AWS resources are created by controllers inside the cluster, not by Terraform, and if the cluster goes first, those resources stay and keep billing. Karpenter launches the GPU instances, so delete its node pools while it still runs. The load balancer controller creates the NLB, so delete the vllm namespace and its Service while that controller still runs. The teardown script does this in order, then empties ECR, then runs terraform destroy.

Emptying ECR also needs care. Docker pushes an image index plus untagged child manifests, one for the image itself and one for a build attestation. In a single batch-delete-image call, ECR deletes the index but refuses each child, because the index still references it at that moment. It reports those as failures and still exits with status 0. Our first teardown stopped on exactly this. The script now repeats the call until the list is empty.

make destroy
# ==> Kubernetes: removing workloads that own AWS resources
# nodepool.karpenter.sh "gpu" deleted
# Waiting for Karpenter to terminate the GPU instances (usually 5-7 minutes).
# ...
# ==> Terraform: destroying everything (review the plan, then type yes)

empty_ecr_repository() {
  # Docker pushes an image index plus child manifests. ECR won't delete a child
  # while an index in the same call still references it, so repeat until empty.
  for pass in 1 2 3; do
    ids="$(aws ecr list-images --repository-name "$repo" --query 'imageIds[*]' --output json)"
    [ "$ids" = "[]" ] && return 0
    aws ecr batch-delete-image --repository-name "$repo" --image-ids "$ids" >/dev/null
  done
  echo "ECR repository $repo is still not empty" >&2; exit 1
}

# stop paying for the GPU without tearing down: KEDA owns the replica count
kubectl annotate -n vllm scaledobject vllm autoscaling.keda.sh/paused-replicas=0 --overwrite
# back on
kubectl annotate -n vllm scaledobject vllm autoscaling.keda.sh/paused-replicas-

You should see Terraform's state empty, and no instance, load balancer, target group, EFS file system, ECR repository, IAM role or log group left with the stack's name. Two things stay on purpose: the state bucket, and the master-key secret, which Secrets Manager keeps for a 30-day recovery window. A rebuild creates a new secret name, so the two never clash.

To stop paying for the GPU without tearing everything down, pause KEDA at zero replicas. In the rehearsal, Karpenter terminated the empty node 5 minutes 20 seconds later. Resuming took about 9 minutes, plus a three-minute wait: the first spot launch failed with InsufficientCapacityError, and Karpenter waited about three minutes before it tried again and succeeded.

10Hardening it for production

The lab settings save money. Production needs different choices, and each one is a small change to what you already built:

• Turn on the interface endpoints (enable_vpc_endpoints = true). Bedrock and ECR traffic then stays off the internet path. Some security reviews require that, whatever the cost.
• Make the gateway the only way to reach vLLM. Limit the NLB's source to the gateway's security group, or start vLLM with --api-key. Today, anything inside the VPC can call it.
• Use one NAT gateway per zone. A single NAT gateway is a single point of failure for every outbound call, including Bedrock.
• Allow scale-out. Raise the spot quota to 8 vCPUs or more. Then raise the node pool's CPU limit and KEDA's maxReplicaCount together. KEDA can then add a second vLLM replica when the queue grows, and Karpenter adds the node for it.
• Shorten the cold start. The image pull dominates it. Two common options: an EBS snapshot with the image already pulled, attached by the node class, or lazy image loading (SOCI). Both need measuring on your own workload.
• Plan for no spot capacity. Add more GPU families, or an on-demand fallback. On-demand GPUs need their own quota, which also starts at 0. Until then, Bedrock is the fallback, at Bedrock's price.

11Pitfalls: what the builds found

Every row happened during this build, and every fix is already in the repository. The rows marked "rebuild 1" are the ones the original build missed.

Conclusion

A self-hosted model saves money only when the parts around it are cheap and safe. A gateway gives it only the work it can do. Bedrock answers whenever it cannot. Karpenter starts the spot GPU on demand and you pause it between sessions, the weights load in seconds from shared storage, and a teardown leaves nothing behind. This workshop builds them with Terraform, Kubernetes manifests and a Makefile, one phase at a time, each with a check that proves it worked.

The most useful step was the one that felt unnecessary: deleting a working system and rebuilding it from the written steps. Rebuild 1 found the bugs a first build hides, because the first build already knew the answers. What this workshop does not claim yet is how much the routing saves. That needs real traffic measured against an all-Bedrock baseline, and it is the next step.

References

• Workshop source: enterprise-ai-production-blueprints / 01-hybrid-model-router, including docs/workshop/ modules 00–09.
• Amazon EKS User Guide: access entries, and network load balancing with the AWS Load Balancer Controller.
• Karpenter documentation: Kubernetes version compatibility, NodePools and EC2NodeClasses, disruption.
• KEDA documentation: cluster compatibility, Prometheus scaler, pausing autoscaling.
• vLLM documentation: GPU installation (compute capability 7.5 or higher), environment variables (the Kubernetes VLLM_PORT warning), production metrics.
• Service Quotas User Guide; Amazon EC2 quota "All G and VT Spot Instance Requests" (L-3819A6DF).
• Amazon EFS User Guide: mount targets and DNS names. Bottlerocket documentation: NVIDIA variants.
• Qwen/Qwen2.5-7B-Instruct-AWQ model card on Hugging Face.