~/ nmadiraju.com Gen AI · Evals on AWS
arrow_back Back to nmadiraju
Decision Tool · Evals · On AWS

How to Run LLM Evals on AWS

Answer five questions to pick how to run your evals on AWS, then follow Northwind's claims through a CI gate, a nightly judge job and human review. Builds on Build an Eval Harness You Can Ship On and its lab.

LLM evals on AWS in 60 seconds

  • Gate every PR yourself. Our pick: a CI harness diffs each case against a baseline and exits 1. Bedrock jobs are asynchronous: better as reports.
  • Judge nightly with Bedrock Evaluations. Custom judge metrics on your saved answers, billed as judge tokens. You still check the judge against your experts.
  • Send borderline cases to people. A Bedrock human evaluation job with your own work team: $0.21 per completed task.
  • For agents, grade the trace. AgentCore Evaluations scores goals and tool calls: online on live traffic, on demand, or in batch.
  • Our pattern: all three. One S3 dataset: the CI gate, the nightly judge, and human review of borderline cases.

1Make the call

One labeled dataset in S3 used three ways: a CI gate, a nightly Bedrock Evaluations job and human review Left, outside AWS: GitHub Actions running the harness, a hosted eval platform, and Northwind's adjusters. Right, Northwind's AWS account: IAM role eval-ci, EventBridge Scheduler and a Bedrock human evaluation job; Bedrock models, a Bedrock Evaluations judge job and AgentCore Evaluations; the S3 bucket with the dataset, baseline and results; Athena and a CloudWatch dashboard. Numbered markers 1 to 3 show where each use starts. OUTSIDE AWS GitHub Actions on: pull_request harness: gate.py EXTERNAL Hosted eval platform LangSmith · Braintrust Langfuse · Phoenix EXTERNAL Northwind adjusters your own work team rate in a web portal EXTERNAL NORTHWIND AWS ACCOUNT · us-east-1 IAM role eval-ci trusts GitHub's OIDC repo northwind/triage EventBridge Scheduler cron(0 2 * * ? *) a dated job name each night Human eval job Bedrock Evaluations Bedrock models triage model and judge Bedrock Evaluations LLM-as-a-judge job AgentCore Evaluations grades agent traces for agents S3 northwind-evals DATASETS claims.jsonl · 100 BASELINES v2.json · per case RESULTS one dataset, three uses Amazon Athena SQL over results/ CloudWatch dashboard one point per run

    2Choose or diagnose: check yourself

    Check yourself