WORDS YOU'LL MEETKnow them already? Skip ahead. Tap any dotted word later to see its card.

labelstep A1

The answer a person says is right for one case.

Like the expected value in a unit test.

C-1042 → urgent

graderstep A2

Code that decides whether one answer matches its label.

Like an assert.

"urgent" == "urgent" → pass

pass ratestep A2

The share of cases the grader passed.

Like the score on an exam.

74 of 100 → 74%

recallstep B2

Of the cases that really have a label (say urgent), the share the model gave that label.

Like: of the real fires, how many alarms rang.

6 of 9 urgent caught → 67%

baselinestep B4

A saved record of which cases passed with the last good version, to compare a change against.

Like a snapshot test.

baselines/v2.json

judgestep C2

A second model that grades free text against a written rubric (LLM-as-a-judge). Trust it only after calibration.

Like a code reviewer with a checklist.

{"verdict": "fail", "rule_violated": "uninhabitable_home"}

Build an Eval Harness You Can Ship On

1Visual Walkthrough