Eval Harness

An eval harness is infrastructure that runs a prompt or model against a fixed test set and computes aggregate scores per metric. It decouples what you test (the eval set) from how you run it (the harness), so the same tests can run against different models, prompt variants, or decoding settings with no code changes. A typical harness handles dataset loading, prompt templating, inference, scoring, and report generation. It is distinct from unit tests: eval harnesses produce distributional scores — accuracy, pass@k, F1, judge scores — not binary pass/fail assertions. A mature eval harness is the backbone of any prompt or model change review, making regressions visible before they reach production.