A 15-part course

Evals from First Principles

Most eval advice is prose and dashboards. This one builds the measurement stack by hand, one concept at a time, so you can tell a real 2% gain from noise: metrics and the confusion matrix, golden sets, annotator agreement, LLM-as-judge and its biases, confidence intervals, significance and power, pass@k, arenas, calibration, and a CI regression gate. Every part runs on a small labeled set you can trace end to end, with runnable code and an interactive figure.

Start with Part 1: What Is a Score?

Core track · Parts 1–11

1 What Is a Score? 14 min read · code Before any statistic, evals need three atoms: a task, a gold label, and a metric. Part 1 builds the 2x2 confusion matrix by hand from ten graded outputs, derives accuracy, precision, recall, and F1 from its four counts, and shows on an imbalanced set why accuracy alone lies.
2 Building a Golden Set 17 min read · code A metric is only as good as the labeled set beneath it. Part 2 builds a trustworthy golden set by hand on 40 support tickets: stratified sampling pins a rare class at exactly 2 urgent per draw while simple random sampling swings from 0 to 4, a leakage trap inflates accuracy from an honest 0.667 up to 0.800, and a three-criteria rubric turns 'good answer?' into a countable pass-at-5 bar.
3 Do Humans Even Agree? 14 min read · code The gold labels you grade against are themselves opinions, and opinions have noise. Part 3 takes 20 support tickets graded by two humans who agree on 18 of them, a scary-high 90%, and shows by hand why most of that agreement is chance: Cohen's kappa collapses it to 0.61, Fleiss' kappa handles three raters, and Krippendorff's alpha waits as the general case.
4 LLM-as-Judge by Hand 13 min read · code When you cannot afford a human grader for every output, you hand the rubric to a model instead. Part 4 runs twelve labeled support answers through a deterministic mock judge, parses each free-text verdict into PASS, FAIL, or ABSTAIN, and scores the judge against human gold on the ten it could grade.
5 Judging the Judge 15 min read · code An LLM judge is a model, so it carries systematic, measurable biases. Part 5 baits a deterministic mock judge with a slot-A bonus and a per-word reward, measures its position bias (7 of 12 verdicts flip when you swap who goes first) and verbosity bias (the longer answer wins all 5 tied probes), then corrects the position bias by averaging both orderings and watches disagreement with the truth fall from 5/12 to 1/12.
6 Uncertainty 13 min read · code One eval run graded 30 outputs and gave accuracy 0.70, a single number that looks solid until you ask how much it would wobble on a different sample. Part 6 builds the bootstrap by hand: resample the 30 items with replacement 10000 times, read the 2.5th and 97.5th percentiles as a 95% CI of [0.53, 0.87], and watch that interval halve as the set grows.
7 Is the Difference Real? 15 min read · code Two models are graded on the same 50-item set: A gets 41/50 (82%), B gets 42/50 (84%), a headline gap of +2 points. Part 7 builds the paired 2x2 of agreements by hand, runs McNemar's test on the 9 items where the models disagree (both an exact binomial p and the chi-square approximation), finds it insignificant, and then works out that catching a true 2-point gain would need thousands of examples.
8 pass@k 12 min read · code One coding task, five sampled completions, two that pass the hidden tests: this part derives the unbiased pass@k estimator as a combinatorial identity, checks it by brute-force enumeration, and proves in exact arithmetic why the tempting plug-in formula reports too low.
9 Arenas 16 min read · code How leaderboards turn a pile of head-to-head battles into one ranking. Part 9 runs four chatbots through an 18-battle round-robin arena, ranks them two ways (Elo updated match by match, then the Bradley-Terry MLE fit to the aggregate win matrix), and shows numerically why Elo depends on match order while Bradley-Terry does not.
10 Calibration 13 min read · code A confidence is a promise: of the answers a model tags 80% sure, about 80% should be right. Part 10 scores the same ten trivia confidences twice, once for an over-confident model and once for a well-calibrated one, builds the reliability curve by hand across three bins, and reduces the whole picture to ECE and the Brier score.
11 Regression Gates 13 min read · code A score is worth nothing if it cannot say no. Part 11 turns the whole series into one CI gate that runs three checks on two candidate runs of a 20-item eval, a bootstrap-CI threshold, an n-gram contamination scan, and a PSI drift comparison, and blocks the deploy of the run whose mean looked fine but whose interval and distribution did not.

Frontier track · Parts 12–15

The production and agent territory you reach for once the offline gate is solid: the eval flywheel, online A/B and guardrail metrics, agent-trajectory evals, and a tiny end-to-end harness.

12 The Eval Flywheel 15 min read · code Your best eval set is not something you write, it is something you mine from what your model already did. Part 12 takes 12 production traces from a support assistant, ranks them by a cheap suspicion score, and spends a budget of 6 annotations to grow a golden set that catches all 4 real failures where random sampling catches only 2.
13 Online Evals 15 min read · code Offline scores are necessary, not sufficient: the final judge is live traffic. Part 13 splits one week of a support chatbot's sessions 50/50, computes the resolution-rate lift and a by-hand two-proportion z-test, and then lets a refusal-rate guardrail overrule a real, significant +8 pp win into a DO-NOT-SHIP verdict.
14 Agent-Trajectory Evals 17 min read · code Grading an agent's final answer misses how it got there. Part 14 scores the tool-call path of eight runs against an approved reference trajectory four ways (exact, order-aware, set-overlap, and step-level credit), then grades a pass/fail rubric with a mock judge and a human and chance-corrects their agreement with Cohen's kappa.
15 A Tiny Eval Harness 14 min read · code The capstone wires the whole series into one small pure-Python harness: a dataset, a swappable scorer, a bootstrap confidence interval, and a gate. We run it three ways on two tiny cases and watch swapping only the scorer flip the same ten-item QA data from HOLD to SHIP.

Companion code — runnable scripts & step-by-step notebooks for every part GitHub ↗