Level 400: Engineering AI systems · Module 400.6

Three colleagues at a table of blank index cards; one holds up a single card and asks a question.
Out of how many?

Evaluations that hold up · Technical gear

You will be able to: design an evaluation for a Big Widget task (held-out data matched to the deployment distribution, a baseline, a harm-linked metric and an a-priori sample size from a power calculation); identify contamination and adaptive test-set reuse, and decompose a score drop on a same-process replication into adaptivity, distribution and generalization gaps; and report an eval with standard errors (naive, clustered and paired), decide whether a difference is distinguishable from noise or large enough to act on, and validate an LLM judge against independent human labels.

This module is as of November 2026. Results about particular models are dated to the papers that report them. Big Widget Company is made up, and every Big Widget number below is made up for teaching.

1. A replication that did not replicate

Setup

Recht, Roelofs, Schmidt and Shankar (arXiv 1902.10811; ICML 2019) formalise what a test score is supposed to estimate.1 For a classifier f̂ and a data distribution D, the population loss is

L_D(f̂) = E_(x,y)∼D [ 𝟙[f̂(x) ≠ y] ],

and a test set S drawn from D gives the estimate

L_S(f̂) = (1/|S|) Σ_(x,y)∈S 𝟙[f̂(x) ≠ y].

They state that this assumption, that a low L_S means similar performance on future draws from D, "underlies essentially all empirical evaluations in machine learning".1

They then collected new test sets S′ from a distribution D′ built to match D, by repeating the CIFAR-10 and ImageNet creation processes (same source pools, Flickr images from a similar time frame for ImageNet, and MTurk tasks modelled on the originals).1 The drop on S′ decomposes into three terms:1

L_S − L_S′ = (L_S − L_D) + (L_D − L_D′) + (L_D′ − L_S′) adaptivity gap distribution gap generalization gap

Results

Accuracy fell by 3 to 15 percentage points on CIFAR-10 and 11 to 14 on ImageNet (the paper writes these absolute drops as "%").1 Examples from their Table 1: ResNet-152, 78.3% → 67.0% top-1 on ImageNet; the best CIFAR-10 model, 98.4% → 95.5%.1

To separate adaptivity from distribution shift, they used many models. If later, more-tuned models gained mainly through adaptivity, new-set accuracy would show diminishing returns. Instead the fit is linear with slope above 1:1

acc_new ≈ 1.69 · acc_orig − 72.7% (CIFAR-10), acc_new ≈ 1.11 · acc_orig − 20.2% (ImageNet top-1),

with bootstrap 95% intervals on the slope of [1.63, 1.76] and [1.07, 1.19].1 The authors call adaptivity "an unlikely explanation" and point to the distribution gap.1 Supporting that, three ImageNet sampling strategies drawn from the same candidate pool, all correctly labelled, shifted average top-1 accuracy from −11.8% to +2.1% relative to the original validation set, a spread of about 14 points.1

Two limits the authors state themselves: the experiments "do not exclude all variants" of adaptive overfitting (for example, a roughly constant drop affecting all models equally), and testing that would need "a new test set that is truly i.i.d.", which motivates their "super hold-out" proposal.1

What it shows for an engineer: a point estimate on one test set measures that set. Small, undocumented choices in collection and labelling produce shifts far larger than the sampling error, and rankings can be stable while absolute scores are not.1

The misconception this corrects: "If the model never saw the test file, the score is clean." The new sets were independent of every model by construction, so the gap was not leakage, and the scores still fell well outside sampling error.1

2. Contamination and adaptive reuse

Taxonomy

Sainz et al. (arXiv 2310.18018; Findings of EMNLP 2023) define contamination broadly as "any breach in the strict control of datasets required by the experimental protocol", focusing on an LLM that processed a benchmark during pre-training.2 Their levels:2

TypeWhat leakedConsequence (as the paper states it)
GuidelineAnnotation guidelinesAdvantage, especially for zero- and few-shot evaluation
Raw textSource text before annotation (e.g. Wikipedia-based datasets)Partial information about the labels, e.g. entity boundaries
Annotation, evaluation splitTest labels"the experiment is completely invalidated"
Annotation, train/dev splitTraining labelsComparisons with models trained on those splits survive; zero-/few-shot claims are invalidated

They place contamination at three stages: pre-training, supervised fine-tuning, and after deployment, where API-served models "have been iteratively improved with user input, leading to evaluation data exposure".2 HELM (Liang et al., arXiv 2211.09110; TMLR 2023) widens the definition to exposure to the evaluation distribution, and distinguishes strong from weak contamination by how much of that distribution the model saw.3

Detection

HELM states the practical position for most evaluators: with "uneven and usually limited information" on training, "we cannot verify whether the model was trained on the data we evaluate it on"; they flag disclosed contamination next to affected results.3

Adaptive reuse

Reuse is the adaptivity term from section 1. In Recht et al. it arises when researchers tune hyperparameters "directly on the test set".1 This course reads the mechanism (dependence between f̂ and S) more widely than hyperparameters: it is produced by whoever lets test results flow back into model, prompt or threshold selection, whether the model's builders, the evaluating team, or both. Updating weights is not required; choosing among prompts by test score is enough.

A one-look policy (course practice; the cited studies do not give a rule for repeated release decisions):

  1. A development set, used freely for prompt and hyperparameter work.
  2. A test set with n fixed by the power calculation (section 3), opened once for one release decision. The CI in section 4 is a fixed-n, single-look interval; it is not a stopping rule.
  3. If the decision is hold, do not append items to the same test set and recompute until the lower bound clears 0. Repeated looks with data-dependent stopping let test results feed back into the decision, the same dependence as the adaptivity term; and the section 4 interval was computed for a fixed n and one look, not for a procedure that chooses when to stop by looking. Retire the opened set into development, redo the power calculation, and draw a fresh test set from inputs disjoint from everything seen.
  4. Count attempts. The decision record lists every prior test set and its outcome. Logging an opening, or calling a rerun "a new release decision", does not restore independence. A fresh set gives each attempt a valid fixed-n interval; it gives no family-wise bound across attempts, so the chance of at least one false allow grows with the number of attempts. The decider sees the full count. If the plan needs a campaign-wide error bound, it needs an error-allocation rule written before the first attempt; this module does not supply one.
  5. A hidden super hold-out, opened rarely, to check whether reuse of the working test sets has crept in.

A group-sequential or other multiple-look design, with its error budget allocated before the first look, is a legitimate alternative. It is not taught here, and must be specified in the pre-registered plan by someone qualified to design it.

3. Design: data, baseline, metric, sample size

An evaluation needs held-out data that matches real use, a baseline, a metric tied to the harm, and a sample size planned in advance. In Miller's worked example, about 969 independent questions are needed to tell two models 3 points apart 80% of the time, under his stated assumptions.
Subtle differences in how a test set is collected move the score (s1). A score makes sense relative to other models' scores (s3). Choosing the metric from the harm is the course's advice. Miller's worked example needs about 969 independent questions to tell two models 3 points apart 80% of the time, under assumptions he calls “fictional” but “reasonable” (s3).
Figure data as text

Four parts of an evaluation

PartWhat it shows
1 · HELD-OUT DATA [s1]Like real use. Same sources and mix as service; collection details move the score.
2 · BASELINE [s3]Compared with what? A score mostly makes sense beside other scores: today's model, rule or people.
3 · METRIC (COURSE)Tied to the harm. Pick the metric from the harm you are trying to prevent, and say why.
4 · SAMPLE SIZE [s3]Planned in advance. Miller: about 969 independent questions to tell two models 3 points apart 80% of the time, under his stated assumptions.

Why external benchmarks do not transfer

Section 1 shows replication shift; section 2 shows unverifiable contamination. HELM adds measurement conditions: before HELM, models had on average been evaluated on 17.9% of its 16 core scenarios, and prompting details produced "wild swings in accuracies", from 30% to 80% for the same model and scenario.3 Zheng et al. (arXiv 2306.05685; NeurIPS 2023 Datasets and Benchmarks) report that benchmarks like MMLU and HELM "cannot effectively tell the difference" between aligned chat models and their base models, though people strongly prefer the aligned ones.4

The misconception this corrects: "A public leaderboard score tells you how a model will do on your task." It is evidence about a different distribution, under different conditions, with unknown contamination.

The four design choices

  1. Held-out data from the deployment distribution. Sample from the inputs the model will serve, from a period disjoint from anything used in development. For drifting inputs, prefer a temporal split (most recent period) to a random split (course design advice).
  2. A baseline on the same items. Miller (arXiv 2411.00640, 2024, a preprint): a single model's score "usually does not have any inherent meaning".5 Run the incumbent (model, rule or human process) on the identical items so you can use a paired analysis (section 4).
  3. A metric tied to the harm. HELM targets seven metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) for each core scenario "to the extent possible", which it reports as 87.5% of the time: a reminder that accuracy is one column of many, and that an evidence matrix should show its empty cells.3 Choose the metric whose errors correspond to the harm, and report the others you need as guards.
  4. A sample size from a power calculation, fixed in advance. For a paired comparison with significance level α, power 1 − β and minimum detectable effect δ, Miller gives5

    n = (z_α/2 + z_β)² (ω² + σ²_A/K_A + σ²_B/K_B) / δ²,

    where ω² = Var(x_A) + Var(x_B) − 2Cov(x_A, x_B) is the variance of the per-question mean difference, σ²_A and σ²_B are the within-question (resampling) variances, and K_A, K_B the number of samples per question. With σ²_A = σ²_B = 0, ω² = 1/9, δ = 0.03, α = 0.05 and β = 0.20, this gives n ≈ 969, so he suggests new evals "should contain at least 1,000 questions".5 Inverted, the same equation gives the minimum detectable effect for a fixed n; in his example with n = 198, raising K from 1 to 10 cuts it from 13.2% to 7.5%.5

Big Widget: the safety-claim flagger (made up)

ChoiceDecision
TaskFlag free-text warranty claims that may describe a safety defect, for same-day engineer review
HarmA missed safety claim (false negative); secondarily, alert fatigue from false flags
Primary metricRecall on safety claims ("catch rate"), denominator = engineer-labelled safety claims
Guard metricFalse-flag rate on non-safety claims, with a ceiling the safety lead sets
BaselineCurrent keyword rule, scored on the same claims
Test setAll claims from the most recent full quarter; two engineers label independently before any model run; disagreements adjudicated; claims touched in prompt development excluded; stored outside the prompt writers' access
Sample sizePlanned for the positive class: at a prevalence of about 3%, 10,000 claims hold about 300 safety claims
DeciderThe safety lead signs the plan before the test set is opened, makes the release decision, and is the named person who can release a hold; the model and the evaluation pipeline decide nothing
Release rule (pre-registered, one look)Allow (replace the rule) only if the paired 95% CI for recall(model) − recall(rule) has a lower bound above 0 and the guard holds. If the CI spans 0: hold; the keyword rule stays, and any further attempt needs a fresh test set under the one-look policy (section 2). If the upper bound is below 0 or the guard fails: deny

4. Uncertainty, comparisons and judges

Standard errors

Treating eval questions as drawn from a super-population, Miller gives the CLT standard error of the mean score5

SE = √( Var(s) / n ), and for binary scores SE = √( s̄(1 − s̄) / n ),

with CI₉₅ = s̄ ± 1.96 · SE.5 He recommends reporting n and the SE in parentheses beside each score.5 Big Widget (made up): 270/300 safety claims caught, s̄ = 0.90, SE = √(0.9 · 0.1 / 300) ≈ 0.0173, CI₉₅ ≈ [0.866, 0.934].

Clustering. When questions are drawn in groups (several questions per passage; one question in many languages), independence fails and the naive SE is inconsistent. Miller's clustered SE adds the within-cluster cross-terms,

SE_clustered = ( SE²_CLT + (1/n²) Σ_c Σ_i Σ_(j≠i) (s_i,c − s̄)(s_j,c − s̄) )^½,

and on real eval data scored with Anthropic models it came out up to about 3× the naive SE (DROP: 1.34 vs 0.44, ratio 3.05).5 Claims from the same vehicle or dealer are a plausible cluster at Big Widget; recall also sits on top of the labelling step, which this formula does not cover.

Paired differences

For two models on the same n items, with d_i = s_A,i − s_B,i, Miller's paired SE is5

SE_paired = √( Var(d) / n ), where Var(μ̂_A−B, paired) = Var(μ̂_A−B, unpaired) − 2 Cov(x_A, x_B)/n.

Because models tend to agree on which items are hard, the covariance is usually positive, so pairing is "a 'free' reduction in estimator variance"; he recommends it "wherever practicable".5 In his fictional table, MATH gives +2.5% (SE 0.7%), CI (+1.2%, +3.8%), significant; MGSM gives −2.7% (SE 1.7%), CI (−6.1%, +0.7%), not significant at 5%.5

Worked Big Widget example (made up): 200 claims scored by models A and B. B is right where A is wrong on 14; A is right where B is wrong on 10; elsewhere they agree. Here we want B − A, so take d_i = s_B,i − s_A,i (the reverse of the convention above; a positive d̄ favours B, and the variance is the same either way). Then d̄ = (14 − 10)/200 = +0.020, the sample variance of d is (24 − 200 · 0.020²)/199 ≈ 0.1202, SE_paired = √(0.1202/200) ≈ 0.0245, and CI₉₅ ≈ +2.0 ± 4.8 points = (−2.8, +6.8). Honest report: "B − A = +2.0 points (SE 2.5; 95% CI −2.8 to +6.8; n = 200, paired). The data do not distinguish the two models." That is not evidence of equivalence. Under the release rule this is a hold, and the next attempt needs a fresh test set (section 2). Whether a difference that is distinguishable is worth acting on is the release rule's job (section 3), and the safety lead's decision, not the test's.

The misconception this corrects: "A two-point gain on a benchmark is a real improvement." With n = 200 and this discordance, a 2-point gain is well inside the noise.

Validating an LLM judge

Zheng et al. study three judge formats (pairwise comparison, single-answer grading, reference-guided grading) with 2023-era judges (GPT-4, GPT-3.5, Claude-v1).4 Measured failure modes:4

BiasProbeResult (as reported)
PositionTwo similar answers, order swappedConsistency: GPT-4 65.0%, GPT-3.5 46.2%, Claude-v1 23.8% (default prompt)
Verbosity"Repetitive list" padding on 23 answersFailure rate: Claude-v1 91.3%, GPT-3.5 91.3%, GPT-4 8.7%
Self-enhancementWin rates vs humansGPT-4 +10%, Claude-v1 +25% for themselves; the authors say their data cannot determine whether it is a bias
Maths grading10 questions, positions swapped (20 judgments)GPT-4 failures: default 14/20, chain-of-thought 6/20, reference-guided 3/20

Agreement. They define agreement between two judge types as the probability that randomly chosen individuals of each type agree on a randomly chosen question.4 On first-turn MT-bench (58 expert labellers, mostly graduate students):4

SetupChance (R)GPT-4 (pairwise) vs humanHuman vs human
S1: ties and position-inconsistent votes counted as ties33%66%63%
S2: non-tie votes only50%85%81%

Agreement with humans rose "from 70% to nearly 100%" as the win-rate gap between the compared models grew, so the judge was least reliable on close pairs.4 And when experts who disagreed with GPT-4 were shown its judgment, they rated it reasonable in 75% of cases and changed their choice in 34%.4 This course reads that as a route by which a judge can contaminate the human labels it is checked against.

A validation protocol (course practice built on these results):

  1. Fix the decision unit and label set (single-output classification such as faithful/unfaithful, or pairwise preference) and the tie rule. Humans and judge label the same unit.
  2. Sample items stratified to include close calls and long outputs.
  3. Collect ≥ 2 independent human labels per item before anyone sees the judge's output.
  4. Run the judge; for pairwise judging, run both orders and count inconsistency as a tie, as Zheng et al. do.4
  5. Report matched estimators on a common sample: judge–individual-human agreement and human–human agreement over the same items, in the sense of Zheng et al.'s definition above, with chance, n and an interval. Agreement with human consensus on the subset where humans agree is a different, conditional quantity; label it as such if you report it at all.
  6. Retain the human-disagreement items. They are the hard cases; send them for adjudication and analyse them, do not filter them out.
  7. Read the disagreements by stratum. A named person records the judge's permitted role (allow: replace; constrain: pre-sort; deny: none), the evidence, and the date.

The misconception this corrects: "An LLM judge agrees with people." Agreement is task-, format-, tie-rule- and difficulty-dependent; it is a measured property of this judge on this task.

Big Widget (made up): binary faithfulness labels on 50 claim summaries. Two engineers agree on 44 (88%). The judge matches the engineers' consensus on 36 of those 44 (81.8%), but that is conditional on human agreement and excludes the hard cases, so it is not comparable with 88%. The matched estimator: on the 44 agreed items the judge matches both engineers 36 times and neither 8 times; on the 6 disputed items a binary judge matches exactly one engineer. Judge–engineer agreement is therefore (2 · 36 + 6)/100 = 78%, against 88% engineer–engineer, on the same 50 items. With n = 50 both intervals are wide, and the 100 judge–engineer comparisons are not independent (two per item). Most judge misses are long summaries marked "faithful": a reason to run a targeted verbosity probe, not proof of the local cause. Decision recorded: constrain (pre-sort only, not replacement), made and signed by the quality lead; the 6 disputed items go to adjudication.

5. The three yeses for an evaluation

The evaluation is evidence for a gate; it is not the gate. A named person decides at each stage. The Foundation's three yeses (Get to Yes, Stay at Yes, Recover to Yes) are a cycle, not a one-time approval.

What you keep, and where:

Words to know: population loss, adaptivity gap, distribution gap, generalization gap, contamination (guideline, raw-text, annotation), extractability, standard error, clustered standard error, paired difference, minimum detectable effect, power, judge agreement, position bias, verbosity bias, development set, one-look policy, matched agreement estimator, not detected, not checked, work record, receipt.

Sources

  1. s1 Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar (2019). Do ImageNet Classifiers Generalize to ImageNet?. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, separate session from the writer), round 3 re-check 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  2. s2 Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre (2023). NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, separate session from the writer), round 3 re-check 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s5 Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, et al. (2022). Holistic Evaluation of Language Models. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, separate session from the writer), round 3 re-check 19 Nov 2026. ↩ ↩ ↩ ↩ ↩
  4. s4 Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, separate session from the writer), round 3 re-check 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  5. s3 Evan Miller (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, separate session from the writer), round 3 re-check 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩

Next: the quiz