Try each exercise first, then open its model answer.
Evaluations that hold up · Workbook
Big Widget Company and all its numbers are made up for teaching.
Exercise 1 (400.6-a)
Big Widget's parts desk wants a language model to sort incoming dealer emails into "order", "return" or "other", so that returns reach the returns team the same day. Today a clerk sorts them by hand. A vendor points to its model's high score on a public language benchmark.
Write a one-page evaluation plan with these parts:
- The held-out test set: where the emails come from, which period, who labels them and when, and who may not see it.
- The baseline you will compare against.
- The main metric, tied to the harm, and one second metric. Say why each fits.
- How you will choose the sample size, and what you must count.
- The release rule, written before the run: who decides, and what happens if the result is too close to call.
Then say in one or two sentences why the vendor's benchmark score does not answer the question.
Model answer
A good plan contains:
- Held-out data that matches real use: real dealer emails from the most recent full period the team did not use while building, in their real mix and messiness. They are labelled by people who know the categories (for example two clerks) before any model sees them. Emails used to write or test the prompt are excluded, and the test set is stored where the prompt writers cannot read it.
- A baseline: the clerk's current sorting on the same emails. A score makes sense only next to another score (Miller).
- A metric tied to the harm: the harm is a return that waits, so the main metric is the share of real returns the model sorts as "return". A second metric is the share of other emails wrongly sent to the returns team, because a flooded team stops trusting the sort. Accuracy over all emails alone would not show the harm (HELM measured seven metrics so that ones beyond accuracy were not lost).
- A sample size planned in advance: the main metric's denominator is the number of return emails, so count those, not all emails. Choose the size from the smallest difference you need to see; Miller's worked example needed about 969 independent questions to detect a 3-point difference.
- A release rule written first and signed by a named person before the run: for example, the returns lead signs a rule that the model replaces the clerk's first sort only if, on one look at the paired test, the 95% interval for the difference in the return metric lies above zero and the wrong-send rate is no worse than a limit the returns lead sets (go ahead). If the interval includes zero, the clerk keeps sorting (wait for a person), and the returns lead decides whether a new attempt is worth a fresh test set. The team does not add emails to the same test and look again until the interval clears zero (the one-look rule), and every attempt is counted in the decision record. If the model is clearly worse, the answer is no. The model never decides its own release.
Why the benchmark score does not answer it: it measures a different test under different conditions. A new test built the same way can score lower (Recht and colleagues), the model may have seen the benchmark (contamination), and prompt details alone swung HELM's accuracies from 30% to 80%.
Exercise 2 (400.6-b)
A colleague says: "Our model scored 92% on the public reading benchmark, and the vendor confirms it never trained on the test file. So the score is clean, and it will hold up for us." Over the last quarter, the team also checked the same internal test set after every prompt change and kept whatever raised the score.
Write a short reply that:
- Names two ways a model can be contaminated without ever training on the test file.
- Explains what the team's habit with its internal test set does to that score, and what to do instead.
- Uses the ImageNet v2 result to explain why a score can drop on fresh data drawn the same way, and says what that result does not show.
- Writes one honest line for the report about what could not be checked.
Model answer
- Contamination without the test file: the model may have trained on the original text the benchmark was built from (raw text contamination; Wikipedia-based datasets are the example), or on the annotation guidelines (guideline contamination) (Sainz and colleagues). Models reached through an API may also have been improved with user input, so test items people typed in can leak. HELM also counts exposure to the evaluation distribution, not only the exact items.
- Test-set reuse: checking the test score after each change and keeping what raises it is the adaptivity gap: the test set slowly becomes training data, so its score flatters the model (Recht and colleagues). Instead, use a development set freely, and open a test set once, for one decision, at a size fixed in advance (the one-look rule). Logging an opening keeps an honest record; it does not make the set unseen again. For the next decision, move the used set into development and draw a fresh test set from inputs the team has not seen. Keep a hidden set back ("super hold-out") to check later whether reuse has crept in.
- ImageNet v2: a new test set built by closely following the original recipe cost models 11 to 14 points of accuracy. It shows that a score belongs to one test set and small differences in how a test set is built move the score (a distribution gap). It does not show that the models had memorised the old test (adaptivity was an unlikely explanation), and it does not show the ranking was wrong (the relative order was almost exactly preserved).
- Honest line: "Contamination not checked: vendor training data not disclosed. Next step: the project lead asks the vendor." A "not checked" line names who will find out, or says that no one can. If the team ran a memorisation probe and the model could not recite the benchmark, the line says "not detected", never "clean": whether a lack of memorisation shows a model was not trained on a benchmark "remains to be solved" (Sainz and colleagues).
Exercise 3 (400.6-c)
Big Widget compared two models, A (in use today) and B, on the same held-out test. The test holds 400 claims that two safety engineers labelled as safety claims before any model saw them, so all 400 are safety claims and 400 is the catch rate's denominator. The claims come from different vehicles and dealers, so the team treats them as independent. The release rule, signed by the safety lead before the run, says B replaces A only if the 95% interval for B minus A lies above zero, on one look at this test.
| Count of the 400 safety claims | |
|---|---|
| Both models flag it | 332 |
| Only B flags it | 30 |
| Only A flags it | 20 |
| Neither flags it | 18 |
| Catch rate | A 88.0% (352 of 400); B 90.5% (362 of 400) |
| Paired difference, B minus A | +2.5 points, 95% interval −1.0 to +6.0 |
The team also tried an LLM judge to grade claim summaries. Each summary is one unit, labelled "faithful" or "not faithful". Two engineers labelled the same 60 summaries independently, before seeing the judge, and agreed on 51. The judge labelled the same 60. On the 51 summaries the engineers agree on, the judge matched both engineers on 40 and neither on 11. Most of the judge's misses were long summaries it rated "faithful". Later the team also wants the same judge to pick the better of two summaries of one claim; it has not checked the judge for that use.
- Write the honest one-line report of the A-versus-B comparison, and say what happens next.
- A colleague proposes adding 200 more safety claims to the same test and running it again. Reply in two or three sentences.
- A colleague reports "the judge agrees with the engineers on 40 of 51 (78%), close to their 85%". Explain why those two numbers cannot be set side by side, and work out a judge figure that can.
- Say who decides what the judge may do, what you would recommend, and what the check still needs before anyone relies on it, including for the paired use.
Model answer
- "B minus A: +2.5 points in catch rate (95% interval −1.0 to +6.0; 400 safety claims, paired). The data do not tell the two models apart." The interval includes zero, so the release rule is not met: the answer is wait for a person. Model A stays, and the safety lead decides whether a new attempt is worth a fresh test set. This does not show the two models are equal. Miller recommends paired comparisons and reporting the uncertainty, and warns against the "highest number is best" reading. (The discordant counts make the interval checkable: B gains 30 − 20 = 10 claims, or 2.5 points.)
- "No. The interval was built for one look at a size fixed in advance. Topping up the same test and looking again until it clears zero is the same feedback loop as tuning on the test. If the safety lead wants another attempt, these 400 claims move into development, we plan a new size, and we draw a fresh test set from claims the team has not seen, counting this attempt in the decision record." A fresh test makes the next attempt's interval honest for that attempt, but one of many attempts can still pass by luck, so the safety lead sees every attempt, not only the latest. The hold is not a yes waiting to happen: the decision record names the safety lead as the one who can release it.
- 40 of 51 is agreement with the engineers' consensus, on a subset with the 9 hard cases removed; the engineers' 85% is two people agreeing across all 60. They are different measures. Zheng and colleagues measure agreement as how often a randomly chosen member of one group agrees with a randomly chosen member of the other, on the same questions, so compare the judge with each engineer on all 60. On the 51 shared items the judge makes 40 × 2 = 80 matches. On the 9 disputed items, with only two labels, it must match exactly one engineer: 9 matches. So judge–engineer agreement is (80 + 9) / 120 ≈ 74%, against 85% engineer–engineer, on the same 60 summaries. With only 60 items both figures are uncertain, so report n beside them, with the chance rate. The 9 disputed summaries go to the quality lead to settle and are kept for the next check.
- A named person decides, not the judge: here, the quality lead, who records the decision with the role and the date. The judge agrees with the engineers less often than they agree with each other, so it may not replace them. A fair recommendation is only this far: it may sort summaries for the engineers to read first. The lean toward long summaries is a reason to test for verbosity bias on purpose (Zheng and colleagues found judges preferred padded answers), not proof that length caused the misses. Before anyone relies on it, the check still needs: the chance rate beside both agreement figures, a look at the disagreements and close calls, and people's labels always collected before they see the judge's verdicts, because the judge's reasoning can sway people. The paired use is a different unit and needs its own check: people and the judge label the same pairs under the same tie rule, and the judge runs each pair twice with the order swapped, because of position bias.