
Evaluations that hold up · Plain gear
You will be able to: plan a fair test for an AI tool at Big Widget, using real examples it has not seen, something to compare it with, a score that matches the harm, and enough examples chosen before you start; spot when a test has leaked or been used too often, and explain why scores can drop on new examples gathered the same way; and report a result with how sure you are, say when two tools are too close to call, and check an AI "judge" against people before you trust it.
This module is as of November 2026. Big Widget Company is made up. So are its people, plans and numbers. The studies are real, and each one is dated.
1. The test that was copied, and scored lower
Think of a driving test. Your instructor has a set of practice routes. You learn them well. Then the real test uses a new route, built the same way. You expect to do just as well. Do you?
In 2019, four researchers asked this question about AI.1 Two famous tests for image AI, called CIFAR-10 and ImageNet, had been used by researchers for almost ten years.1 The researchers built new copies of both tests. They followed the original recipe closely: same kinds of photos, same sources, same way of labelling them.1 Then they ran many well-known AI models on the new copies.
Every model did worse. On the new ImageNet test, the models lost 11 to 14 points of accuracy.1 One well-known model went from about 78 right out of every 100 pictures to about 67.1 The authors said that was like losing about five years of progress in the field.1
What does this show? A test score belongs to that one test. It is not a promise about every example you will meet. Even small changes in how a test is put together can move the score a lot.1
What does it not show? It does not show the models had memorised the old test. The researchers checked for that, and it was not the likely cause.1 It also does not show the ranking was wrong. The best models on the old test were still the best on the new one.1
A common mistake: "If the AI never saw the test, the score is fair." The models had never seen the new pictures. They still scored lower.1 A fair score also needs a test that looks like real life.
2. Two ways a test can leak
The AI saw the test
Large AI models learn from huge piles of text taken from the internet. Tests are often on the internet too. So the AI may have read a test before it takes it. Researchers call this contamination. It makes the AI look better than it is.2
It can happen in sneaky ways. The AI might have seen the answer key. It might have seen the original text the test was made from. It might have seen the instructions the test makers used.2 If it saw the answers to the test itself, the result is worthless.2
It is hard to check from the outside. Many AI makers do not share what their AI read.2 One big study said plainly that it could not check whether the models it tested had trained on its test data.3 And AI tools you use over the internet can learn from what users type in. So test questions that people type in can end up inside the tool.2
The team saw the test
The second leak is through people. Imagine a student who sees their test score, changes their study plan, and takes the same test again. Soon they are learning the test, not the subject. Teams who keep checking an AI on one test, and tweaking it each time, can do the same thing.1 The ImageNet researchers suggested keeping a "super hold-out": a test kept secret for years, so it can be used later to check whether this has happened.1
At Big Widget: the team keeps a practice set they can use any time. They keep a real test set locked away. They open it once, for one decision. After that, everyone has seen it, so it becomes practice too. Writing down that they opened it is honest, but it does not make the test secret again. The next decision gets a new, unseen test set.
3. Planning a fair test
Figure data as text
Four parts of an evaluation
| Part | What it shows |
|---|---|
| 1 · HELD-OUT DATA [s1] | Like real use. Same sources and mix as service; collection details move the score. |
| 2 · BASELINE [s3] | Compared with what? A score mostly makes sense beside other scores: today's model, rule or people. |
| 3 · METRIC (COURSE) | Tied to the harm. Pick the metric from the harm you are trying to prevent, and say why. |
| 4 · SAMPLE SIZE [s3] | Planned in advance. Miller: about 969 independent questions to tell two models 3 points apart 80% of the time, under his stated assumptions. |
Why a public score is not enough
AI makers often share scores on public tests. These are a starting point, not an answer. A new test built the same way can score lower.1 The AI may have seen the public test.2 And small changes in how questions are asked can swing a score from 30% to 80% for the same AI.3
A common mistake: "A high score on a public chart means it will do well on our job." Only a test on your job tells you that.
Four things a fair test needs
- Real examples it has not seen. Use examples like the ones the AI will really get, from a time the team did not use while building it.
- Something to compare with. A score means little on its own. It makes sense next to another score.4 Compare with what you do today.
- A score that matches the harm. One number rarely tells the whole story. A big study set out to measure seven things, not just accuracy, and said where it could not.3 Pick the score that tracks the harm you care about.
- Enough examples, chosen before you start. Too few examples, and a real difference can hide in the noise. In one worked example, a researcher found that spotting a small difference between two AIs needed about 1,000 questions.4 Pick the size, the score and the pass mark before you run the test. Then the result cannot change the rules.
At Big Widget (made up)
Dealers send Big Widget written warranty claims. A few describe safety problems: brakes, steering, a fire. Today a simple word list flags those claims for a safety engineer. The team wants to try an AI instead.
- The harm: a safety claim that is missed waits in the slow queue. So the main score is: out of the real safety claims, how many did the AI flag?
- Compare with: the word list, on the same claims.
- Real examples: all claims from last quarter. Two safety engineers mark which ones are safety claims before any AI sees them.
- Enough examples: safety claims are rare. The team needs enough safety claims in the test, not just enough claims.
- Pass mark, written first: the safety lead signs it before the test is opened. The AI replaces the word list only if it clearly catches more safety claims, without burying the engineers in false alarms. That is a go ahead. If it is too close to call, the answer is wait for a person: the word list stays, and the safety lead decides whether to try again. A new try needs a new test set. They do not keep adding examples to the old one until the answer looks right.
- Who decides: the safety lead. The AI never decides about itself.
4. Saying how sure you are
Every score wobbles
A test is a sample. A different set of questions would give a slightly different score.4 So an honest report gives the score and how much it could wobble. One researcher suggests always showing how many questions were used, and how big the wobble is.4 More questions means less wobble.1
Too close to call
Imagine two runners in a race. One crosses the line a hair ahead. Was that runner faster, or just lucky that day? If you only ran the race once, you cannot tell.
AI scores work the same way. In one example with two made-up AIs, the first looked worse on two of three tests. A careful check showed its one win was real, but the two gaps against it were too small to tell apart from luck.4 Reading only the top-line numbers would have picked the wrong one.4
A common mistake: "Two points better is better." It might be. Without the wobble, you cannot know. And a real gain can still be too small to be worth a change. "Too close to call" also does not mean "the same". It means you do not know yet.
You also cannot run the race again and again until your runner wins, then call that the result. One test, one look. If it is too close, a new try needs a new test. Even then, if you try many times, one try may pass by luck. So the team writes down every try, and the person who decides sees them all.
Checking an AI judge
Sometimes there is no answer key. Is this summary good? Is this reply helpful? Teams sometimes ask another AI to be the judge. In 2023, researchers tested AI judges of that time.5 They found habits:
- Some judges liked whichever answer came first.5
- Most judges liked longer answers, even when the extra words added nothing.5
- Judges could mark wrong maths answers as right, even on problems they could solve alone.5
The best judge agreed with expert people about as often as the experts agreed with each other.5 That sounds great. But it agreed less on close calls.5 Close calls are the hardest ones, and the ones you most need a judge for. And when experts saw the AI's reasons, they sometimes changed their own minds.5
So before you trust an AI judge, test it:
- Ask two or more people to mark the same examples on their own, before they see the AI's view.
- Ask the AI judge to mark them too. If it compares two answers, swap the order and ask again.5
- Compare fairly. Check the AI against each person on all the examples. Put that next to how often the people agree with each other, on the same examples.
- Keep the examples the people disagree on. They are the hard ones. Do not throw them out to make the AI look better. Ask a person to settle them.
- Look closely at where the AI and people disagree.
- A named person decides what the AI judge may do (replace people, only sort work for them, or nothing) and writes it down. The AI judge does not decide its own job.
A common mistake: "An AI judge agrees with people." Sometimes it does. You only know for your job if you check.
At Big Widget (made up): two engineers mark 50 AI summaries of claims as "faithful" or not. They agree with each other on 44 of the 50. Checked against each engineer on all 50, the AI judge agrees less often than that. Many of its misses are long summaries it called "faithful". That is a reason to test whether it likes long answers too much; it does not prove it yet. The 6 summaries the engineers disagree on go to the quality lead to settle. Decision: the AI judge may put summaries in order for the engineers, but it may not replace them. That is only this far. The quality lead decides and signs it, not the AI.
5. The three yeses for a test
A test does not say yes. A person does, using the test. The three yeses are a cycle, not a one-time OK.
- Get to Yes: write the test plan first. Say what examples you use, what you compare with, which score, how many examples, and the pass mark. Name the person who decides, and have them sign the plan before the test is opened. Say what you could not check.3 "I don't know yet" is an honest answer. Write down who will find out. A yes covers this AI, this job and this test. Nothing more.
- Stay at Yes: check again after every change to the AI. Use the practice set for everyday checks. A new big decision gets a new test set. Keep a secret test set back, too.1
- Recover to Yes: if a test leaked, say so. Put any decision that relied on it on hold. Throw out the results it affected. Researchers ask for the same: flag results that a leak has spoiled.2 Throwing results out is not the same as being fixed. You are back at yes only when a new test has been built and run, and the person in charge has decided again.
What you keep, and where:
- The work notes stay inside the team: the test plan, the test examples, the results with their wobble, and the judge check.
- The receipt is the shareable record of the decision: what was decided (go ahead, only this far, wait for a person, or no), by whom, and when. It never holds the test examples, the claims, the AI's answers, or anyone's private details. A receipt shows a decision was made. It does not show the AI is good.
Words to know: test set, practice set, compare with, contamination, leak, wobble (uncertainty), too close to call, one test one look, AI judge, not checked, work notes, receipt.