
Testing output before you trust it · Standard gear
Reviewed and published, version 1.0.0. As of 18 Nov 2026. Not a credential. Research results are dated in the text; treat them as "as of" the date given.
You will be able to:
- build a small test set for one task from the user's side (realistic examples with known good answers, including hard ones, made up or approved and redacted) and score the tool's outputs against a written rubric of what a right and usable answer is (200.4-a);
- explain the known biases of using one model to grade another's output (position, length, self-preference) and decide when a person must check instead (200.4-b);
- decide how much checking a task needs from the cost of an error and who it lands on, from spot checks to checking every output (200.4-c).
Big idea: a few good tries tell you the tool can do the job. A test set tells you how often it does. Then the cost of being wrong tells you how much of its work a person still has to check.
Module 100.8 taught you to check one AI answer: its source, its numbers, its sense, and a second person. This module is about checking a tool before you rely on it for a task, and then deciding how much of its everyday output still needs checking. Big Widget Company, its people and its documents in this module are made up.
1. The case: a test nobody ran
Figure data as text
Limits from one study [s1]
| Part | What it means |
|---|---|
| POSITION | Order can change the verdict. The grader may favor one position. |
| LENGTH | More words can sway the grade. Even padding can beat a shorter answer. |
| POSSIBLE SELF-PREFERENCE | Not established by this study. The authors could not tell if the tested models favored their own answers. |
| COURSE PRACTICE | Compare scores with a person’s scores. A person makes the final decision. |
On 1 March 2023, a lawyer filed a court paper in Mata v. Avianca, a case brought by a passenger who said he was hurt on a flight, in the US District Court for the Southern District of New York. The paper cited and quoted court decisions that were said to be published in the Federal Reporter, the Federal Supplement and Westlaw.1 Several of them did not exist. The lawyers later admitted that six of the decisions "were generated by ChatGPT and do not exist".1
The judge's opinion of 22 June 2023 sets out what happened.1
- The writer trusted the tool without testing it. The lawyer who did the research, Steven Schwartz, had not used ChatGPT before. He told the court he had "falsely assumed" it was "like a super search engine".1 He said he was "operating under the false perception that this website [i.e., ChatGPT] could not possibly be fabricating cases on its own."1
- His outside check came back empty. Mr. Schwartz testified that he knew of free websites where a known case citation could be entered and the decision displayed. He entered the citation to "Varghese" "but could not find it". The court found that he knew this before making a submission to the court, "but kept silent".1
- He asked the tool to check itself. He asked ChatGPT "Is Varghese a real case" and "Are the other cases you provided fake". ChatGPT answered that it had supplied "real" authorities that could be found through Westlaw, LexisNexis and the Federal Reporter.1 When he asked is not settled: the court read his May affidavit as suggesting he asked before the 1 March filing or the 25 April affidavit, but his June declaration placed the question after the court's first order to show cause.1
- The signer checked the wrong thing. The lawyer who signed and filed the paper, Peter LoDuca, read it for style: "I was basically looking for a flow, make sure there was nothing untoward or no large grammatical errors." Before he signed, he "did not review any judicial authorities cited in his affirmation."1
The judge did not say that using AI was the problem. He wrote: "there is nothing inherently improper about using a reliable artificial intelligence tool for assistance. But existing rules impose a gatekeeping role on attorneys to ensure the accuracy of their filings."1 The court fined the lawyers and their firm $5,000 and ordered them to send letters to their client and to each judge falsely named as the author of a fake opinion.1 Be precise about why: the judge found bad faith based on "acts of conscious avoidance and false and misleading statements to the Court", and wrote that if they had come clean soon after the fake cases were questioned, "the record now would look quite different."1 So the sanctions were not only for filing the fake cases; they were also for how the lawyers behaved after the cases were questioned.
This one case holds all three lessons of this module.
- Nobody tested the tool on the job before trusting it (section 2).
- The outside check he ran before filing came back empty, and the tool's own word that its cases were real was trusted anyway (section 3).
- The output was a court filing, where one wrong citation lands on the client, the court, the other side and the judges whose names were used. That kind of task needs every item checked (section 4).
The misconception this corrects: "You can judge an AI tool by trying it a few times." ChatGPT's answers looked convincing. Mr. Schwartz said that until the court's first order to show cause, he "still could not fathom that ChatGPT could produce multiple fictitious cases, all of which had various indicia of reliability such as case captions, the names of the judges from the correct locations, and detailed fact patterns and legal analysis that sounded authentic."1 The judge called that claim "highly dubious", and counted this account among his "shifting and contradictory explanations".1 The point here is only how real the fake cases looked, as he described them.
2. Build a small test set and a written rubric
Why a few tries are not enough
When you try a tool a few times, you usually try easy, typical cases, and you judge the answers by how they look. That is this course's point. The US National Institute of Standards and Technology (NIST) makes a broader one about testing generative AI in general. Its 2024 Generative AI Profile says tests "often remain focused on laboratory conditions or restricted to benchmark test datasets" that "may not extrapolate well" to real-world use, and that the gap between benchmarks and real use is likely made worse by "prompt sensitivity and broad heterogeneity of contexts of use".2 Even passing exams written for people does not settle the question: "anecdotal testing of GAI system capabilities through video games or standardized tests designed for humans (e.g., intelligence tests, professional licensing exams) does not guarantee GAI system validity or reliability in those domains." ("GAI" is NIST's short form for generative AI.)2 In plain words: a tool can do well on someone else's test and badly on your task, and small changes in how a request is worded can change the result. A few tries of your own are a test too, only a very small and unplanned one.
What NIST suggests instead is to "Assess the accuracy, quality, reliability, and authenticity of GAI output by comparing it to a set of known ground truth data and by using a variety of evaluation methods (e.g., human oversight and automated evaluation...)".2 "Known ground truth data" means examples where you already know the right answer. That is what a test set is.
A big research study points the same way. In 2022, Percy Liang and colleagues built HELM (Holistic Evaluation of Language Models), which measured language models on 7 kinds of quality (accuracy, calibration, robustness, fairness, bias, toxicity and efficiency) across 16 core scenarios, partly to make sure "metrics beyond accuracy don't fall to the wayside".3 They wrote that "Societally beneficial systems reflect many values, not just accuracy," and that these measures should be evaluated "in the same contexts where we expect to deploy models".3 For you, the "context" is your task, with your kind of input and your reader.
Testing from the user's side
At this level you test from the user's side: you put realistic inputs in, look at what comes out, and score it. You do not need to see inside the tool. If a test fails, you record what went wrong and report it to whoever owns the tool. Tracing which stage inside the tool failed (the source, the search or the writing) is Level 300.
Step 1: write the rubric first
A rubric is a short written list of what a right and usable answer must have. Write it before you run the test, so the tool's polish cannot move the goalposts. Keep it to a few items that a person can mark yes or no. Include:
- Right: the facts that must be correct (names, numbers, part numbers, dates), and the things the answer must never do (invent a cause, a source or a number).
- Usable: what the reader needs (length, format, the one action they must take), and what it should say when the input does not contain the answer ("not stated" rather than a guess).
Mr. LoDuca's check in Mata is a warning: he read for "flow" and "grammatical errors", and never checked that the cases were real.1 A rubric that only scores how the answer reads would have passed those fake cases. Put "every cited source exists and says what the answer says" on the rubric for any task that cites sources.
Step 2: build a small test set
For one task, collect about ten to twenty examples. Each example is an input plus the known good answer, or the facts a good answer must contain, written or approved by someone who knows the work. This course's guidance for the set:
- Realistic. Inputs like the ones the tool will really get, including messy ones.
- Hard ones on purpose. Include cases where a person would slip: two problems in one input, missing information, a number that must be copied exactly, a trick where the obvious answer is wrong, and at least one where the right answer is "the input doesn't say".
- Made up, or approved and redacted. Do not paste real customer, employee or confidential data into a test unless your organisation has approved that use and the data has been redacted. Made-up examples that look like the real thing are often enough (module 200.5 covers data in prompts).
- Kept fixed. Keep the same set, so you can re-run it when the tool, its settings or your prompt change, and compare.
Step 3: run, score and decide
Run every example through the tool the way you will really use it. Score each output against the rubric: pass or fail on each item, with a short note on every failure. Then look at the failures, not just the count. One invented number in a safety notice matters more than five answers that ran a little long.
The result is a short record: the task, the date, the tool and its version or settings, the rubric, the score and the failures. Where you do not know something, such as the tool's version, write "unknown" and who could tell you; do not guess. The record tells you, and the next person, what the tool was tested on and what it was not.
At Big Widget Company
Ana leads a quality team. The team wants to use the company's approved AI tool to draft one-paragraph summaries of quality-defect reports for the weekly review.
Her rubric (written first):
- The part number and the defect are stated exactly as in the report.
- The count of affected units matches the report.
- No cause is given unless the report states one; otherwise it says "cause not yet known".
- It is under 80 words and ends with the action the report asks for.
Her test set (12 made-up reports, each with a known good summary that a senior engineer wrote): eight routine reports; one that mentions two different defects; one with no affected-unit count; one where the technician guesses a cause and marks it as a guess; and one where the part number appears twice with a typo the second time.
The result: the tool passes all eight routine reports. It fails three of the four hard ones: it drops the second defect, it invents a count of "approximately 40 units" and it states the technician's guess as the cause. Trying it a few times on routine reports, Ana would have seen eight good summaries and trusted it. Her record now says what the tool gets wrong, so the team knows what a person must check (section 4).
3. When an AI grades an AI
Scoring outputs by hand takes time, so it is tempting to have a second AI do the grading. Researchers call this LLM-as-a-judge (an LLM is a large language model, the kind of model module 100.1 describes). A 2023 study by Lianmin Zheng and colleagues tested it carefully.4
What they found it can do. Strong judges such as GPT-4 matched human preferences well, "achieving over 80% agreement, the same level of agreement between humans."4 That makes an AI grader a useful helper.
What they found goes wrong. The same paper names the judge's limits: "position, verbosity, and self-enhancement biases," "as well as limited reasoning ability".4 All the numbers below are from models tested in 2023; newer models may do better or worse, so treat them as evidence that the biases exist, not as today's rates.
- Position bias: "Position bias is when an LLM exhibits a propensity to favor certain positions over others."4 The researchers showed judges two answers, then swapped the order and asked again. GPT-4 gave the same verdict both ways in 65.0% of cases; GPT-3.5 in 46.2%; Claude-v1 in 23.8%. They found that "Most LLM judges favor the first position."4
- Length (verbosity) bias: "Verbosity bias is when an LLM judge favors longer, verbose responses," "even if they are not as clear, high-quality, or accurate as shorter alternatives."4 The researchers took 23 answers that contained a numbered list and had them made longer by rephrasing the list "without adding any new information". Claude-v1 and GPT-3.5 judged the padded answer better in 91.3% of the 23 cases; GPT-4 in 8.7%.4 GPT-4 resisted far better, but the authors still concluded that "all LLMs may be prone to verbosity bias".4
- Self-preference (self-enhancement) bias: "the effect that LLM judges may favor the answers generated by themselves."4 In their data, "GPT-4 favors itself with a 10% higher win rate; Claude-v1 favors itself with a 25% higher win rate," though the models "also favor other models and GPT-3.5 does not favor itself." The authors are careful: "Due to limited data and small differences, our study cannot determine whether the models exhibit a self-enhancement bias."4 Treat it as a known risk, not a measured size.
- Limited reasoning: the judge showed "limitations in grading basic math problems which it is capable of solving."4
The researchers also say why people still matter: "While human evaluation is the gold standard for assessing human preferences, it is exceptionally slow and costly."4 An AI grader is a way to save time. It is not a replacement for the gold standard.
In Mata, Mr. Schwartz asked the tool that made the mistake whether its own work was real, and it said yes.1 That is not the bias the study measured, but it has the same flaw: the tool was checking its own work. And the check he ran before filing, a search for "Varghese" on a free website, found nothing.1
Changing the test when the grader is biased
If you use an AI grader, design the test so its biases cannot decide the result.
- Against position bias: run every comparison twice with the order swapped. The researchers' "conservative approach" is to "only declare a win when an answer is preferred in both orders" and, "If the results are inconsistent after swapping, we can call it a tie."4
- Against length bias: give the grader your written rubric and have it mark each item yes or no, rather than asking "which answer is better?" Put length on the rubric as its own item, so a longer answer earns nothing for being long. Then check: if the grader keeps preferring the longer answer, have a person score those cases.
- Against self-preference: do not let a model grade its own output. Use a different model, or a person.
- Against weak reasoning: do not rely on an AI grader for numbers, calculations or facts it cannot look up. A person checks those against the source.
- Check the grader itself. Have a person score a sample of the same outputs and compare. That is how the researchers measured judges: by how often they agreed with people.4 If your grader often disagrees with your people, stop using it for this task.
- Keep the decision with a person. The grader sorts and flags. A named person decides whether the tool is fit for the task.
When a person must check instead
A person, not an AI grader, must do the checking when:
- the answer will be relied on for something whose error is costly or lands on someone else (section 4);
- the check is about facts, sources or numbers that the grader cannot verify (Mata is the example);1
- the grader is the same model as the one being graded;
- a person has not yet confirmed that the grader agrees with people on this task.
The misconception this corrects: "Having a second AI check the answer is as good as a person checking it." A second AI agrees with people often, but not always, and in known, one-sided ways: it can favour the first answer, the longer answer and possibly its own.4 Use it to sort and flag. Let a person decide.
At Big Widget Company
Ana's team tries two prompts for the defect summaries and asks an AI grader which set of summaries is better. The grader prefers prompt B's summaries. Ana notices they are longer. She changes the test: the grader now marks each summary against her four-item rubric (including "under 80 words"), each pair is shown in both orders, and an engineer scores 20 of the summaries by hand to compare. On the rubric, prompt B's summaries break the length rule and add a cause the report did not state. Prompt A wins.
4. How much checking does a task need?
Testing tells you how often a tool goes wrong on a task. It does not tell you what to do about the rest of its output. That depends on what a mistake would cost, and who would pay.
NIST defines risk as "the composite measure of an event's probability (or likelihood) of occurring and the magnitude or degree of the consequences of the corresponding event."2 It says organisations "may additionally wish to allocate risk management resources relative to the severity and likelihood of negative impacts".2 And it suggests that independent evaluations of generative AI be set up so that "the type and robustness of evaluations are proportional to the identified risks."2 Applied to one task at your desk, this course turns that into two questions:
- What does an error cost? Is it a small fix, or money, safety, legal trouble or someone's trust?
- Who does it land on? You, who will notice and fix it? A colleague? Or a customer, a dealer, a regulator or the public, who cannot check it and may act on it?
The Mata opinion lists who paid for the fake citations: "The opposing party wastes time and money in exposing the deception. The Court's time is taken from other important endeavors. The client may be deprived of arguments based on authentic judicial precedents." It also names harm to "the reputation of judges and courts whose names are falsely invoked".1 Almost none of that cost landed on the tool, and most of it landed on people who never saw the draft.
The checking ladder
This course uses four rungs. Pick the lowest rung that fits both answers.
| Rung | What a person does | Fits when |
|---|---|---|
| Spot check | Check a few outputs now and then, and fix what you find | An error is cheap, easy to notice and lands on you or your team |
| Check the risky parts | Check every output, but only the items your test showed it gets wrong (numbers, names, causes) | An error is moderate, and your test set showed where the tool slips |
| Check every output in full | A person checks every output against the source before it is used | An error is costly, or lands on someone outside who will act on it |
| Second person, or don't use it | Every output checked in full and signed off by a second, named person, or the tool is not used for this task | An error could harm safety, health, legal rights or many people |
A filing to a court sits on the top rung, second person: it touches legal rights, and one fake case costs the client, the court, the other side and the judges named. Every citation is checked by a person against the real source, and a second named person signs it off. Placing it there is this course's call; the court's own words are the attorneys' "gatekeeping role".1 In Mata, the second person existed: Mr. LoDuca signed the filing. His check covered style, not the cases.1
Your test set helps you choose. If the tool failed the hard cases on the items that matter, you need a higher rung, or a different tool. If it passed every hard case, you may move down a rung for that task, while you keep re-running the test when the tool changes.
The misconception this corrects: "Every AI output needs the same amount of checking." Checking a lunch order like a legal filing wastes time people should spend on the filing. Checking a legal filing like a lunch order is how Mata happened.
At Big Widget Company
Four tasks, four rungs:
- A joke for the team newsletter. An error is cheap and the team will notice: spot check.
- Ana's defect summaries. An error could send the weekly review the wrong way, and her test showed where the tool slips: check the risky parts. Every summary's part number, count and cause are checked against the report.
- A reply to a customer's warranty claim. An error lands on a customer who will act on it, and it speaks for the company: check every output in full.
- A safety bulletin telling service technicians how to handle a faulty part. An error could hurt someone: second person. A safety engineer checks every line against the engineering report, and a second named person signs it off before it goes out. If the tool cannot reliably produce something that passes, it is not used.
5. The three yeses for testing a tool
The Foundation names three moves: Get to Yes before something runs, Stay at Yes while it runs, and Recover to Yes after something goes wrong.5 A Foundation gate gives one of four answers: "go ahead · only this far · wait for a person · no".5 What follows is this course's application of them to testing a tool, not a quotation.
- Get to Yes (before you rely on the tool for a task). Write the rubric, build the test set, run it and record the failures. Then a named person, not the AI grader, gives one gate answer for this task. Go ahead: it passed, including the hard cases, so use it at the checking rung the task needs. Only this far: it passed some items and failed others, so use it for what it passed and have a person check the rest every time; Ana's defect summaries are here. Wait for a person: the result is unclear, or the grader and your people disagree; name who decides. No: it fails on items that matter and the task sits high on the ladder. Where you do not know something, such as the tool's version or how it was set up, write "unknown" and name who you will ask. The Foundation's Practice site promises: "'I don't know yet' is always an answer, and it always pairs you with someone."6
- Stay at Yes (every day you use it). Check live outputs at the rung you chose. Re-run the same test set whenever the tool, its settings or your prompt change, and compare. One good test run is not a standing approval.
- Recover to Yes (when a wrong output got through). Stop relying on it and tell the tool's owner and the people who acted on it. Add the input that fooled the tool to the test set as a new hard case. Fix the prompt, the checking rung or the tool, then re-run the test before you say "recovered". A pulled output is stopped, not yet recovered.
Keep the evidence, not the secrets. Ana's test record carries the made-up reports, the tool's outputs and her scores. That is a work record. It belongs in Big Widget's own files, under the company's rules for data. The Foundation's signed receipt is a different thing: it "holds no private content".5 It never carries "prompts, outputs, payloads, personal data".6 So the test inputs and the tool's answers stay in the work record, not in a receipt. A receipt can show that a decision was made; it does not show that the tool's answers were right.
6. Bringing it together
My checklist before I trust a tool with a task:
- I wrote a rubric for a right and usable answer before I tested.
- I built a test set of realistic examples with known good answers, including hard ones, made up or approved and redacted.
- I ran and scored every example, and I recorded the failures, the date and the tool.
- If an AI did any grading, I swapped answer order, graded against the rubric, never let a model grade itself, and had a person check a sample.
- I chose how much checking the task needs from what an error costs and who it lands on, and I wrote that down.
- A named person gave the gate answer, and I wrote "unknown" where I did not know.
What you keep. Keep the rubric, the test set and the scores as your work record, with made-up or redacted inputs only. Keep it out of any receipt (section 5). This record is what Level 200's project builds on.
Words to know: test set, known good answer, hard case, rubric, LLM-as-a-judge, position bias, verbosity bias, self-enhancement bias, spot check, checking ladder, gate, receipt.
Further learning (optional, not part of this lesson):
- University of Washington, Carl T. Bergstrom and Jevin D. West, Modern-Day Oracles or Bullshit Machines? No-code lessons on why fluent output is not evidence and how to check it: <https://thebullshitmachines.com/> (free for personal and non-commercial educational use; see the authors' terms).
- Optional, technical (graduate computer science): Carnegie Mellon University, 11-667 Large Language Models: Methods and Applications (Fall 2025), lecture "Automatic evaluation of LLMs": <https://2025.cmu-llms.org/schedule/>.
- Optional, in Mandarin: National Taiwan University, Hung-yi Lee, Introduction to Generative AI and Machine Learning (Fall 2025), lecture on evaluating generative AI: <https://speech.ee.ntu.edu.tw/~hylee/GenAI-ML/2025-fall.php>.