Level 200: Using gen AI well · Module 200.4

Three coworkers compare small cards spread across a café table.
Try the awkward one

Testing output before you trust it · Plain gear

Reviewed and published, version 1.0.0. As of 18 Nov 2026. Not a credential.

In this lesson you will learn to:

Big Widget Company and everyone in it are made up for this course.

1. The lawyer who never tested the tool

An AI grader may favor a position or a longer answer. The study did not establish self-preference. Compare with a person’s scores and leave the decision with a person.
Zheng and colleagues documented position and length effects and did not determine whether the models tested favored their own answers (s1). The course keeps the decision with a person.
Figure data as text

Limits from one study [s1]

PartWhat it means
POSITIONOrder can change the verdict. The grader may favor one position.
LENGTHMore words can sway the grade. Even padding can beat a shorter answer.
POSSIBLE SELF-PREFERENCENot established by this study. The authors could not tell if the tested models favored their own answers.
COURSE PRACTICECompare scores with a person’s scores. A person makes the final decision.

In 2023, a New York lawyer named Steven Schwartz needed court cases to support his client's argument. He had never used ChatGPT before. He thought it was "like a super search engine". So he asked it for cases, and it gave him some. They looked real. They had names, judges and details.1

He put them in a court paper. On 1 March 2023, another lawyer at his firm, Peter LoDuca, signed it and filed it.1

The cases were fake. The tool had made them up. The lawyers later admitted that six of the cases "were generated by ChatGPT and do not exist".1

Look at the checks that happened:

The judge said that using a reliable AI tool is not wrong in itself. The problem was that lawyers must make sure what they file is true.1 On 22 June 2023, the court fined the lawyers and their firm $5,000. It made them write to their client and to every real judge whose name was put on a fake case.1 The judge also said things would have gone much better if they had owned up quickly once the cases were questioned. They did not.1

This story has three lessons. Nobody tested the tool first. The real check he ran came back empty, and the tool's word was trusted anyway. And a court paper is the kind of job where every single fact needs checking.

2. Test the tool before you trust it

The mistake: "I tried it a few times and it was fine, so it works."

Think about a new cook at a restaurant. You would not hire them because they made you one nice sandwich. You would ask them to make the dishes on your menu, including the hard ones. Then you would taste each one.

A US standards body, NIST, makes a similar point about AI. It says tests done in a lab, or on ready-made test questions, "may not extrapolate well": they may not hold up in real use.2 It also warns that good scores on tests made for people, like exams, do not prove an AI is reliable at that kind of work.2 Your own few tries are a test too, just a tiny one. So this lesson asks for more.

NIST suggests comparing the tool's answers with "a set of known ground truth data".2 That means examples where you already know the right answer.

A big research project called HELM argues that AI should be measured on more than being right, and in the same kind of situation where it will be used.3 For your job, that means: the answer must be right and useful to the person who reads it.

So here is how you test a tool for one job.

Step 1. Write down what a good answer looks like. Do this first. This list is called a rubric. Keep it short. Each line should be a yes-or-no question:

Write the rubric before you see any answers. Otherwise a smooth, confident answer can talk you into lowering the bar. Mr. LoDuca checked the grammar, not the facts.1 A good rubric checks the facts.

Step 2. Gather ten to twenty examples. For each one, write down the right answer, or have an expert write it. Make them like the real job. And add some hard ones on purpose:

Use made-up examples, or real ones that your company has approved with private details removed. Never paste private customer or staff information into a test just because it's handy.

Step 3. Run every example and mark every answer. Use your rubric. Write down what went wrong each time, not just the score. Keep the examples, so you can test again when the tool changes.

Write down the date and which tool you used. If you don't know something, like the tool's version, write "not known". Don't guess.

At this level, you test from the outside: you put things in and look at what comes out. Finding out where inside the tool things went wrong comes in Level 300.

At Big Widget Company

Ana runs a quality team. Her team wants the company's AI tool to write short summaries of reports about faulty parts.

First, Ana writes her rubric. The part number must be exact. The number of faulty parts must be right. The summary must not guess a cause. It must be short and end with the next action.

Then she makes 12 practice reports, with a summary for each written by a senior engineer. Eight are normal. Four are hard: one has two faults; one leaves out how many parts were faulty; one has a technician's guess about the cause; one has a typo in the part number.

The tool gets all eight normal ones right. But it gets three of the four hard ones wrong. It leaves out the second fault. It makes up a number of faulty parts. And it says the technician's guess is the cause.

If Ana had just tried it a few times on normal reports, she would have trusted it. Now she knows exactly what to watch for.

3. Can another AI do the checking?

The mistake: "If a second AI checks the answer, that's as good as a person checking it."

Checking takes time. So people often ask a second AI to mark the first AI's work. Researchers studied this in 2023.4 They found that a strong AI checker agreed with people quite often: as often as people agree with each other.4 That makes it a useful helper.

But they also found that AI checkers have blind spots.4 These were AI models from 2023, and newer ones may be different. The blind spots are still worth knowing.

The researchers called people "the gold standard" for judging answers. People are just slower and cost more.4

Remember Mr. Schwartz. He asked ChatGPT whether its own cases were real, and it said yes.1 That is a tool checking its own work. It is not a check.

If you do use an AI checker, make the test fair:

The AI checker sorts and flags. A person decides.

A person must check when: a mistake would be costly or would hurt someone else; the check is about facts, sources or numbers; or nobody has yet shown that the AI checker agrees with people on this job.

At Big Widget Company

Ana tries two ways of asking for summaries. She asks an AI checker which set is better. It picks set B. Ana notices that set B's summaries are longer. So she changes the test. The AI checker now marks each summary against her rubric, which says "short". It sees each pair in both orders. And an engineer marks 20 summaries by hand. Now set B loses: its summaries are too long, and they guess at causes.

4. How much checking does the job need?

The mistake: "Every AI answer needs the same amount of checking."

Think about two things you might check before you leave home. Did you pack a snack? Did you turn off the stove? You don't check the snack twice. You might check the stove twice. A mistake with the stove costs much more, and it can hurt other people.

NIST says groups may choose to put more of their effort where harm is more likely and would be worse.2 This course turns that into two questions about any job:

  1. What would a mistake cost? A small fix? Or money, safety, trust or legal trouble?
  2. Who would it land on? You, who will spot it? Or a customer, or the public, who can't check it and might act on it?

In Mata, the fake cases cost the other side time and money. They took up the court's time. They could have hurt the client's case. And they put real judges' names on fake decisions.1 Most of that landed on people who never saw the AI's answers.

Here is a ladder of checking, from least to most:

  1. Spot check. Look at a few answers now and then. For cheap mistakes that land on you.
  2. Check the risky parts. Check the parts your test showed the tool gets wrong, every time.
  3. Check everything. A person checks every answer before it's used. For costly mistakes, or ones that land on other people.
  4. Two people, or don't use AI. Every answer is checked, and a second named person signs it off. Or the tool is not used for this job. For anything about safety, health, the law or many people.

A court paper is on rung 4, the top. It is about the law, and mistakes land on many people. A person looks up every case. Then a second named person signs it off. In Mata, that second person was there: the lawyer who signed. But he checked the grammar, not the cases.1

At Big Widget Company

5. The three yeses

The Foundation names three moves: Get to Yes, Stay at Yes and Recover to Yes.5 A Foundation gate gives one of four answers: "go ahead · only this far · wait for a person · no".5 Here is how this course uses them for testing a tool.

Keep the evidence, not the secrets. Ana's test record holds the made-up reports, the tool's answers and her marks. That is a work record. It stays in her team's own files. The Foundation's signed receipt is a different thing: it "holds no private content".5 It never carries "prompts, outputs, payloads, personal data".6 So the examples and the answers stay in the work record, not in a receipt.

6. What to remember

Words to know: test set, rubric, hard case, AI checker, spot check, checking ladder, gate, receipt.

Want to learn more? (optional) The University of Washington's free lessons Modern-Day Oracles or Bullshit Machines? by Carl T. Bergstrom and Jevin D. West explain why smooth AI answers are not proof: <https://thebullshitmachines.com/> (free for personal and non-commercial educational use; see the authors' terms).

Sources

  1. s4 P. Kevin Castel, U.S. District Court for the Southern District of New York (2023). Mata v. Avianca, Inc., No. 22-cv-1461 (PKC), Opinion and Order on Sanctions (docket entry 54). Record. Checked 2026-11-18 by source-checker role, Devin session f9ae423f (did not write, revise or red-team this module), remaining-sources check, 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  2. s3 Chloe Autio et al. (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). Standard. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the writer and reviser), 18 Nov 2026. ↩ ↩ ↩ ↩
  3. s2 Percy Liang et al. (2022). Holistic Evaluation of Language Models. Paper. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the writer and reviser), 18 Nov 2026. ↩
  4. s1 Lianmin Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Paper. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the writer and reviser), 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  5. s5 AiGovOps Foundation (n.d.). AiGovOps Foundation Manifesto. Official doc. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the writer and reviser), 18 Nov 2026; page opened 18 Nov 2026, labelled "Fall 2026 Release", no numeric version or date shown. ↩ ↩ ↩
  6. s6 AiGovOps Foundation (n.d.). AiGovOps Foundation Practice (home page). Official doc. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the writer and reviser), 18 Nov 2026; page opened 18 Nov 2026, footer "practice v1.0.0-rc.1", "Fall 2026 Release". ↩

Next: the quiz