Level 200: Using gen AI well · Module 200.7

One coworker sets down a notebook while another sips from a mug.
What did they measure?

Checking claims about AI tools · Plain gear

You will be able to: turn a big claim about an AI tool into questions you can check; explain why a tool that says it can replace a lawyer (or any expert) must be tested against that expert; and plan a small, fair trial on your own work before your team starts using a tool.

This module is as of 18 November 2026. Reviewed and published, version 1.0.0. It is not a credential.

Big Widget Company, its people and its tools are made up. So are the sellers' claims about them. Your own workplace's rules come first.

In module 100.8 you learned three questions to ask before you repeat a claim. Now the question is bigger: should your team buy or start using a tool because of what someone says it can do?

1. Break the claim into questions

Claims about AI tools are short and shiny. "Answers 95% of questions right." "Saves four hours a week." "Top of the leaderboard." (We made these up as examples.) You can't check a sentence like that as it is. You have to break it into questions that have real answers.

Think of buying a used car. "Runs great!" is not something you can check. "How many miles? Has it been in a crash? Can my mechanic look at it?" Those you can check.

Here are five questions to ask about any AI claim:

  1. Which task? What work exactly was the tool doing?
  2. Measured how? What was counted? Who decided which answers were right?
  3. Against what? Better than what: the way we do it now, another tool, a person?
  4. Out of how many? How many tries, people and weeks were in the test?
  5. By whom? Who ran the test? Do they win if the result looks good?

If nobody can answer one, write "not stated". That shows where the proof stops. Then write who will ask, and by when. Until then, the claim is "not checked". That does not mean it's false. It means nobody knows yet.

Some questions feel like checks but aren't: Is the company famous? Was the demo cool? Do other companies use it? Do we like it? Those can be good to know. But they don't tell you if the tool does the job it promises, on your work.

Why one test score isn't enough

A benchmark is a fixed set of test questions with a way to score them, like a standard exam. Sellers love to quote one.

A big study in 2022, called HELM, tested 30 AI language models. It measured 7 different things, not just one score: for example, how often the model was right, and whether it still worked when the question had small mistakes in it.1 Here is what it found:

The US standards agency NIST says the same about any AI system. Results from a lab can be different from what happens in real life. A tool should be shown to work in conditions like the ones where it will really be used.2

The mistake to avoid: "A high test score means it will work on my job." A test score tells you how the tool did on that test. To know how it does on your work, you test it on your work.

At Big Widget Company

A seller tells the warranty team: "Our assistant answers dealer warranty questions with 95% accuracy!" (A made-up claim.) Sam, the team lead, asks:

Until there are answers, Sam's note says: "Seller's claim, not checked. Sam will follow up."

2. "Works like a lawyer" needs a test against lawyers

Some tools say they can do an expert's job: a lawyer, an accountant, an engineer. That is a claim that the tool is as good as the expert. The only way to show that is to test the tool and the expert on the same work, and have people who know the job judge the results.

Real case: the "robot lawyer"

DoNotPay sold an AI service. It called itself "the world's first robot lawyer".3

In September 2024, the US Federal Trade Commission (FTC), the US consumer protection agency,4 brought a complaint against the company. The complaint said the company never tested whether its chatbot's work was as good as a human lawyer's. It also said the company did not hire or keep any lawyers itself.3 Remember: a complaint is what the FTC said. It is not something a court decided.

In January 2025, the case was settled with a final order.4 Here is what it says:

So to claim "works like a lawyer", you need a fair test, judged by people who know law. The order lists other expert jobs too, like accounting and engineering.4

The mistake to avoid: "If it's for sale, someone must have checked it." According to the FTC's complaint, DoNotPay itself never ran that test.3 Being on sale only tells you someone is selling it.

At Big Widget Company

A brochure says a new AI tool "does the work of a quality engineer". (Made up.) Dana, the quality manager, asks the seller: Did you test it against real quality engineers, on the same cases? Who judged the answers, and did they know which ones came from the tool? What did it miss?

If there was no such test, the honest claim is smaller: "helps a quality engineer by writing a first draft". That might still be useful. But it is a different promise.

3. Try it fairly before everyone uses it

The study counted issues resolved per hour, comparing agents with their own earlier work and with agents without access. The groups already differed before rollout.
The customer-support study measured issues resolved per hour. Staggered access let the researchers use agents’ earlier work and compare people with and without access. The groups already differed before rollout (s5).
Figure data as text

Method from one study [s5]

PartWhat it means
MEASUREIssues resolved per hour
BASELINE AND COMPARISONCompare like work over time. Compare with agents’ own earlier work and with agents who did not have the tool.
STARTING POINTSDon’t credit the tool for a head start. The group that got the tool was already resolving more issues per hour.

The seller's test is about the seller's work. To know what a tool does for you, try it on your work, and decide how you'll judge it before you start. It's like a science-fair experiment: you write down what you'll measure before you run it, not after.

A real example: help for support workers

Researchers studied a chat helper given to customer-support workers at a big software company. The helper was rolled out in 2020 and 2021.5 They did a few smart things:

This is one study of one company, with an older tool. It does not mean "AI makes everyone 15% faster." The lesson is how they checked.

NIST also says to test AI before you use it and keep testing while you use it. It says checks by people who aren't part of the project can make testing better.2

Seven steps for a fair trial

These are this course's steps. Write the first five down before anyone uses the tool.

  1. The claim. Write it as the five questions from section 1.
  2. The task. Pick real work your team does, the way you really do it.2
  3. What you'll count. One main thing (like minutes per report), plus one thing that catches harm (like mistakes found by a checker). Don't change them later.
  4. The "before". Count the same thing, the same way, without the tool.
  5. Fair groups and two lines. Split the people into two groups by a rule, like drawing names. One group uses the tool and one doesn't. Don't pick the fans. And don't let the same people take turns, one job with the tool and the next without. People learn from the tool, as the study found.5 So their "without" work gets some of the tool's help, and the tool looks worse than it is. Then write two lines:
    • The claim line: what result would show the seller's claim is true. If they say "half the time", the time has to drop by about half.
    • The "no" line: what result means "no, don't roll it out". A trial that can't say no isn't a real test.
  6. Who counts. Someone outside the project counts the results.2
  7. What you report. Whether to roll it out, and whether the seller's claim held. Also: how many people and pieces of work it's based on, who gained and who didn't, and what you couldn't measure.2

The trial doesn't decide. A named person does. They give one of four answers: go ahead, only this far (for example, only for one kind of job), wait for a person (the result is unclear), or no.

You can still ask the team if they like it. Just keep that separate. Liking a tool and the tool saving time are two different things.

The mistake to avoid: "The team loves it, so it works." Loving it matters, because people use tools they like. But whether it saves time or makes better work is answered by what you counted, against the "before".

At Big Widget Company

A seller says its writing tool "cuts the time to write a supplier fix-it request in half". (Made up.) Lee plans a six-week trial:

Say the time drops by 35%, and mistakes don't go up. (Made up.) That passes the "no" line, so the team may roll it out. But it is not "half". So the report says: "Good enough for us. The seller's claim was not shown." If the tool doesn't save time at all, this trial will show that too. That's what makes it fair.

4. The three yeses

Checking a claim is one part of saying yes to a tool. It is not the whole thing.

What you keep. Write down the claim, your questions and answers, your trial plan and the result, in your own work notes. A receipt in this course records the decision, who made it and when. It never holds what you typed into the AI, its answers, or anyone's private details. A receipt shows a decision was made. It doesn't show the seller's claim was true.

Words to know: claim, benchmark (a standard test), baseline (the "before"), fair groups, claim line, "no" line, complaint, not checked.

Want more? (optional, not part of this lesson): University of Washington, Calling Bullshit (<https://callingbullshit.org/>) and Modern-Day Oracles or Bullshit Machines? (<https://thebullshitmachines.com/>).

Sources

  1. s3 Percy Liang et al. (2022). Holistic Evaluation of Language Models. Paper. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, session separate from the writer), 18 Nov 2026. ↩ ↩ ↩ ↩ ↩
  2. s4 Elham Tabassi (National Institute of Standards and Technology) (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). Standard. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, fresh session separate from the writer), round 2 re-check, 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩
  3. s1 Federal Trade Commission (2024). FTC Announces Crackdown on Deceptive AI Claims and Schemes (Operation AI Comply), archived copy. Official doc. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, session separate from the writer), 18 Nov 2026; confirmed on the live FTC page at live_url, because the fetch tool refused the web.archive.org url (see checks/source-check.md). ↩ ↩ ↩
  4. s2 Federal Trade Commission (2025). In the Matter of DoNotPay, Inc., Decision and Order, Docket No. C-4812. Record. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, fresh session separate from the writer), round 2 re-check, 18 Nov 2026; read in full from the same PDF on www.ftc.gov (/system/files/ftc_gov/pdf/2323042_donotpay_decision_and_order_0.pdf), because the search.ftc.gov url returned no response this session (see checks/source-check.md). ↩ ↩ ↩ ↩ ↩ ↩
  5. s5 Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond (2023). Generative AI at Work. Paper. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, fresh session separate from the writer), round 2 re-check, 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩

Next: the quiz