
Checking claims about AI tools · Plain gear
You will be able to: turn a big claim about an AI tool into questions you can check; explain why a tool that says it can replace a lawyer (or any expert) must be tested against that expert; and plan a small, fair trial on your own work before your team starts using a tool.
This module is as of 18 November 2026. Reviewed and published, version 1.0.0. It is not a credential.
Big Widget Company, its people and its tools are made up. So are the sellers' claims about them. Your own workplace's rules come first.
In module 100.8 you learned three questions to ask before you repeat a claim. Now the question is bigger: should your team buy or start using a tool because of what someone says it can do?
1. Break the claim into questions
Claims about AI tools are short and shiny. "Answers 95% of questions right." "Saves four hours a week." "Top of the leaderboard." (We made these up as examples.) You can't check a sentence like that as it is. You have to break it into questions that have real answers.
Think of buying a used car. "Runs great!" is not something you can check. "How many miles? Has it been in a crash? Can my mechanic look at it?" Those you can check.
Here are five questions to ask about any AI claim:
- Which task? What work exactly was the tool doing?
- Measured how? What was counted? Who decided which answers were right?
- Against what? Better than what: the way we do it now, another tool, a person?
- Out of how many? How many tries, people and weeks were in the test?
- By whom? Who ran the test? Do they win if the result looks good?
If nobody can answer one, write "not stated". That shows where the proof stops. Then write who will ask, and by when. Until then, the claim is "not checked". That does not mean it's false. It means nobody knows yet.
Some questions feel like checks but aren't: Is the company famous? Was the demo cool? Do other companies use it? Do we like it? Those can be good to know. But they don't tell you if the tool does the job it promises, on your work.
Why one test score isn't enough
A benchmark is a fixed set of test questions with a way to score them, like a standard exam. Sellers love to quote one.
A big study in 2022, called HELM, tested 30 AI language models. It measured 7 different things, not just one score: for example, how often the model was right, and whether it still worked when the question had small mistakes in it.1 Here is what it found:
- Before HELM, models had often been tested on different tests, so their scores could not be fairly compared.1
- The model that was most often right was not always the best at the other things.1
- One model's score on one test fell from about 73% to about 39% when small changes, like typos, were added to the questions.1 Real work is full of typos.
- Even HELM said its tests left things out.1
The US standards agency NIST says the same about any AI system. Results from a lab can be different from what happens in real life. A tool should be shown to work in conditions like the ones where it will really be used.2
The mistake to avoid: "A high test score means it will work on my job." A test score tells you how the tool did on that test. To know how it does on your work, you test it on your work.
At Big Widget Company
A seller tells the warranty team: "Our assistant answers dealer warranty questions with 95% accuracy!" (A made-up claim.) Sam, the team lead, asks:
- Which task? Easy questions, like dates? Or hard ones, like arguments over a broken part?
- Measured how? Who decided which answers were right?
- Against what? How often are our own experts right today? If it's 97%, then 95% is worse.
- Out of how many? 95% of 40 questions, or 40,000?
- By whom? The seller's own test, or someone else's?
Until there are answers, Sam's note says: "Seller's claim, not checked. Sam will follow up."
2. "Works like a lawyer" needs a test against lawyers
Some tools say they can do an expert's job: a lawyer, an accountant, an engineer. That is a claim that the tool is as good as the expert. The only way to show that is to test the tool and the expert on the same work, and have people who know the job judge the results.
Real case: the "robot lawyer"
DoNotPay sold an AI service. It called itself "the world's first robot lawyer".3
In September 2024, the US Federal Trade Commission (FTC), the US consumer protection agency,4 brought a complaint against the company. The complaint said the company never tested whether its chatbot's work was as good as a human lawyer's. It also said the company did not hire or keep any lawyers itself.3 Remember: a complaint is what the FTC said. It is not something a court decided.
In January 2025, the case was settled with a final order.4 Here is what it says:
- DoNotPay did not admit, and did not deny, what the complaint said.4
- It had to pay $193,000. It also had to email a notice to people who signed up from 2021 to 2023. The notice says the FTC claims DoNotPay did not have enough proof that it works like a human lawyer when it writes demand letters and starts small-claims court cases.4
- From now on, it may not say it works like a lawyer, or like any expert, unless it has real proof. The order says what counts as proof: tests or studies, based on what experts in that field know, done fairly by qualified people, in a way the field accepts.4
So to claim "works like a lawyer", you need a fair test, judged by people who know law. The order lists other expert jobs too, like accounting and engineering.4
The mistake to avoid: "If it's for sale, someone must have checked it." According to the FTC's complaint, DoNotPay itself never ran that test.3 Being on sale only tells you someone is selling it.
At Big Widget Company
A brochure says a new AI tool "does the work of a quality engineer". (Made up.) Dana, the quality manager, asks the seller: Did you test it against real quality engineers, on the same cases? Who judged the answers, and did they know which ones came from the tool? What did it miss?
If there was no such test, the honest claim is smaller: "helps a quality engineer by writing a first draft". That might still be useful. But it is a different promise.
3. Try it fairly before everyone uses it
Figure data as text
Method from one study [s5]
| Part | What it means |
|---|---|
| MEASURE | Issues resolved per hour |
| BASELINE AND COMPARISON | Compare like work over time. Compare with agents’ own earlier work and with agents who did not have the tool. |
| STARTING POINTS | Don’t credit the tool for a head start. The group that got the tool was already resolving more issues per hour. |
The seller's test is about the seller's work. To know what a tool does for you, try it on your work, and decide how you'll judge it before you start. It's like a science-fair experiment: you write down what you'll measure before you run it, not after.
A real example: help for support workers
Researchers studied a chat helper given to customer-support workers at a big software company. The helper was rolled out in 2020 and 2021.5 They did a few smart things:
- They picked one main thing to count: how many customer problems each worker solved per hour.5
- They had a "before" and a group without the tool. Workers got the tool at different times, and some never got it during the study. So the researchers could compare.5
- They noticed who went first. The workers who got the tool were already a bit faster before they had it.5 If you just compared "tool users" with "everyone else", the tool would get credit it didn't earn.
- They looked at who gained. On average, workers solved about 15% more problems per hour. Newer and less skilled workers gained the most. The most skilled workers got a little faster, but the quality of their work dropped a little.5
- They found that people learn from the tool. Sometimes the tool broke and gave no help. Even then, workers who had been using it were faster than before they had it. The tool had taught them things they kept. The researchers say the tool broke only rarely, so these numbers are rough.5
This is one study of one company, with an older tool. It does not mean "AI makes everyone 15% faster." The lesson is how they checked.
NIST also says to test AI before you use it and keep testing while you use it. It says checks by people who aren't part of the project can make testing better.2
Seven steps for a fair trial
These are this course's steps. Write the first five down before anyone uses the tool.
- The claim. Write it as the five questions from section 1.
- The task. Pick real work your team does, the way you really do it.2
- What you'll count. One main thing (like minutes per report), plus one thing that catches harm (like mistakes found by a checker). Don't change them later.
- The "before". Count the same thing, the same way, without the tool.
- Fair groups and two lines. Split the people into two groups by a rule, like drawing names. One group uses the tool and one doesn't. Don't pick the fans. And don't let the same people take turns, one job with the tool and the next without. People learn from the tool, as the study found.5 So their "without" work gets some of the tool's help, and the tool looks worse than it is. Then write two lines:
- The claim line: what result would show the seller's claim is true. If they say "half the time", the time has to drop by about half.
- The "no" line: what result means "no, don't roll it out". A trial that can't say no isn't a real test.
- Who counts. Someone outside the project counts the results.2
- What you report. Whether to roll it out, and whether the seller's claim held. Also: how many people and pieces of work it's based on, who gained and who didn't, and what you couldn't measure.2
The trial doesn't decide. A named person does. They give one of four answers: go ahead, only this far (for example, only for one kind of job), wait for a person (the result is unclear), or no.
You can still ask the team if they like it. Just keep that separate. Liking a tool and the tool saving time are two different things.
The mistake to avoid: "The team loves it, so it works." Loving it matters, because people use tools they like. But whether it saves time or makes better work is answered by what you counted, against the "before".
At Big Widget Company
A seller says its writing tool "cuts the time to write a supplier fix-it request in half". (Made up.) Lee plans a six-week trial:
- Count: minutes from opening the case to sending the request, from the case system's clock. Also, how many requests the boss sends back for mistakes.
- Fair groups: names are drawn to split the team in half. One half uses the tool; the other half doesn't.
- Before: the same counts for both halves, for the three weeks before the trial.
- Claim line, written on day one: the seller is right only if the tool half's time drops by about half.
- "No" line, written on day one: if the time doesn't drop by at least 30%, or mistakes go up, the answer is no.
- Who counts: someone from another team.
- Who decides: Morgan, the manager, after reading the result.
Say the time drops by 35%, and mistakes don't go up. (Made up.) That passes the "no" line, so the team may roll it out. But it is not "half". So the report says: "Good enough for us. The seller's claim was not shown." If the tool doesn't save time at all, this trial will show that too. That's what makes it fair.
4. The three yeses
Checking a claim is one part of saying yes to a tool. It is not the whole thing.
- Get to Yes: ask the five questions. If it says it can replace an expert, ask for the test against that expert. Run your own fair trial, with your two lines written first. Then a named person gives the answer: go ahead, only this far, wait for a person, or no. The yes is for the job you tested.
- Stay at Yes: keep counting after you start using it. NIST says to keep testing while a tool is in use.2 Tools change, so date your results. A new version of the tool needs a new check.
- Recover to Yes: if the tool stops doing what you counted on, or a claim turns out to have no proof, stop relying on it, tell the people who used it, name who will fix it, and change how you check the next claim. Stopping is not the end. You're back at yes only when a new count shows the tool works again.
What you keep. Write down the claim, your questions and answers, your trial plan and the result, in your own work notes. A receipt in this course records the decision, who made it and when. It never holds what you typed into the AI, its answers, or anyone's private details. A receipt shows a decision was made. It doesn't show the seller's claim was true.
Words to know: claim, benchmark (a standard test), baseline (the "before"), fair groups, claim line, "no" line, complaint, not checked.
Want more? (optional, not part of this lesson): University of Washington, Calling Bullshit (<https://callingbullshit.org/>) and Modern-Day Oracles or Bullshit Machines? (<https://thebullshitmachines.com/>).