
Checking claims about AI tools · Standard gear
You will be able to: turn a vendor's or a headline's claim about an AI tool into testable questions (which task, measured how, against what, out of how many, by whom); explain, using the US Federal Trade Commission's (FTC) DoNotPay action, why a claim that a tool can replace a professional needs evidence of testing against that professional; and plan a small pilot on your own work, with a baseline and a measure chosen in advance, before a tool is rolled out.
Facts about particular cases, studies and rules are dated in the text. This module is as of 18 November 2026. Reviewed and published, version 1.0.0. Not a credential.
Big Widget Company, its people, its tools and the vendor claims in this module are made up. Your own employer's rules come first.
Module 100.8 gave you three questions for any claim about AI: out of how many, measured how, and over how long. This module takes the next step. You are no longer only deciding whether to repeat a claim. You are deciding whether your team should buy, roll out or rely on a tool because of one.
1. Turn the claim into questions you can test
A claim about an AI tool usually arrives as one sentence: "answers 95% of questions correctly", "saves each engineer four hours a week", "scores top of the leaderboard". (These three are made-up examples.) A sentence like that cannot be tested as it stands. Before anyone can check it, it has to be broken into questions that have answers.
Use five questions. They are this course's checklist, not an outside standard.
| Question | What you are asking | Example of a good answer (made up) |
|---|---|---|
| Which task? | What work exactly was the tool doing? | "Drafting replies to dealer warranty emails in English" |
| Measured how? | What was counted, and who decided what counts as right? | "Minutes from opening the email to sending; replies marked right or wrong by a warranty specialist" |
| Against what? | Compared with what: the old way, another tool, a person? | "Half the team, chosen by drawing names, doing the same kind of email without the tool, in the same weeks" |
| Out of how many? | How many cases, people and weeks were in the test? | "1,200 emails, 30 people, eight weeks" |
| By whom? | Who ran the test, and do they gain if the result is good? | "Our own quality team, not the vendor" |
If a claim cannot answer a question, write that down. "Not stated" is an answer, and it tells you where the evidence stops. Pair it with a next step: who will ask, and by when. Until then the claim is "vendor says, not checked", not "true" and not "false".
Some questions sound like checks but test nothing about the claim: Is the vendor well known? How good did the demo look? Do other companies use it? Does our team like it? The answers may be worth knowing, but none of them tells you whether the tool does the stated task, to the stated level, on your work.
Why one benchmark score is not your answer
A benchmark is a fixed set of test questions with a scoring rule. Vendors often quote one. A 2022 study, Holistic Evaluation of Language Models (HELM), shows why one score is not enough.
- Different tests, different results. HELM measured 30 language models on 7 things, not one: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency, across 16 core test scenarios and 26 more targeted ones. Before HELM, models had on average been tested on only 17.9% of its core scenarios, "with some prominent models not sharing a single scenario in common".1 Two vendors' headline scores may not even come from the same test.
- The best on one measure is not always the best on another. HELM found that accuracy, robustness and fairness mostly went together, but it also saw "trade-offs where the most accurate model is not the most robust or most fair".1
- Small changes to the input can move the score a lot. On one of HELM's scenarios, NarrativeQA, one model's accuracy dropped from 72.6% to 38.9% when HELM made small changes to the inputs, such as typos.1 Real work is full of typos, shorthand and half-finished sentences.
- Even a broad test leaves things out. HELM chose its scenarios for coverage and feasibility, "noting what's missing or underrepresented".1 No benchmark covers your work unless it was built from your work.
The US National Institute of Standards and Technology (NIST) makes the same point for any AI system. Its AI Risk Management Framework (AI RMF 1.0, January 2023) says that measuring in "a laboratory or a controlled environment may yield important insights pre-deployment", but "these measurements may differ from risks that emerge in operational, real-world settings".2 It asks that performance be "demonstrated for conditions similar to deployment setting(s)", and that the limits of how far results carry "beyond the conditions under which the technology was developed" be written down.2
The misconception this corrects: "A benchmark score tells me how the tool will do on my work." A benchmark score tells you how a model did on that benchmark, scored that way, at that time. It is a reason to test the tool on your work, not a replacement for doing so.
At Big Widget Company
A vendor tells the warranty team: "Our assistant answers dealer warranty questions with 95% accuracy, and it tops a public leaderboard." (A made-up claim.) Sam, the warranty lead, turns it into questions:
- Which task? Answering which questions: simple coverage dates, or disputed claims with photos? In which languages? For widgets like ours?
- Measured how? What counts as "accurate"? Who marked the answers, and did they know the right answer for each one? Is the leaderboard score about warranty questions at all?
- Against what? 95% compared with what our own specialists get today? If they already get 97% right, "95%" would be a step back.
- Out of how many? 95% of 40 questions, or of 40,000? Asked over how long?
- By whom? The vendor's own test, or an independent one? Can we see the test set?
Until those have answers, Sam's note says: "Vendor claim, not checked. Questions sent 18 Nov; Sam owns the follow-up."
2. "Replaces a professional" is a claim about a comparison
Some claims say a tool can do the work of a professional: a lawyer, an accountant, an engineer. That is a comparison claim. It says the tool's work is as good as the professional's. A comparison claim needs evidence from a comparison: the tool and the professional tested on the same work, with the results judged by people qualified to judge them.
Real case: DoNotPay's "robot lawyer"
Keep three things apart in this case: what the FTC's complaint alleged, what was proposed to settle it, and what the final order requires.
What the complaint alleged (September 2024). On 25 September 2024, the FTC announced a set of cases against deceptive AI claims, which it called Operation AI Comply. One was against DoNotPay, a company that, the FTC said, claimed to offer "the world's first robot lawyer".3 The FTC's release says: "The complaint alleges that the company did not conduct testing to determine whether its AI chatbot's output was equal to the level of a human lawyer", and that "the company itself did not hire or retain any attorneys".3 These are allegations: what the FTC said, not facts found by a court. The release also described a proposed order: DoNotPay would pay $193,000 and notify consumers who subscribed between 2021 and 2023.3
What the final order says (January 2025). The FTC issued its Decision and Order (Docket No. C-4812) on 14 January 2025.4
- DoNotPay "neither admits nor denies any of the allegations in the Complaint", apart from the facts needed to give the FTC jurisdiction.4 So the order settles the case. It does not prove the complaint's allegations true.
- DoNotPay must pay $193,000 and email a notice to the new subscribers it can identify who joined between 1 January 2021 and 31 December 2023.4 The notice, written into the order, says: "the FTC says we did not have sufficient proof for our claims that DoNotPay operates like a human lawyer when it generates demand letters and initiates cases in small claims court".4 It is the FTC's claim, reported in DoNotPay's notice, not an admission.
- Most useful for this module is what the order says about the future. DoNotPay must not claim that its product "operates like a human lawyer", or that it "performs a Professional Service like a professional in the relevant field", unless it has "competent and reliable evidence" that the claim is true.4 The order defines that evidence as "tests, analyses, research, studies, or other evidence based on the expertise of professionals in the relevant area, that (1) have been conducted and evaluated in an objective manner by qualified persons and (2) are generally accepted in the profession to yield accurate and reliable results."4
Read that definition as a checklist. To say "works like a lawyer", you need a test, judged by people who know law, done in a way the profession would accept. The order's list of professional services is not only law: it also names accounting, engineering, consulting, research and translation, among others.4 This order is about DoNotPay, not other companies. But its definition is a clear statement of what evidence for a "replaces a professional" claim looks like.
The misconception this corrects: "If a product is on sale, someone has checked that it works as claimed." In this case, according to the complaint, the company itself had not tested whether its output was equal to a human lawyer's.3 The FTC's complaint came in 2024, and the customers who were to be notified had joined from 2021 to 2023.34 A product being on sale tells you that someone is selling it. It does not tell you that anyone has tested the claim.
At Big Widget Company
A vendor offers an AI tool that, its brochure says, "does the work of a quality engineer". (A made-up claim.) Before anyone budgets for it, Dana, the quality manager, asks the vendor:
- Did you test the tool against quality engineers, on the same cases?
- Which cases, how many, and who judged whose answers were right? Were the judges qualified quality engineers, and did they know which answers came from the tool?
- What did the tool get wrong that an engineer would have caught?
If the vendor has no such test, the claim is not "works like a quality engineer". At most it is "drafts material for a quality engineer to check". That may still be useful. It is a different claim, with a different price.
3. Plan a small pilot before you roll a tool out
Figure data as text
Method from one study [s5]
| Part | What it means |
|---|---|
| MEASURE | Issues resolved per hour |
| BASELINE AND COMPARISON | Compare like work over time. Compare with agents’ own earlier work and with agents who did not have the tool. |
| STARTING POINTS | Don’t credit the tool for a head start. The group that got the tool was already resolving more issues per hour. |
A vendor's evidence is about the vendor's test. The only way to know what a tool does on your work is to try it on your work, and to plan how you will judge it before you start.
A measured example: AI help for customer-support agents
Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied a chat assistant, built on OpenAI's GPT-3, given to customer-support agents at a Fortune 500 company that sells business-process software. Their paper was first released in 2023 and published in The Quarterly Journal of Economics in 2025. The published version covers 5,172 agents (the 2023 working paper gave 5,179).5 It is useful here less for its result than for how it was measured.
- One main measure, defined clearly. The main measure was issues resolved per hour. The researchers also tracked its parts (time per chat, chats per hour, share resolved) and customer satisfaction surveys.5
- A baseline and a comparison. The tool reached agents at different times, mostly in late 2020 and early 2021, because training sessions and the budget were limited. Some agents never got it during the study. That let the researchers compare each agent with their own record before the tool, and with agents who did not have it.5
- A warning about who goes first. The agents who got the tool were already doing better before they had it: 2.0 issues resolved per hour, against 1.7 for agents who never got it.5 A simple "tool users versus everyone else" comparison would have given the tool credit for that head start.
- An average hides a spread. In the published paper, access to the tool raised issues resolved per hour by 15% on average (the 2023 working paper said 14%). Less experienced and less skilled agents gained about 30%. The most experienced and highest-skilled agents saw "small gains in speed and small declines in quality".5
- What people learn from a tool stays with them. The researchers "find evidence that AI assistance facilitates worker learning". During software outages, when the tool gave no suggestions, agents who had been using it still handled chats faster than they had before they had it. Because outages were rare, the authors call these estimates "noisy".5 This matters for how you design a pilot (step 5 below).
This is one study, of one company's support agents, with a tool rolled out in 2020 and 2021. Do not repeat "AI makes workers 15% faster" as a general fact. The point is the method: a measure, a baseline, a comparison, and a look at who gained and who did not.
NIST asks for the same kind of discipline when AI systems are tested: AI systems "should be tested before their deployment and regularly while in operation", test sets and metrics should be documented, and test processes should be "objective, repeatable, or scalable", followed and documented.2 It also says that "Processes for independent review can improve the effectiveness of testing and can mitigate internal biases and potential conflicts of interest", and that what cannot be measured should be written down too.2
Seven steps for a small pilot
These steps are this course's own plan, drawn from the ideas above. Write the first five down before anyone uses the tool.
- The claim, as questions. Write the claim you are testing, in the five-question form from section 1.
- The task. Choose real work your team does, in the conditions it is really done in: your documents, your systems, your deadlines.2
- The measure. Choose one main measure (for example, minutes per report) and at least one guard measure, so you can see if speed costs quality (for example, errors found by a checker). Say exactly how each is counted. Do not change them after you see the results.
- The baseline. Measure the same work, the same way, without the tool, first or in the same weeks.
The comparison, the claim line and the decision rule. Split the people, not the work: decide by a rule (drawing names, a coin) which people use the tool and which do not, and do not pick the keenest. Why not let the same people take turns, one case with the tool and the next without? Because what people learn from a tool stays with them when they work without it, as the study above found.5 Their "no tool" cases then pick up part of the tool's effect, and the measured saving looks smaller than it is. (People in the two groups can still share tips, so even a split by people is not perfectly clean. Write that down as a limit.) Then write two lines:
- The claim line: the result that would show the vendor's claim holds. If the claim is "halves the time", that is a fall of about half.
- The "no" line: the result that would mean you do not roll it out, for example "if minutes per report do not fall by at least a fifth, or errors rise, we do not roll it out." A pilot that cannot say no is not a test.
The two lines can differ. A tool can be worth rolling out without doing what the vendor claimed.
- Who measures. Someone who did not choose the tool and does not run the project counts the results.2
- What you report. The result against the decision rule and against the vendor's claim, how many people and pieces of work it rests on, who gained and who did not, and what you could not measure.2 A small pilot can show a large gain, or a clear lack of one. It rarely proves a small gain. Say which you found.
The pilot's result is evidence for a decision. It is not the decision. A named person, not the tool and not the vendor, gives one of four answers: go ahead (roll it out for the task you tested), only this far (roll it out for a narrower task, or with a limit), wait for a person (the result is unclear, so it waits for a named person to decide what more is needed), or no. (These are course rules.)
You can still ask the team what they think of the tool. Keep it as a separate question. A survey of how people feel about a tool measures how they feel. It does not measure the time or the errors.
The misconception this corrects: "If the team likes the tool, it is working." Liking a tool is worth knowing, and it matters for whether people will use it. Whether the tool saves time or improves the work is a different question, answered by the measure you chose in advance against the baseline.
At Big Widget Company
The vendor of a drafting tool claims it "halves the time to write a supplier corrective-action request". (A made-up claim.) Lee, in supplier quality, plans a six-week pilot:
- Task: real corrective-action requests from the team's own queue.
- Main measure: minutes from opening the case to sending the request, from the case system's own timestamps. Guard measure: requests sent back by the supplier quality lead for errors.
- Comparison: names are drawn to split the team in half. One half uses the tool for the six weeks; the other half does not.
- Baseline: the same two measures for both halves' requests in the three weeks before the pilot. The result is how much more the tool half's time fell than the other half's.
- Claim line (written on day one): the vendor's claim holds only if median minutes for the tool half fall by about half, compared with the other half.
- Decision rule (written on day one): roll out only if median minutes fall by at least 30% and send-backs do not rise. Otherwise, no.
- Who measures: an analyst outside the supplier quality team.
- Who decides: Morgan, the supplier quality manager, gives the answer after reading the analyst's report.
- Known limits, written down: the two halves sit in the same office and may share tips; six weeks may be too short to see effects that take longer to appear.
Suppose the time falls by 35% and send-backs stay flat. (A made-up result.) That passes the roll-out line, but it does not confirm the vendor's claim. The report says both: "Meets our roll-out rule. Vendor's 'halves the time' claim not confirmed: we measured about a third." If the time does not fall, this design can show that too. That is what makes it a test.
4. The three yeses for claims about tools
The Foundation's three yeses are a cycle, not a one-time approval. Here they are applied to claims about AI tools. Checking a claim is one part of a yes, not all of it: the data, security and policy gates in the other modules still apply.
- Get to Yes: before roll-out, turn the vendor's claim into the five questions. If the claim says the tool works like a professional, ask for the test against that professional. Run your own pilot, with the measure, the claim line and the "no" line written down first. Then a named person gives the answer: go ahead, only this far, wait for a person, or no. A yes covers the task you tested, not every task the vendor names.
- Stay at Yes: keep measuring after roll-out, with the same measure. NIST asks for testing "regularly while in operation".2 Tools, models and vendors change; date every result. A yes given for one version of a tool is not a yes for the next one until someone checks.
- Recover to Yes: if the tool stops meeting the measure, or a claim you relied on turns out to be unsupported, stop relying on it for that task, tell the people who used the claim, name who owns the fix, and change how the next claim is checked. Stopping is not yet recovery. You are back at yes only when a new measurement, checked by someone outside the project, shows the tool meets the rule again.
What you keep. Keep the claim, the five questions with their answers, your pilot plan and the result in your own controlled work record. In this course, a receipt records the decision, who made it and when. It never holds prompts, the tool's answers, customer details or anyone's private information. A receipt shows that a decision was made. It does not show that the vendor's claim was true.
Words to know: claim, benchmark, baseline, comparison group, guard measure, claim line, decision rule, allegation, consent order, substantiation, not checked.
Further learning (optional, not part of this lesson): University of Washington, Calling Bullshit: Data Reasoning in a Digital World: <https://callingbullshit.org/>; Carl T. Bergstrom and Jevin D. West (University of Washington), Modern-Day Oracles or Bullshit Machines?: <https://thebullshitmachines.com/>; Stanford University, CS324 Large Language Models, Capabilities lecture (Winter 2022, technical): <https://stanford-cs324.github.io/winter2022/lectures/capabilities/>.