Level 200: Using gen AI well · Module 200.7

One coworker sets down a notebook while another sips from a mug.
What did they measure?

Checking claims about AI tools · Standard gear

You will be able to: turn a vendor's or a headline's claim about an AI tool into testable questions (which task, measured how, against what, out of how many, by whom); explain, using the US Federal Trade Commission's (FTC) DoNotPay action, why a claim that a tool can replace a professional needs evidence of testing against that professional; and plan a small pilot on your own work, with a baseline and a measure chosen in advance, before a tool is rolled out.

Facts about particular cases, studies and rules are dated in the text. This module is as of 18 November 2026. Reviewed and published, version 1.0.0. Not a credential.

Big Widget Company, its people, its tools and the vendor claims in this module are made up. Your own employer's rules come first.

Module 100.8 gave you three questions for any claim about AI: out of how many, measured how, and over how long. This module takes the next step. You are no longer only deciding whether to repeat a claim. You are deciding whether your team should buy, roll out or rely on a tool because of one.

1. Turn the claim into questions you can test

A claim about an AI tool usually arrives as one sentence: "answers 95% of questions correctly", "saves each engineer four hours a week", "scores top of the leaderboard". (These three are made-up examples.) A sentence like that cannot be tested as it stands. Before anyone can check it, it has to be broken into questions that have answers.

Use five questions. They are this course's checklist, not an outside standard.

QuestionWhat you are askingExample of a good answer (made up)
Which task?What work exactly was the tool doing?"Drafting replies to dealer warranty emails in English"
Measured how?What was counted, and who decided what counts as right?"Minutes from opening the email to sending; replies marked right or wrong by a warranty specialist"
Against what?Compared with what: the old way, another tool, a person?"Half the team, chosen by drawing names, doing the same kind of email without the tool, in the same weeks"
Out of how many?How many cases, people and weeks were in the test?"1,200 emails, 30 people, eight weeks"
By whom?Who ran the test, and do they gain if the result is good?"Our own quality team, not the vendor"

If a claim cannot answer a question, write that down. "Not stated" is an answer, and it tells you where the evidence stops. Pair it with a next step: who will ask, and by when. Until then the claim is "vendor says, not checked", not "true" and not "false".

Some questions sound like checks but test nothing about the claim: Is the vendor well known? How good did the demo look? Do other companies use it? Does our team like it? The answers may be worth knowing, but none of them tells you whether the tool does the stated task, to the stated level, on your work.

Why one benchmark score is not your answer

A benchmark is a fixed set of test questions with a scoring rule. Vendors often quote one. A 2022 study, Holistic Evaluation of Language Models (HELM), shows why one score is not enough.

The US National Institute of Standards and Technology (NIST) makes the same point for any AI system. Its AI Risk Management Framework (AI RMF 1.0, January 2023) says that measuring in "a laboratory or a controlled environment may yield important insights pre-deployment", but "these measurements may differ from risks that emerge in operational, real-world settings".2 It asks that performance be "demonstrated for conditions similar to deployment setting(s)", and that the limits of how far results carry "beyond the conditions under which the technology was developed" be written down.2

The misconception this corrects: "A benchmark score tells me how the tool will do on my work." A benchmark score tells you how a model did on that benchmark, scored that way, at that time. It is a reason to test the tool on your work, not a replacement for doing so.

At Big Widget Company

A vendor tells the warranty team: "Our assistant answers dealer warranty questions with 95% accuracy, and it tops a public leaderboard." (A made-up claim.) Sam, the warranty lead, turns it into questions:

Until those have answers, Sam's note says: "Vendor claim, not checked. Questions sent 18 Nov; Sam owns the follow-up."

2. "Replaces a professional" is a claim about a comparison

Some claims say a tool can do the work of a professional: a lawyer, an accountant, an engineer. That is a comparison claim. It says the tool's work is as good as the professional's. A comparison claim needs evidence from a comparison: the tool and the professional tested on the same work, with the results judged by people qualified to judge them.

Real case: DoNotPay's "robot lawyer"

Keep three things apart in this case: what the FTC's complaint alleged, what was proposed to settle it, and what the final order requires.

What the complaint alleged (September 2024). On 25 September 2024, the FTC announced a set of cases against deceptive AI claims, which it called Operation AI Comply. One was against DoNotPay, a company that, the FTC said, claimed to offer "the world's first robot lawyer".3 The FTC's release says: "The complaint alleges that the company did not conduct testing to determine whether its AI chatbot's output was equal to the level of a human lawyer", and that "the company itself did not hire or retain any attorneys".3 These are allegations: what the FTC said, not facts found by a court. The release also described a proposed order: DoNotPay would pay $193,000 and notify consumers who subscribed between 2021 and 2023.3

What the final order says (January 2025). The FTC issued its Decision and Order (Docket No. C-4812) on 14 January 2025.4

Read that definition as a checklist. To say "works like a lawyer", you need a test, judged by people who know law, done in a way the profession would accept. The order's list of professional services is not only law: it also names accounting, engineering, consulting, research and translation, among others.4 This order is about DoNotPay, not other companies. But its definition is a clear statement of what evidence for a "replaces a professional" claim looks like.

The misconception this corrects: "If a product is on sale, someone has checked that it works as claimed." In this case, according to the complaint, the company itself had not tested whether its output was equal to a human lawyer's.3 The FTC's complaint came in 2024, and the customers who were to be notified had joined from 2021 to 2023.34 A product being on sale tells you that someone is selling it. It does not tell you that anyone has tested the claim.

At Big Widget Company

A vendor offers an AI tool that, its brochure says, "does the work of a quality engineer". (A made-up claim.) Before anyone budgets for it, Dana, the quality manager, asks the vendor:

If the vendor has no such test, the claim is not "works like a quality engineer". At most it is "drafts material for a quality engineer to check". That may still be useful. It is a different claim, with a different price.

3. Plan a small pilot before you roll a tool out

The study counted issues resolved per hour, comparing agents with their own earlier work and with agents without access. The groups already differed before rollout.
The customer-support study measured issues resolved per hour. Staggered access let the researchers use agents’ earlier work and compare people with and without access. The groups already differed before rollout (s5).
Figure data as text

Method from one study [s5]

PartWhat it means
MEASUREIssues resolved per hour
BASELINE AND COMPARISONCompare like work over time. Compare with agents’ own earlier work and with agents who did not have the tool.
STARTING POINTSDon’t credit the tool for a head start. The group that got the tool was already resolving more issues per hour.

A vendor's evidence is about the vendor's test. The only way to know what a tool does on your work is to try it on your work, and to plan how you will judge it before you start.

A measured example: AI help for customer-support agents

Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied a chat assistant, built on OpenAI's GPT-3, given to customer-support agents at a Fortune 500 company that sells business-process software. Their paper was first released in 2023 and published in The Quarterly Journal of Economics in 2025. The published version covers 5,172 agents (the 2023 working paper gave 5,179).5 It is useful here less for its result than for how it was measured.

This is one study, of one company's support agents, with a tool rolled out in 2020 and 2021. Do not repeat "AI makes workers 15% faster" as a general fact. The point is the method: a measure, a baseline, a comparison, and a look at who gained and who did not.

NIST asks for the same kind of discipline when AI systems are tested: AI systems "should be tested before their deployment and regularly while in operation", test sets and metrics should be documented, and test processes should be "objective, repeatable, or scalable", followed and documented.2 It also says that "Processes for independent review can improve the effectiveness of testing and can mitigate internal biases and potential conflicts of interest", and that what cannot be measured should be written down too.2

Seven steps for a small pilot

These steps are this course's own plan, drawn from the ideas above. Write the first five down before anyone uses the tool.

  1. The claim, as questions. Write the claim you are testing, in the five-question form from section 1.
  2. The task. Choose real work your team does, in the conditions it is really done in: your documents, your systems, your deadlines.2
  3. The measure. Choose one main measure (for example, minutes per report) and at least one guard measure, so you can see if speed costs quality (for example, errors found by a checker). Say exactly how each is counted. Do not change them after you see the results.
  4. The baseline. Measure the same work, the same way, without the tool, first or in the same weeks.
  5. The comparison, the claim line and the decision rule. Split the people, not the work: decide by a rule (drawing names, a coin) which people use the tool and which do not, and do not pick the keenest. Why not let the same people take turns, one case with the tool and the next without? Because what people learn from a tool stays with them when they work without it, as the study above found.5 Their "no tool" cases then pick up part of the tool's effect, and the measured saving looks smaller than it is. (People in the two groups can still share tips, so even a split by people is not perfectly clean. Write that down as a limit.) Then write two lines:

    • The claim line: the result that would show the vendor's claim holds. If the claim is "halves the time", that is a fall of about half.
    • The "no" line: the result that would mean you do not roll it out, for example "if minutes per report do not fall by at least a fifth, or errors rise, we do not roll it out." A pilot that cannot say no is not a test.

    The two lines can differ. A tool can be worth rolling out without doing what the vendor claimed.

  6. Who measures. Someone who did not choose the tool and does not run the project counts the results.2
  7. What you report. The result against the decision rule and against the vendor's claim, how many people and pieces of work it rests on, who gained and who did not, and what you could not measure.2 A small pilot can show a large gain, or a clear lack of one. It rarely proves a small gain. Say which you found.

The pilot's result is evidence for a decision. It is not the decision. A named person, not the tool and not the vendor, gives one of four answers: go ahead (roll it out for the task you tested), only this far (roll it out for a narrower task, or with a limit), wait for a person (the result is unclear, so it waits for a named person to decide what more is needed), or no. (These are course rules.)

You can still ask the team what they think of the tool. Keep it as a separate question. A survey of how people feel about a tool measures how they feel. It does not measure the time or the errors.

The misconception this corrects: "If the team likes the tool, it is working." Liking a tool is worth knowing, and it matters for whether people will use it. Whether the tool saves time or improves the work is a different question, answered by the measure you chose in advance against the baseline.

At Big Widget Company

The vendor of a drafting tool claims it "halves the time to write a supplier corrective-action request". (A made-up claim.) Lee, in supplier quality, plans a six-week pilot:

Suppose the time falls by 35% and send-backs stay flat. (A made-up result.) That passes the roll-out line, but it does not confirm the vendor's claim. The report says both: "Meets our roll-out rule. Vendor's 'halves the time' claim not confirmed: we measured about a third." If the time does not fall, this design can show that too. That is what makes it a test.

4. The three yeses for claims about tools

The Foundation's three yeses are a cycle, not a one-time approval. Here they are applied to claims about AI tools. Checking a claim is one part of a yes, not all of it: the data, security and policy gates in the other modules still apply.

What you keep. Keep the claim, the five questions with their answers, your pilot plan and the result in your own controlled work record. In this course, a receipt records the decision, who made it and when. It never holds prompts, the tool's answers, customer details or anyone's private information. A receipt shows that a decision was made. It does not show that the vendor's claim was true.

Words to know: claim, benchmark, baseline, comparison group, guard measure, claim line, decision rule, allegation, consent order, substantiation, not checked.

Further learning (optional, not part of this lesson): University of Washington, Calling Bullshit: Data Reasoning in a Digital World: <https://callingbullshit.org/>; Carl T. Bergstrom and Jevin D. West (University of Washington), Modern-Day Oracles or Bullshit Machines?: <https://thebullshitmachines.com/>; Stanford University, CS324 Large Language Models, Capabilities lecture (Winter 2022, technical): <https://stanford-cs324.github.io/winter2022/lectures/capabilities/>.

Sources

  1. s3 Percy Liang et al. (2022). Holistic Evaluation of Language Models. Paper. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, session separate from the writer), 18 Nov 2026. ↩ ↩ ↩ ↩
  2. s4 Elham Tabassi (National Institute of Standards and Technology) (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). Standard. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, fresh session separate from the writer), round 2 re-check, 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s1 Federal Trade Commission (2024). FTC Announces Crackdown on Deceptive AI Claims and Schemes (Operation AI Comply), archived copy. Official doc. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, session separate from the writer), 18 Nov 2026; confirmed on the live FTC page at live_url, because the fetch tool refused the web.archive.org url (see checks/source-check.md). ↩ ↩ ↩ ↩ ↩
  4. s2 Federal Trade Commission (2025). In the Matter of DoNotPay, Inc., Decision and Order, Docket No. C-4812. Record. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, fresh session separate from the writer), round 2 re-check, 18 Nov 2026; read in full from the same PDF on www.ftc.gov (/system/files/ftc_gov/pdf/2323042_donotpay_decision_and_order_0.pdf), because the search.ftc.gov url returned no response this session (see checks/source-check.md). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  5. s5 Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond (2023). Generative AI at Work. Paper. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, fresh session separate from the writer), round 2 re-check, 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩

Next: the quiz