Try each exercise first, then open its model answer.
Checking claims about AI tools · Workbook
Three exercises, one for each thing this module teaches. Everything at Big Widget Company is made up: its people, its tools, the vendors and their claims. Do not use real names, customers or documents from your own work. Write short answers. "Not stated" and "I don't know yet" are honest answers if you also say who will find out, and by when.
Exercise 1 (200.7-a)
Five questions for one claim. Ana in quality gets this slide from a vendor. (Made up.)
"Our AI assistant cuts quality-report time by 60%. In our test of 50 reports, writers using the assistant finished 60% faster. #1 on a public leaderboard for business writing."
- Write the five questions (which task, measured how, against what, out of how many, by whom) for this claim. For each one, say what the slide answers, or write "not stated".
- Ana's manager says: "It's number one on a public leaderboard. That settles it." Answer the manager in two or three sentences.
- Ana's colleague suggests three more checks: "Is the vendor well known? Did the demo look good? Do other companies use it?" Do these test the claim? Why or why not?
- Write the note Ana puts in the team's tool log today.
Model answer
- A good answer is close to this:
- Which task? "Quality reports", but which kind? Short weekly summaries, or full investigation reports? On whose documents? Not stated.
- Measured how? "Finished 60% faster", but counted from when to when? Did anyone check that the faster reports were still right? Not stated.
- Against what? "Faster" than what: the same writers without the assistant, other writers, last year? On the same kind of reports? Not stated.
- Out of how many? 50 reports. How many writers, and over how many weeks: not stated.
- By whom? "Our test": the vendor ran it, and the vendor gains if the result looks good.
- A leaderboard score tells you how a model did on that benchmark, scored that way, at that time. It may not even be about quality reports. In the HELM study, one model's accuracy on one test fell from 72.6% to 38.9% when small changes such as typos were added, and the most accurate model was not always the most robust or fair. NIST says results from a lab or controlled setting may differ from real-world use. So the score is a reason to test the tool on our reports, not a replacement for it.
- No. They may be worth knowing, but none of them tells you whether the tool cuts report time, by 60%, on Big Widget's reports. A famous vendor can make an untested claim, a demo is chosen to look good, and other companies' use is not a measurement.
- Something like: "Vendor claim: 60% faster quality reports. Vendor says, not checked. Task, measure, comparison, number of writers and weeks: not stated. Test run by the vendor on 50 reports. Questions sent today; Ana owns the follow-up, due in two weeks." The note does not say "true" or "false". It says where the evidence stops and who will find out more.
Exercise 2 (200.7-b)
"Does the work of a professional." A vendor offers Big Widget a tool that, its brochure says, "does the work of a professional translator" for dealer manuals. (Made up.)
- What kind of claim is this, and what kind of evidence would it need?
- In 2025 the FTC's final order in the DoNotPay case defined the evidence DoNotPay needs before saying its product works like a professional. Using that definition as a checklist, write three things the vendor's evidence should have.
- The vendor sends three things: (a) a page of happy customer reviews; (b) a top score on a public translation benchmark; (c) "We've been on the market since 2022." Say whether each meets your checklist, and why.
- A colleague says: "The FTC proved DoNotPay never tested its robot lawyer." Correct this, keeping what the complaint alleged apart from what the final order says.
- If the vendor has no test against professional translators, what is the honest claim?
Model answer
- It is a comparison claim: the tool's work is as good as a professional translator's. A comparison claim needs evidence from a comparison. The tool and professional translators do the same work, and people qualified to judge translation judge the results.
The order's definition of "competent and reliable evidence" gives the checklist. The evidence should be:
- tests or studies based on the expertise of professionals in the field (here, translation);
- done and judged in an objective manner by qualified people, for example professional translators who do not know which translation came from the tool;
- of a kind the profession generally accepts as giving accurate and reliable results.
The order's list of professional services names translation, along with law, accounting, engineering, consulting and research. The order is about DoNotPay, not this vendor. Its definition is still a clear statement of what this kind of evidence looks like.
- None of the three meets the checklist.
- (a) Reviews show that some customers were pleased. Nobody compared the tool with a professional translator, and nobody qualified judged the results.
- (b) A benchmark score shows how the tool did on that test. It is not a comparison with professional translators on dealer manuals.
- (c) Being on sale shows that someone is selling it. In the DoNotPay case, people subscribed from 2021 to 2023, and the FTC's 2024 complaint alleged the company had not tested whether its chatbot's output was equal to a human lawyer's.
- The FTC did not prove it. The complaint (September 2024) alleged that DoNotPay did not test whether its chatbot's output was equal to a human lawyer's. An allegation is what the FTC said, not a fact found by a court. The final order (14 January 2025) settled the case: DoNotPay neither admitted nor denied the allegations. It had to pay $193,000 and email a notice to subscribers who joined from 2021 to 2023. The notice says "the FTC says" DoNotPay did not have sufficient proof, which reports the FTC's claim and is not an admission. From now on, DoNotPay may not claim its product works like a lawyer or another professional unless it has competent and reliable evidence.
- "Drafts translations for a professional translator to check." That may still be useful, but it is a different claim, and it is worth a different price.
Exercise 3 (200.7-c)
Plan a pilot that can say no. A vendor claims its AI tool "cuts the time to answer a dealer warranty email by 40%". (Made up.) The warranty team has 20 people. Sam, the warranty lead, asks you to plan a pilot before anyone buys it.
- Write the plan using the module's seven steps. Include a main measure, a guard measure, a comparison, a baseline, a claim line and a "no" line, who measures, and who decides.
- Kai suggests: "Easier to let each of us take turns: one email with the tool, the next without." Why does your plan not do that?
- Suppose the time falls by 25% and errors stay flat. (A made-up result.) Using your two lines, what does the report say?
- Suppose instead the time does not fall at all, but in a survey most of the team say they love the tool. What does the report say?
Model answer
A good plan looks something like this. Steps 1 to 5 are written down before anyone uses the tool.
- The claim, as questions. Which task: answering dealer warranty emails. Measured how: minutes from opening to sending. Against what: the same team's emails without the tool. Out of how many: 20 people over six weeks. By whom: our own analyst, not the vendor.
- The task. Real warranty emails from the team's own queue, in our own systems.
- The measures. Main: minutes from opening each email to sending the reply, from the email system's timestamps. Guard: replies a warranty specialist marks wrong, so you can see if speed costs quality. Both are fixed now and not changed after the results come in.
- The baseline. The same two measures for everyone for the three weeks before the pilot.
- The comparison and the two lines. Draw names to split the team: ten use the tool for six weeks, ten do not. Do not pick the keenest. Claim line: the vendor's claim holds only if the tool half's time falls by about 40% compared with the other half. "No" line: roll out only if median minutes fall by at least 20% (the team's own choice) and errors do not rise; otherwise no.
- Who measures. An analyst outside the warranty team, who did not choose the tool.
- What you report. The result against the roll-out rule and against the vendor's claim; how many people and emails it rests on; who gained and who did not; and what you could not measure. Known limits, written down: the two halves sit together and may share tips; six weeks may be too short to see some effects.
Who decides: Sam reads the report and gives one of four answers: go ahead (for this task), only this far (a narrower task, or with a limit), wait for a person (the result is unclear), or no.
- What people learn from a tool stays with them. In the customer-support study, agents who had used the AI tool were still faster than before during rare outages when it gave no suggestions (the authors call those estimates noisy). If the same people take turns, their "no tool" emails pick up part of the tool's effect, and the saving looks smaller than it is. So the plan splits the people, not the emails.
- "Meets our roll-out rule: time fell 25% and errors did not rise. The vendor's 40% claim was not confirmed." It also gives how many people and emails, who gained and who did not, and the limits. Then Sam decides. Passing the "no" line and confirming the claim are two different results.
- "The tool did not cut the time, so it does not meet our roll-out rule. The team likes it (survey, reported separately)." Liking a tool is worth knowing, and it matters for whether people use it, but it does not measure the time or the errors. Sam's answer would be no, or wait for a person if there is a good reason to test something else.
What you keep. The claim, your questions and answers, the plan and the result go in your team's own controlled work record. In this course, a receipt records only the decision, who made it and when. It never holds emails, the tool's answers, dealer details or anyone's private information.