
Checking claims about AI tools · Technical gear
You will be able to: operationalise a performance claim about an AI tool into testable components (task, metric, comparator, sample, evaluator); explain, from the FTC's DoNotPay record, why a claim of professional-level performance requires substantiation by testing against that profession, keeping the complaint's allegations apart from the January 2025 consent order; and design a small pilot with a pre-specified primary measure, a baseline, a comparison and a decision rule that can return "no".
This module is as of 18 November 2026. Reviewed and published, version 1.0.0. Not a credential. Primary sources are listed in sources.yml. The FTC release (s1) reports allegations; the Decision and Order (s2) is the operative record. Big Widget Company, its people, tools and vendor claims are made up; your employer's rules come first.
1. Operationalising a claim
The five components
An unqualified claim ("95% accurate", "saves four hours a week"; both made up) is not testable until it is decomposed. This course uses five components:
| Component | Specify | Typical gap in vendor claims |
|---|---|---|
| Task T | Input distribution and output required | Benchmark task differs from the deployment task |
| Metric M | What is counted, how ground truth is established, by whom | "Accuracy" undefined; graders not blind |
| Comparator C | Baseline process, alternative tool or human | No comparator, or a weak one |
| Sample n | Items, users, sites, time window | Small or unstated; cherry-picked |
| Evaluator E | Who ran it; independence and incentives | Vendor-run, unpublished |
A relative claim ("cuts time by 60%") is a ratio:
\Delta_{\%} = \frac{\bar{y}_{\text{baseline}} - \bar{y}_{\text{tool}}}{\bar{y}_{\text{baseline}}} \times 100
It is undefined until you know $\bar{y}_{\text{baseline}}$ (measured how, on what), which population contributed each mean, and $n$ for each. Questions that do not bear on T, M, C, n or E (brand reputation, demo quality, other customers, team enthusiasm) do not test the claim.
Benchmarks and the deployment distribution
A benchmark score estimates performance on the benchmark's item distribution under its scoring rule. Your interest is performance on your own work distribution. Nothing guarantees the two agree, and HELM's results show several ways they diverge.1
- Coverage. HELM (Liang et al., 2022; TMLR 2023) evaluated 30 models on 16 core and 26 targeted scenarios (42 in all) across 7 metrics: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. Before HELM, models "on average were evaluated on just 17.9% of the core HELM scenarios, with some prominent models not sharing a single scenario in common"; HELM raised this to 96.0%.1 Headline scores from different vendors may not be on comparable test sets at all.
- Metric trade-offs. Accuracy, robustness and fairness were strongly correlated across scenarios, but HELM reports "trade-offs where the most accurate model is not the most robust or most fair".1 A leaderboard rank on one metric does not transfer to another.
- Sensitivity to perturbation. On NarrativeQA, TNLG v2 (530B), then the third-most accurate model there, "precipitously drops from 72.6% standard accuracy ... to 38.9% accuracy in the presence of robustness perturbations", where robustness is worst-case accuracy over perturbations such as typos.1 Clean benchmark inputs understate the noise in operational inputs.
- Acknowledged incompleteness. HELM selected scenarios on "coverage and feasibility, noting what's missing or underrepresented".1
NIST AI RMF 1.0 frames this as a measurement problem. Section 1.2.1: measurement "in a laboratory or a controlled environment may yield important insights pre-deployment, these measurements may differ from risks that emerge in operational, real-world settings."2 Table 3 asks that performance criteria be "demonstrated for conditions similar to deployment setting(s)" (MEASURE 2.3), that "Limitations of the generalizability beyond the conditions under which the technology was developed are documented" (MEASURE 2.5), and that measurement approaches be "connected to deployment context(s) and informed through consultation with domain experts and other end users" (MEASURE 4.1).2
The misconception this corrects: "A benchmark score tells me how the tool will do on my work." It is evidence about a different distribution, metric and time. Its proper use is to shortlist tools for a test on your own distribution.
Worked example (Big Widget, made up)
Vendor claim: "95% accuracy on dealer warranty questions; #1 on a public leaderboard."
- T: which question types (coverage dates, disputed claims with photos), languages, product lines? Is the leaderboard's task warranty question-answering at all?
- M: what counts as correct; who established ground truth; were graders blind to the source?
- C: the warranty specialists' current rate on the same items. If that is 97%, the claim is a regression.
- n: items and time window; is the test set available for inspection?
- E: vendor-run or independent?
Status until answered: "vendor claim, not checked", with a named owner and a date for the follow-up. "Not stated" and "not checked" are findings in their own right; they are not evidence that the claim is false, nor that it is true.
2. Substantiating a claim of professional-level performance
A claim that a tool performs "like a lawyer" (or engineer, or accountant) is a comparative claim whose comparator is the professional. The evidence that can support it is a comparison against that professional, on the same work, evaluated by people competent in that profession.
The DoNotPay record: three stages, kept apart
1. Complaint allegations (announced 25 September 2024). In its Operation AI Comply release, the FTC described DoNotPay as having claimed to be "the world's first robot lawyer". "The complaint alleges that the company did not conduct testing to determine whether its AI chatbot's output was equal to the level of a human lawyer", and that "the company itself did not hire or retain any attorneys".3 These are allegations, not adjudicated findings.
2. Proposed consent order (25 September 2024). The release described a "proposed Commission order" requiring payment of $193,000 and notice to consumers who subscribed between 2021 and 2023, subject to public comment.3
3. Decision and Order, Docket No. C-4812 (issued 14 January 2025).4
- Posture. The respondent "neither admits nor denies any of the allegations in the Complaint, except as specifically stated in this Decision and Order, and that only for purposes of this action, it admits the facts necessary to establish jurisdiction."4 The order is a settlement; it is not a finding that the allegations are true.
- Scope. "Professional Services" are activities traditionally performed by a person that require advanced knowledge acquired by "a prolonged course of specialized intellectual instruction" (or creative talent). The examples include legal advice and representation, accounting, engineering, consulting, research, translation and medical services.4
- Provision I (substantiation). DoNotPay must not represent that a covered product "operates like a human lawyer", or that it "performs a Professional Service like a professional in the relevant field", "unless the representation is non-misleading, including that, at the time such representation is made, they possess and rely upon competent and reliable evidence that is sufficient in quality and quantity based on standards generally accepted in the relevant fields ... to substantiate that the representation is true."4 "Competent and reliable evidence" means "tests, analyses, research, studies, or other evidence based on the expertise of professionals in the relevant area, that (1) have been conducted and evaluated in an objective manner by qualified persons and (2) are generally accepted in the profession to yield accurate and reliable results."4
- Provision II. It must not misrepresent "that a Covered Product or Service will save consumers legal fees".4
- Monetary relief and notice. $193,000 (Provision III); emailed notice to identified new subscribers between 1 January 2021 and 31 December 2023 (Provision VI), in the form of Attachment A, which says "the FTC says we did not have sufficient proof for our claims that DoNotPay operates like a human lawyer when it generates demand letters and initiates cases in small claims court" and "DoNotPay neither admits nor denies the FTC's claims".4
- Record-keeping (Provision IX.F). For each covered representation, DoNotPay must keep "all materials that were relied upon in making the representation" and any evidence that "contradicts, qualifies, or otherwise calls into question the representation".4
Reading the standard as a test specification
Provision I's definition maps onto the components in section 1:
| Order language | Component |
|---|---|
| "tests, analyses, research, studies" | A test exists, with T and M defined |
| "based on the expertise of professionals in the relevant area" | C and ground truth come from the profession |
| "conducted and evaluated in an objective manner by qualified persons" | E is competent and the evaluation is objective |
| "sufficient in quality and quantity" | n adequate for the claim |
| "at the time such representation is made" | Evidence exists before the claim, not after |
This order binds DoNotPay. As a buyer, you can still ask any vendor making a professional-equivalence claim for the equivalent of its Provision IX.F file: the evidence relied on, and anything that contradicts it.
The misconception this corrects: "If a product is on sale, someone has checked that it works as claimed." On the FTC's allegations, the claim was marketed without the comparison test.3 The notice period covers subscribers from 2021 to 2023; the complaint was announced in 2024 and the order issued in 2025.34 Availability for sale is not evidence of substantiation.
Worked example (Big Widget, made up)
A brochure: "does the work of a quality engineer". Ask for: the comparison test against quality engineers on the same cases; how cases were sampled and how many; who judged correctness, their qualifications, and whether they were blind to the source; and the tool's errors that an engineer would have caught. Without that, the supportable claim is narrower ("drafts for an engineer to review"), and should be priced and governed as such.
3. Designing a small pilot
Figure data as text
Method from one study [s5]
| Part | What it means |
|---|---|
| MEASURE | Issues resolved per hour |
| BASELINE AND COMPARISON | Compare like work over time. Compare with agents’ own earlier work and with agents who did not have the tool. |
| STARTING POINTS | Don’t credit the tool for a head start. The group that got the tool was already resolving more issues per hour. |
A reference design: Brynjolfsson, Li and Raymond
Generative AI at Work (NBER Working Paper 31161, 2023; The Quarterly Journal of Economics 140(2), 2025) studies a GPT-3-based chat assistant deployed to customer-support agents at "a Fortune 500 firm that sells business-process software".5 Figures below are from the published article unless marked.
- Sample and rollout. 5,172 agents (the 2023 working paper: 5,179). Rollout "primarily took place during fall 2020 and winter 2021", with timing driven by limited training capacity and budget; individuals in the same team were onboarded at different times. Some agents were never treated in the sample period.5
- Primary outcome. Resolutions per hour (RPH), with its components (average handle time, chats per hour, resolution rate) and customer satisfaction (net promoter score from a random post-chat survey) as further outcomes.5
- Selection. Treated agents "already had higher resolutions per hour prior to AI model deployment (2.0 chats) relative to never-treated agents (1.7)".5 A raw treated-versus-untreated contrast would mix the tool's effect with that head start.
- Estimator. A difference-in-differences regression with year-month fixed effects and, in the preferred specifications, agent fixed effects, reporting effects as "percentage changes off the baseline pretreatment means":5
y_{it} = \delta_t + \alpha_i + \beta\,\mathrm{AI}_{it} + \gamma X_{it} + \epsilon_{it}
- Results and heterogeneity. Access raised RPH by 15% on average (working paper: 14%). Less skilled and less experienced workers saw "a 30% increase in the number of issues resolved per hour"; "the most experienced and highest-skilled workers see small gains in speed and small declines in quality".5
- Carry-over (learning). The authors "find evidence that AI assistance facilitates worker learning": "experience with AI recommendations can lead to durable learning". Using software outages, they find "that workers see productivity gains relative to their pre-AI baseline even when AI recommendations are unavailable"; "even during outage periods when the AI system is not working, AI-exposed agents continue to handle calls faster". They add that "Because AI outages are rare, our estimates are noisy".5 Note that their comparison is between agents (and each agent's own pre-period), not between treated and untreated items for the same agent. That choice matters for pilot design: see item 5 below.
Treat the numbers as one setting's result (one firm, one task, a 2020–21 GPT-3-based tool). The transferable part is the design: a defined primary outcome, a pre-period baseline, a comparison group, attention to selection, and reporting of heterogeneity and a quality guard.
NIST's measurement expectations
AI RMF 1.0, section 5.3: "AI systems should be tested before their deployment and regularly while in operation"; processes should include "comparisons to performance benchmarks"; "Processes for independent review can improve the effectiveness of testing and can mitigate internal biases and potential conflicts of interest"; and after MEASURE, "objective, repeatable, or scalable" TEVV (test, evaluation, verification and validation) processes "are in place, followed, and documented".2 Table 3: "Test sets, metrics, and details about the tools used during TEVV are documented" (MEASURE 2.1), and "The risks or trustworthiness characteristics that will not – or cannot – be measured are properly documented" (MEASURE 1.1).2
A pilot protocol
This protocol is the course's own synthesis. Fix items 1–5 in writing before the first use.
- Claim. Restate it as T, M, C, n, E.
- Task. Real work items in real conditions (your documents, systems, time pressure), per MEASURE 2.3.2
- Primary and guard measures. One primary outcome $y$ (e.g. minutes per item from system timestamps) and at least one guard $g$ (e.g. rework rate judged by a reviewer blind to condition). Define counting rules precisely; do not change them after seeing data.
- Baseline. Collect $y$ and $g$ without the tool, over a pre-period, on comparable items.
Assignment, claim threshold and decision rule. Assign people (or whole teams) to tool / no-tool by a rule fixed in advance (randomisation, or drawing names), not by volunteer choice, which reproduces the selection problem above. Do not alternate items or weeks within the same person. If using the tool teaches people things they keep using without it, as s5 found,5 the no-tool items of a tool user are contaminated by carry-over: the within-person contrast $\bar{y}{\text{tool}} - \bar{y}{\text{no tool}}$ is biased toward zero and can turn a real saving into a "no". Assigning people removes that within-person carry-over, though not spillover between colleagues who share tips; record that as a limitation. With a pre-period for both arms, a simple estimate is the difference in differences
\hat{\beta} = (\bar{y}_{T,\text{post}} - \bar{y}_{T,\text{pre}}) - (\bar{y}_{C,\text{post}} - \bar{y}_{C,\text{pre}})Because people, not items, are assigned, the effective sample size is closer to the number of people than to the number of items; report both. Then fix two thresholds in writing:
- Claim threshold. The effect size at which the vendor's claim would hold (for "halves the time", a relative fall of about 50%).
- Decision rule. The "no" outcome for roll-out, e.g. roll out only if the median of $y$ falls by at least a stated threshold and $g$ does not rise.
These answer different questions, and a result can pass the decision rule while failing the claim threshold.
- Evaluator. Someone outside the project and not involved in selecting the tool computes the result.2
- Report. The estimate against the decision rule and against the claim threshold, $n$ per arm (people and items), the spread (not only the mean), subgroup differences, and what was not measured (MEASURE 1.1).2 A small pilot can detect a large effect or rule out a large one; it is weak evidence about a small effect, and the report should say so.
The report is evidence for a decision; it does not make one. A named person records one of the four gate answers: go ahead (the tested task), only this far (a narrower task or a limit), wait for a person (inconclusive; a named person decides what further evidence is needed), or no. (Course rules.)
User satisfaction can be surveyed, but as a separate outcome. It measures perception, not $y$ or $g$.
The misconception this corrects: "If the team likes the tool, it is working." Adoption sentiment is a real outcome, but it is not the primary measure. The claim is tested by the pre-specified $y$ and $g$ against the baseline and comparison.
Worked example (Big Widget, made up)
Claim: a drafting tool "halves the time to write a supplier corrective-action request". Six-week pilot; the team is split in half by drawing names, one half with the tool and one without; $y$ = minutes from case open to request sent (case-system timestamps); $g$ = send-backs for errors by the supplier quality lead; baseline = the same measures for both halves over the three preceding weeks; estimate = difference in differences. Written on day one: claim threshold, a fall in median $y$ of about 50% relative to the no-tool half; decision rule, roll out only if median $y$ falls by at least 30% and $g$ does not rise. Evaluator = an analyst outside the team; decision = Morgan, the supplier quality manager. Recorded limitations: spillover between the halves; one six-week window; a small number of people per arm.
A made-up result of a 35% fall with $g$ flat passes the decision rule but not the claim threshold. The report says both: roll-out rule met; vendor's claim not confirmed. The design can also show that the tool did not save time, which is what makes it a test.
4. The three yeses for claims about tools
- Get to Yes: decompose the claim into T, M, C, n, E; for professional-equivalence claims, ask for the comparison test against the profession; run a pre-specified pilot with a claim threshold and a decision rule; a named person gives the gate answer (go ahead, only this far, wait for a person, no). The yes is scoped to the task tested. Other gates (data, security, policy) still apply.
- Stay at Yes: keep measuring $y$ and $g$ in operation; NIST asks for testing "regularly while in operation".2 Date every result; tools, models and vendors change, and a yes for one version is not a yes for the next until it is re-measured.
- Recover to Yes: if the measure degrades or a relied-on claim proves unsubstantiated, withdraw reliance for that task, notify the people who relied on it, name the owner of the fix, and change the intake check for future claims. Withdrawal is containment, not recovery: return to yes only on a new, independently computed measurement that meets the rule.
What you keep. The decomposed claim, the vendor's answers, the pilot protocol and the result, in your own controlled work record. A course receipt records the decision, who made it and when; it never holds prompts, outputs, customer data or personal information. It is evidence that a decision was made, not evidence that the claim was true.
Words to know: operationalise, benchmark, distribution shift, comparator, baseline, difference-in-differences, selection, carry-over, guard measure, claim threshold, decision rule, allegation, consent order, substantiation, competent and reliable evidence.
Further learning (optional, not part of this lesson): Stanford University, CS324 Large Language Models, Capabilities lecture (Winter 2022): <https://stanford-cs324.github.io/winter2022/lectures/capabilities/>; University of Washington, Calling Bullshit: Data Reasoning in a Digital World: <https://callingbullshit.org/>; Carl T. Bergstrom and Jevin D. West, Modern-Day Oracles or Bullshit Machines?: <https://thebullshitmachines.com/>.