Level 200: Using gen AI well · Module 200.7

One coworker sets down a notebook while another sips from a mug.
What did they measure?

Checking claims about AI tools · Technical gear

You will be able to: operationalise a performance claim about an AI tool into testable components (task, metric, comparator, sample, evaluator); explain, from the FTC's DoNotPay record, why a claim of professional-level performance requires substantiation by testing against that profession, keeping the complaint's allegations apart from the January 2025 consent order; and design a small pilot with a pre-specified primary measure, a baseline, a comparison and a decision rule that can return "no".

This module is as of 18 November 2026. Reviewed and published, version 1.0.0. Not a credential. Primary sources are listed in sources.yml. The FTC release (s1) reports allegations; the Decision and Order (s2) is the operative record. Big Widget Company, its people, tools and vendor claims are made up; your employer's rules come first.

1. Operationalising a claim

The five components

An unqualified claim ("95% accurate", "saves four hours a week"; both made up) is not testable until it is decomposed. This course uses five components:

ComponentSpecifyTypical gap in vendor claims
Task TInput distribution and output requiredBenchmark task differs from the deployment task
Metric MWhat is counted, how ground truth is established, by whom"Accuracy" undefined; graders not blind
Comparator CBaseline process, alternative tool or humanNo comparator, or a weak one
Sample nItems, users, sites, time windowSmall or unstated; cherry-picked
Evaluator EWho ran it; independence and incentivesVendor-run, unpublished

A relative claim ("cuts time by 60%") is a ratio:

\Delta_{\%} = \frac{\bar{y}_{\text{baseline}} - \bar{y}_{\text{tool}}}{\bar{y}_{\text{baseline}}} \times 100

It is undefined until you know $\bar{y}_{\text{baseline}}$ (measured how, on what), which population contributed each mean, and $n$ for each. Questions that do not bear on T, M, C, n or E (brand reputation, demo quality, other customers, team enthusiasm) do not test the claim.

Benchmarks and the deployment distribution

A benchmark score estimates performance on the benchmark's item distribution under its scoring rule. Your interest is performance on your own work distribution. Nothing guarantees the two agree, and HELM's results show several ways they diverge.1

NIST AI RMF 1.0 frames this as a measurement problem. Section 1.2.1: measurement "in a laboratory or a controlled environment may yield important insights pre-deployment, these measurements may differ from risks that emerge in operational, real-world settings."2 Table 3 asks that performance criteria be "demonstrated for conditions similar to deployment setting(s)" (MEASURE 2.3), that "Limitations of the generalizability beyond the conditions under which the technology was developed are documented" (MEASURE 2.5), and that measurement approaches be "connected to deployment context(s) and informed through consultation with domain experts and other end users" (MEASURE 4.1).2

The misconception this corrects: "A benchmark score tells me how the tool will do on my work." It is evidence about a different distribution, metric and time. Its proper use is to shortlist tools for a test on your own distribution.

Worked example (Big Widget, made up)

Vendor claim: "95% accuracy on dealer warranty questions; #1 on a public leaderboard."

Status until answered: "vendor claim, not checked", with a named owner and a date for the follow-up. "Not stated" and "not checked" are findings in their own right; they are not evidence that the claim is false, nor that it is true.

2. Substantiating a claim of professional-level performance

A claim that a tool performs "like a lawyer" (or engineer, or accountant) is a comparative claim whose comparator is the professional. The evidence that can support it is a comparison against that professional, on the same work, evaluated by people competent in that profession.

The DoNotPay record: three stages, kept apart

1. Complaint allegations (announced 25 September 2024). In its Operation AI Comply release, the FTC described DoNotPay as having claimed to be "the world's first robot lawyer". "The complaint alleges that the company did not conduct testing to determine whether its AI chatbot's output was equal to the level of a human lawyer", and that "the company itself did not hire or retain any attorneys".3 These are allegations, not adjudicated findings.

2. Proposed consent order (25 September 2024). The release described a "proposed Commission order" requiring payment of $193,000 and notice to consumers who subscribed between 2021 and 2023, subject to public comment.3

3. Decision and Order, Docket No. C-4812 (issued 14 January 2025).4

Reading the standard as a test specification

Provision I's definition maps onto the components in section 1:

Order languageComponent
"tests, analyses, research, studies"A test exists, with T and M defined
"based on the expertise of professionals in the relevant area"C and ground truth come from the profession
"conducted and evaluated in an objective manner by qualified persons"E is competent and the evaluation is objective
"sufficient in quality and quantity"n adequate for the claim
"at the time such representation is made"Evidence exists before the claim, not after

This order binds DoNotPay. As a buyer, you can still ask any vendor making a professional-equivalence claim for the equivalent of its Provision IX.F file: the evidence relied on, and anything that contradicts it.

The misconception this corrects: "If a product is on sale, someone has checked that it works as claimed." On the FTC's allegations, the claim was marketed without the comparison test.3 The notice period covers subscribers from 2021 to 2023; the complaint was announced in 2024 and the order issued in 2025.34 Availability for sale is not evidence of substantiation.

Worked example (Big Widget, made up)

A brochure: "does the work of a quality engineer". Ask for: the comparison test against quality engineers on the same cases; how cases were sampled and how many; who judged correctness, their qualifications, and whether they were blind to the source; and the tool's errors that an engineer would have caught. Without that, the supportable claim is narrower ("drafts for an engineer to review"), and should be priced and governed as such.

3. Designing a small pilot

The study counted issues resolved per hour, comparing agents with their own earlier work and with agents without access. The groups already differed before rollout.
The customer-support study measured issues resolved per hour. Staggered access let the researchers use agents’ earlier work and compare people with and without access. The groups already differed before rollout (s5).
Figure data as text

Method from one study [s5]

PartWhat it means
MEASUREIssues resolved per hour
BASELINE AND COMPARISONCompare like work over time. Compare with agents’ own earlier work and with agents who did not have the tool.
STARTING POINTSDon’t credit the tool for a head start. The group that got the tool was already resolving more issues per hour.

A reference design: Brynjolfsson, Li and Raymond

Generative AI at Work (NBER Working Paper 31161, 2023; The Quarterly Journal of Economics 140(2), 2025) studies a GPT-3-based chat assistant deployed to customer-support agents at "a Fortune 500 firm that sells business-process software".5 Figures below are from the published article unless marked.

y_{it} = \delta_t + \alpha_i + \beta\,\mathrm{AI}_{it} + \gamma X_{it} + \epsilon_{it}

Treat the numbers as one setting's result (one firm, one task, a 2020–21 GPT-3-based tool). The transferable part is the design: a defined primary outcome, a pre-period baseline, a comparison group, attention to selection, and reporting of heterogeneity and a quality guard.

NIST's measurement expectations

AI RMF 1.0, section 5.3: "AI systems should be tested before their deployment and regularly while in operation"; processes should include "comparisons to performance benchmarks"; "Processes for independent review can improve the effectiveness of testing and can mitigate internal biases and potential conflicts of interest"; and after MEASURE, "objective, repeatable, or scalable" TEVV (test, evaluation, verification and validation) processes "are in place, followed, and documented".2 Table 3: "Test sets, metrics, and details about the tools used during TEVV are documented" (MEASURE 2.1), and "The risks or trustworthiness characteristics that will not – or cannot – be measured are properly documented" (MEASURE 1.1).2

A pilot protocol

This protocol is the course's own synthesis. Fix items 1–5 in writing before the first use.

  1. Claim. Restate it as T, M, C, n, E.
  2. Task. Real work items in real conditions (your documents, systems, time pressure), per MEASURE 2.3.2
  3. Primary and guard measures. One primary outcome $y$ (e.g. minutes per item from system timestamps) and at least one guard $g$ (e.g. rework rate judged by a reviewer blind to condition). Define counting rules precisely; do not change them after seeing data.
  4. Baseline. Collect $y$ and $g$ without the tool, over a pre-period, on comparable items.
  5. Assignment, claim threshold and decision rule. Assign people (or whole teams) to tool / no-tool by a rule fixed in advance (randomisation, or drawing names), not by volunteer choice, which reproduces the selection problem above. Do not alternate items or weeks within the same person. If using the tool teaches people things they keep using without it, as s5 found,5 the no-tool items of a tool user are contaminated by carry-over: the within-person contrast $\bar{y}{\text{tool}} - \bar{y}{\text{no tool}}$ is biased toward zero and can turn a real saving into a "no". Assigning people removes that within-person carry-over, though not spillover between colleagues who share tips; record that as a limitation. With a pre-period for both arms, a simple estimate is the difference in differences

    \hat{\beta} = (\bar{y}_{T,\text{post}} - \bar{y}_{T,\text{pre}}) - (\bar{y}_{C,\text{post}} - \bar{y}_{C,\text{pre}})

    Because people, not items, are assigned, the effective sample size is closer to the number of people than to the number of items; report both. Then fix two thresholds in writing:

    • Claim threshold. The effect size at which the vendor's claim would hold (for "halves the time", a relative fall of about 50%).
    • Decision rule. The "no" outcome for roll-out, e.g. roll out only if the median of $y$ falls by at least a stated threshold and $g$ does not rise.

    These answer different questions, and a result can pass the decision rule while failing the claim threshold.

  6. Evaluator. Someone outside the project and not involved in selecting the tool computes the result.2
  7. Report. The estimate against the decision rule and against the claim threshold, $n$ per arm (people and items), the spread (not only the mean), subgroup differences, and what was not measured (MEASURE 1.1).2 A small pilot can detect a large effect or rule out a large one; it is weak evidence about a small effect, and the report should say so.

The report is evidence for a decision; it does not make one. A named person records one of the four gate answers: go ahead (the tested task), only this far (a narrower task or a limit), wait for a person (inconclusive; a named person decides what further evidence is needed), or no. (Course rules.)

User satisfaction can be surveyed, but as a separate outcome. It measures perception, not $y$ or $g$.

The misconception this corrects: "If the team likes the tool, it is working." Adoption sentiment is a real outcome, but it is not the primary measure. The claim is tested by the pre-specified $y$ and $g$ against the baseline and comparison.

Worked example (Big Widget, made up)

Claim: a drafting tool "halves the time to write a supplier corrective-action request". Six-week pilot; the team is split in half by drawing names, one half with the tool and one without; $y$ = minutes from case open to request sent (case-system timestamps); $g$ = send-backs for errors by the supplier quality lead; baseline = the same measures for both halves over the three preceding weeks; estimate = difference in differences. Written on day one: claim threshold, a fall in median $y$ of about 50% relative to the no-tool half; decision rule, roll out only if median $y$ falls by at least 30% and $g$ does not rise. Evaluator = an analyst outside the team; decision = Morgan, the supplier quality manager. Recorded limitations: spillover between the halves; one six-week window; a small number of people per arm.

A made-up result of a 35% fall with $g$ flat passes the decision rule but not the claim threshold. The report says both: roll-out rule met; vendor's claim not confirmed. The design can also show that the tool did not save time, which is what makes it a test.

4. The three yeses for claims about tools

What you keep. The decomposed claim, the vendor's answers, the pilot protocol and the result, in your own controlled work record. A course receipt records the decision, who made it and when; it never holds prompts, outputs, customer data or personal information. It is evidence that a decision was made, not evidence that the claim was true.

Words to know: operationalise, benchmark, distribution shift, comparator, baseline, difference-in-differences, selection, carry-over, guard measure, claim threshold, decision rule, allegation, consent order, substantiation, competent and reliable evidence.

Further learning (optional, not part of this lesson): Stanford University, CS324 Large Language Models, Capabilities lecture (Winter 2022): <https://stanford-cs324.github.io/winter2022/lectures/capabilities/>; University of Washington, Calling Bullshit: Data Reasoning in a Digital World: <https://callingbullshit.org/>; Carl T. Bergstrom and Jevin D. West, Modern-Day Oracles or Bullshit Machines?: <https://thebullshitmachines.com/>.

Sources

  1. s3 Percy Liang et al. (2022). Holistic Evaluation of Language Models. Paper. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, session separate from the writer), 18 Nov 2026. ↩ ↩ ↩ ↩ ↩
  2. s4 Elham Tabassi (National Institute of Standards and Technology) (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). Standard. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, fresh session separate from the writer), round 2 re-check, 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s1 Federal Trade Commission (2024). FTC Announces Crackdown on Deceptive AI Claims and Schemes (Operation AI Comply), archived copy. Official doc. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, session separate from the writer), 18 Nov 2026; confirmed on the live FTC page at live_url, because the fetch tool refused the web.archive.org url (see checks/source-check.md). ↩ ↩ ↩ ↩
  4. s2 Federal Trade Commission (2025). In the Matter of DoNotPay, Inc., Decision and Order, Docket No. C-4812. Record. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, fresh session separate from the writer), round 2 re-check, 18 Nov 2026; read in full from the same PDF on www.ftc.gov (/system/files/ftc_gov/pdf/2323042_donotpay_decision_and_order_0.pdf), because the search.ftc.gov url returned no response this session (see checks/source-check.md). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  5. s5 Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond (2023). Generative AI at Work. Paper. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, fresh session separate from the writer), round 2 re-check, 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩

Next: the quiz