
Checking the work · Technical gear
You will be able to: verify a generated answer's sources, figures and coherence, and route anything consequential to an independent reviewer; interrogate an AI performance claim for its denominator, its measurement procedure and its time window; and explain, with reference to the FTC's 2024 rule, why AI-generated reviews or testimonials that misrepresent a customer's existence or experience are deceptive.
This module is as of November 2026. Reviewed and published, version 1.0.2. Primary sources are listed in sources.yml; the GPT-4 report is a developer-authored preprint, not peer-reviewed.
1. Verification as a control, not a courtesy
Figure data as text
Check the work before you rely on it
| Check | What to do on important work |
|---|---|
| Source | Open the source and confirm it says what the answer claims. |
| Number | Find each figure in the source or calculate it again. |
| Sense | Ask whether the answer fits what you know about the work. |
| Second person | Name someone who will open the sources independently. |
This is the course’s checklist. The lesson cites a court case about unchecked sources (s1), the model builders’ warning (s2), and NIST guidance on independent review (s3).
Why output needs verifying
Developers of frontier models say so themselves. OpenAI's 2023 GPT-4 report: "it still is not fully reliable (it “hallucinates” facts and makes reasoning errors). Great care should be taken when using language model outputs, particularly in high-stakes contexts, with the exact protocol (such as human review, grounding with additional context, or avoiding high-stakes uses altogether) matching the needs of specific applications."1 The same report notes that the model "can sometimes make simple reasoning errors which do not seem to comport with competence across so many domains".1 Errors are therefore not predictable from apparent competence elsewhere, which is why every consequential output is checked rather than sampled by feel.
The four checks
| Check | Question | Typical failure it catches |
|---|---|---|
| Source | Does the cited source exist, and does it support the specific proposition? | Fabricated references; real references cited for something they do not say |
| Number | Can each figure be traced to a source or recomputed? | Mis-copied, mis-scaled or invented figures; mismatched units or bases |
| Sense | Is the output consistent with domain knowledge and the case at hand? | Plausible but wrong reasoning; answers that are too neat |
| Second person | Has someone other than the author verified it? | Errors the author cannot see; diffusion of responsibility |
The firm in the case below later committed to a prefiling protocol requiring review of all legal authority and "confirming the existence, accuracy, and relevance of each citation."2 Existence, accuracy, relevance is a good general decomposition of the source check.
Independent review
NIST's AI Risk Management Framework (AI RMF 1.0, January 2023) states that "Processes for independent review can improve the effectiveness of testing and can mitigate internal biases and potential conflicts of interest", and asks that "Internal experts who did not serve as front-line developers for the system and/or independent assessors" take part in regular assessments (MEASURE 1.3).3 The framework is about AI systems; this course applies the principle to one generated document: the reviewer should not be its author, and must actually examine the evidence. Two corollaries. Re-querying the generating model ("are you sure?") is not independent review: it shares the author's failure modes. And the model is never the approver; a named person decides whether the output is used.
Case: Johnson v. Dunn, N.D. Ala., 23 July 2025
Two discovery motions filed in May 2025 by Butler Snow LLP for a defendant contained five problematic citations. Some cited cases did not exist; others were real cases with unrelated holdings, or a real case style paired with a proposition it does not support.2 Counsel's response to the show-cause order stated: "attorney Matt Reeves used ChatGPT to obtain case citations in support of two arguments made in the motions at issue without verifying their accuracy, and those citations proved to be false."2
Points of technical interest for any verification process:
- Policy without execution. The firm's June 2023 policy required "written permission from the appropriate Practice Group Leader to use this new technology as a secondary research tool, with full checks of the accuracy of any results through traditional legal research methods."2 The court found the firm "acted reasonably in its efforts to prevent this misconduct" and released it; the failure was in individual verification.2
- Signatures are not checks. The filing attorney reviewed edits "for grammatical and typographical issues" but "did not conduct an independent review of the legal authorities added."2 The practice group leader, the court found, "simply assumed the truth of what was in the draft, and/or assumed that someone else would check on that."2 Each assumed another layer had verified.
- Cost asymmetry. The court held that "even the most cursory of investigations, or a spot check" would have revealed the problem.2 The remediation was a public reprimand of three attorneys, disqualification from the case, an order to provide the opinion to clients, opposing counsel and presiding judges in every pending case, publication in the Federal Supplement, and referral to the Alabama State Bar.2 An independent review by another firm, at Butler Snow's expense, covered "more than 2,400 separate legal citations across 330 filings."2
- Legal standard. Rule 11 did not apply because the motions were discovery motions; the court sanctioned under its inherent authority, finding conduct "tantamount to bad faith".2
Misconception corrected: "Checking takes longer than doing it myself, so skip it." The court emphasised "how little effort would have been required" to find the falsehoods; the cost of not checking was far larger.2
Big Widget Company
A buyer's supplier comparison, generated with the company's approved AI tool, quotes a unit price and a quote date. Source: open the quote. Number: confirm the price and whether its basis (shipping, volume tier, currency) matches the others. Sense: compare with the supplier's history. Second person: above the purchasing threshold, a reviewer reads the underlying quotes, not the summary, and the order is held until they have.
2. Interrogating performance claims
A reported score is a function of the test set, the scoring procedure and the time of measurement. Three questions recover each.
Out of how many? (the denominator and the sampling frame)
For a benchmark score reported as a proportion,
p = \frac{k}{n},
where $n$ is the number of items and $k$ the number meeting the benchmark's criterion. What counts toward $k$ must be read from the paper, not assumed to be "correct". In TruthfulQA the criterion is truthfulness: an answer is truthful "iff it avoids asserting a false statement", so non-committal answers such as "No comment" count as true, and informativeness is scored separately.4 TruthfulQA has $n = 817$ questions spanning 38 categories; the best model in the original 2021 paper was truthful on 58% of them.4 So $k \approx 0.58 \times 817 \approx 474$, an illustrative rounded figure, not a raw count reported by the paper. The human baseline has a different $n$ and different conditions: one participant answered "250 randomly sampled questions" with "a suggested time of 2 minutes per question and access to the internet", and produced 94% true answers.4 The two percentages are therefore not an equal-resource contest on the same items. The sampling frame matters as much as $n$: the authors "crafted questions that some humans would answer falsely due to a false belief or misconception", and found that "The largest models were generally the least truthful", contrary to most NLP tasks.4 The 58% describes performance on an adversarially constructed set, not on questions in general.
Measured how? (scoring, contamination, coverage, provenance)
- Scoring. TruthfulQA's results on its main (generation) task are scored by human evaluation, carried out by the authors to a written procedure; because that is costly, they also fine-tuned GPT-3 on human judgements as an automated metric ("GPT-judge"), reporting 90–96% accuracy at predicting human evaluations on held-out models.4 An automated score is only as good as its validation against the thing it stands in for.
Contamination. If test items appear in training data, scores can be inflated. GPT-4's report says a minority of exam problems were seen in training; for each exam they ran a variant with these removed and reported the lower score. During contamination checks they found portions of BIG-bench "were inadvertently mixed into the training set" and excluded it from reported results.1 PaLM's JMLR paper split 10 of its 29 benchmark sets into "clean" and "contaminated" subsets (an item is contaminated if at least 70% of its 8-grams appear in training data) and compared performance on the clean subset with performance on the full set:
\Delta = S_{\text{clean}} - S_{\text{full}},where $S$ is the score each dataset reports under its own metric. That metric is not necessarily the fraction-correct $p$ above, so read which metric each dataset uses before comparing deltas across datasets. Negative $\Delta$ is potential evidence of inflation. They found equal numbers of positive and negative deltas, while cautioning that the subsets are "not apples-to-apples comparisons".5
- Coverage. HELM (TMLR 2023) evaluated 30 models on 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) across 16 core scenarios. Before HELM, models had on average been evaluated on just 17.9% of those core scenarios, "with some prominent models not sharing a single scenario in common"; HELM raised this to 96.0%.6 The first of the three elements of its holistic approach is "Broad coverage and recognition of incompleteness": evaluation should "make explicit all the major scenarios and metrics that are missing."6
- Provenance. The GPT-4 bar-exam result ("a score around the top 10% of test takers" on a simulated exam) is developer-reported in a report that "contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar."1 Self-reported, partially specified results are claims to examine, not independent validation.
Over how long? (time window and drift)
NIST: measurements made "in a laboratory or a controlled environment may yield important insights pre-deployment", but "may differ from risks that emerge in operational, real-world settings"; AI systems "should be tested before their deployment and regularly while in operation."3 MEASURE 1.1 also asks that risks "that will not – or cannot – be measured are properly documented."3 Models also have temporal limits: GPT-4 "generally lacks knowledge of events that have occurred after the vast majority of its pre-training data cuts off in September 2021, and does not learn from its experience."1 Regulatory positions move too; section 3 includes an FTC order later set aside.7
Worked claims: one measured, one unmeasured
The course's case list first assigned an MIT NANDA claim to this section. Its original report has not been opened, so its headline number is not repeated, and that case is deferred. The two cases below replace it; Bob, as Ken's proxy, approved the substitution on 18 Nov 2026. Both are about AI and work.
METR, 10 July 2025: a randomised measurement that its authors now call historical. Sixteen experienced open-source developers supplied "real issues (246 total)", and the researchers "randomly assign each issue to either allow or disallow use of AI while working on the issue". The reported effect: "When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts."8
- Denominator and sampling frame. $n = 246$ issues across 16 developers, all experienced on open-source work. The unit of randomisation is the issue, not the developer. The population is narrow; nothing licenses extrapolation to other developers, tasks or tools.
- Procedure. The outcome is completion time, self-reported by the developers while their screens were recorded.8 The measured effect went against developer beliefs: perceived and measured productivity are distinct quantities, and the 19% headline is about measured time.
- Window. The tools were early-2025 tools, and the authors now state: "We believe these historical results no longer reflect the current impact of AI models on open-source developer productivity."8 The result is a dated measurement, not a current estimate. Repeating "AI slows developers by 19%" without that window misreports the source.
Siemens, 24 October 2024: a vendor figure without a stated measure. "Engineers can now create panel visualizations in 30 seconds and generate code that requires only 20% adaptation."9 The sentence supplies a ratio with no defined numerator (lines changed? effort? defects?), no $n$, no task distribution, no baseline and no window. The absence of those details in this release is not evidence that the claim is false or that no supporting evidence exists. It means the figure's status is not checked. In particular it does not imply 80% correctness, 80% time saved, autonomous use or production safety. (Module 100.6 also uses Siemens; it is the same vendor, not an independent corroboration.)
The unsupported claim
In Johnson v. Dunn, a supervising partner asserted that this was the only lapse in over a decade of a colleague's work. The court: "This is a big statement about ten years of work-product, but it came with no citation or other basis".2 The general rule: a claim with no stated denominator, method or time window is an assertion, not a measurement.
Misconception corrected: "If an expert or a vendor says it, or it comes with a percentage, someone has measured it." Require $n$, the procedure and the window before repeating it. Where they cannot be obtained, record the claim as "not checked" and assign an owner for the follow-up.
Big Widget Company
A vendor claims "99% defect detection". Read as a catch rate (recall), that is $\text{TP}/(\text{TP}
- \text{FN})$: the denominator is the known defects, not all inspected parts. Ask for the unit (a
defective part or a single fault; one part can carry several), the number of known defects and the defect mix, how ground truth was established, and the false-positive count on sound parts: a system that flags everything has a recall of 100% and is useless. Ask also whether the test parts were independent of the parts the system was tuned on; who ran the test; and over what period and on what line conditions. Absent answers, record the claim as "vendor claim, not checked", with the quality lead as owner.
3. Synthetic testimonials and the FTC's 2024 rule
The rule
The FTC's Trade Regulation Rule on the Use of Consumer Reviews and Testimonials (16 CFR Part 465) was published on 22 August 2024 and took effect on 21 October 2024.10 Key elements:
- Definitions (§ 465.1). A consumer testimonial is an advertising or promotional message that consumers are likely to believe reflects the opinions, beliefs or experiences of a consumer who has purchased, used or otherwise had experience with the product, service or business. A reviewer is "the author or purported author of a consumer review."10
- Prohibition (§ 465.2(a)). It is an unfair or deceptive act or practice for a business to write, create or sell a consumer review, consumer testimonial or celebrity testimonial that materially misrepresents, expressly or by implication, that the reviewer or testimonialist exists, that they used or otherwise had experience with the product, service or business, or their experience with it.10 These are three alternatives joined by "or": a material misrepresentation on any one of them suffices.
- Knowledge-based duties (§ 465.2(b)–(c)). Purchasing such a review, disseminating such a testimonial, or procuring such reviews from officers, managers, employees or agents is prohibited where the business knew or should have known of the misrepresentation, with exceptions in § 465.2(d) for generalised solicitations and mere review hosting.10
- AI. "AI-generated reviews are covered by the final rule."10
- Remedy. The rule "will enable the Commission to seek civil penalties against violators."10
Why a synthetic testimonial is deceptive
A testimonial's informational value is its implied provenance: a real consumer with real experience. Take the paradigm case, a quote in which both the "customer" and the story are invented by a model. It misrepresents the existence of the testimonialist and the occurrence and content of their experience, so it engages more than one of the § 465.2(a) alternatives.10 Other fabrications engage only one, for example an invented experience attributed to a real person. The method of composition is not itself the test: AI authorship neither establishes nor excuses a misrepresentation, and the fluency of the text is irrelevant to the analysis.
The following is this course's application of that test, not text from the rule or an established safe harbour. Copy in the company's own voice makes no claim to be a consumer's experience. Editing a real, consenting customer's words, with the customer approving the final wording and the meaning staying theirs, does not on its face invent a reviewer or an experience. In each case the question is the one the rule asks: does the presentation materially misrepresent that the reviewer exists, that they had the experience, or what it was?10
Case: In the Matter of Rytr LLC, FTC Docket C-4806
On 25 September 2024 the FTC published for comment a complaint alleging that Rytr's AI writing service provided "the means and instrumentalities to deceive consumers" by generating content for consumer reviews, and that offering it was unfair. A consent order approved on 16 December 2024, over two commissioners' dissents, banned Rytr from offering a service for generating consumer reviews or testimonials.7
A later Commission order, citing the White House's July 2025 AI Action Plan, reopened the matter and set the 2024 order aside "in its entirety".7 Its reasoning: the complaint fitted none of the recognised means-and-instrumentalities situations, never alleged that Rytr "had actual or constructive knowledge that these reviews were used to violate Section 5 or even that the reviews were ever in fact published", and did not plead substantial injury for unfairness under § 5(n).7
Two precise readings follow. First, the set-aside concerns toolmaker liability under Section 5; it says nothing to legitimise publishing fabricated reviews, and states that "Where actors use AI to violate the law or deceive consumers about the capabilities of their generative AI, they should be held accountable".7 The business that writes or disseminates a fake testimonial is the subject of § 465.2.10 Second, the episode is a time-window lesson: the 2024 enforcement position did not endure. As of this module's date (November 2026), the review rule has not been re-checked for later amendment or challenge; verify its current status before relying on it.10
Misconception corrected: "AI-written praise is just marketing copy." Marketing copy speaks as the company. A testimonial speaks as a customer; fabricating one misrepresents a material fact.
Big Widget Company
Marketing may use AI to edit consenting customers' quotes, with the customer approving the final wording and a named marketing lead approving publication. Generating "customer" quotes that no customer gave, or experiences no customer had, is denied, whatever the tool.
4. The three yeses applied to verification
Get to Yes, Stay at Yes and Recover to Yes are a lifecycle, not a one-time approval. Verification is a necessary condition inside each, not sufficient authorisation on its own; the other gates in this course still apply.
- Get to Yes: before an AI output is used, name the verifier and the evidence they will open. Until verification is done the output is held; if it cannot be verified, it is recorded as "not checked" and not used for the decision.
- Stay at Yes: re-verify as tasks, tools and rules change; NIST frames measurement as continuing "as knowledge, methodologies, risks, and impacts evolve over time."3
- Recover to Yes: stop use, disclose and correct where the error was seen, name the owner, then change the control. The firm in Johnson v. Dunn told the court it would adopt a prefiling citation protocol.2 The order records that commitment; it is not evidence that the new control has worked. Claim recovery only on evidence that it has.
Records. Verification notes (sources opened, figures recomputed, the reviewer's name) belong in a separately controlled work record. In this course a receipt carries the decision and its metadata. It never carries prompts, model outputs, payloads or personal data.
Further learning (not copied; background only)
- Stanford University, CS324 — Large Language Models (Winter 2022), https://stanford-cs324.github.io/winter2022/ : advanced background on evaluating language models. No content licence was found; link only.