Level 200: Using gen AI well · Module 200.4

Three coworkers compare small cards spread across a café table.
Try the awkward one

Testing output before you trust it · Technical gear

Reviewed and published, version 1.0.0. As of 18 Nov 2026. Not a credential. Research results are dated in the text; the models measured in them are 2023 models.

Objectives. 200.4-a: build a task-specific test set from the user's side (realistic inputs with known good answers, including hard cases; synthetic, or approved and redacted) and score outputs against a written rubric of correctness and usability. 200.4-b: explain the documented biases of LLM-as-a-judge evaluation (position, verbosity, self-enhancement) and decide when human evaluation is required. 200.4-c: set checking intensity from the cost of an error and who bears it, from sampled spot checks to full review of every output.

Scope. This is black-box, user-side evaluation: inputs in, outputs scored. Stage-level tracing (which of source, retrieval or generation failed) is Level 300. Big Widget Company is fictional.

1. Anchor case: Mata v. Avianca (S.D.N.Y. 2023)

An AI grader may favor a position or a longer answer. The study did not establish self-preference. Compare with a person’s scores and leave the decision with a person.
Zheng and colleagues documented position and length effects and did not determine whether the models tested favored their own answers (s1). The course keeps the decision with a person.
Figure data as text

Limits from one study [s1]

PartWhat it means
POSITIONOrder can change the verdict. The grader may favor one position.
LENGTHMore words can sway the grade. Even padding can beat a shorter answer.
POSSIBLE SELF-PREFERENCENot established by this study. The authors could not tell if the tested models favored their own answers.
COURSE PRACTICECompare scores with a person’s scores. A person makes the final decision.

In the Opinion and Order on Sanctions of 22 June 2023 (No. 22-cv-1461 (PKC), ECF 54), Judge P. Kevin Castel found that an "Affirmation in Opposition" filed on 1 March 2023 cited and quoted purported decisions said to be published in the Federal Reporter, the Federal Supplement and Westlaw; the respondents later acknowledged that six of them "were generated by ChatGPT and do not exist".1 The record maps onto each objective of this module.

The court was explicit that tool use was not itself the violation: "there is nothing inherently improper about using a reliable artificial intelligence tool for assistance. But existing rules impose a gatekeeping role on attorneys to ensure the accuracy of their filings."1 It imposed a $5,000 penalty jointly and severally and ordered letters to the client and to each judge falsely identified as the author of a fake opinion.1 The bad-faith finding rested on "acts of conscious avoidance and false and misleading statements to the Court", not on the initial filing alone.1

Misconception corrected: that a handful of plausible outputs validates a tool. Every fake case here looked plausible; the drafting attorney described "various indicia of reliability such as case captions, the names of the judges from the correct locations, and detailed fact patterns and legal analysis that sounded authentic". The court called that claim "highly dubious" and the declaration that contains it one of his "shifting and contradictory explanations".1 It is used here only as his description of how plausible the fabrications looked.

2. Task-specific test sets and rubrics

2.1 Why ad hoc trials and public benchmarks are insufficient

NIST AI 600-1 (July 2024), appendix A.1.4, describes current pre-deployment test, evaluation, verification and validation (TEVV) for generative AI as possibly "inadequate, non-systematically applied, or fail to reflect or mismatched to deployment contexts".2 It notes that testing "often remain[s] focused on laboratory conditions or restricted to benchmark test datasets and in silico techniques that may not extrapolate well", and that benchmark-to-deployment gaps "may likely be exacerbated due to prompt sensitivity and broad heterogeneity of contexts of use."2 The same appendix gives a parallel case about capability claims: "anecdotal testing of GAI system capabilities through video games or standardized tests designed for humans (e.g., intelligence tests, professional licensing exams) does not guarantee GAI system validity or reliability in those domains."2 NIST's point is about benchmarks, lab conditions and human exams. Extending it to a user's handful of trials, which are a small, unstratified sample judged on surface plausibility, is this course's inference. Action MP-2.3-001 states the alternative: "Assess the accuracy, quality, reliability, and authenticity of GAI output by comparing it to a set of known ground truth data and by using a variety of evaluation methods (e.g., human oversight and automated evaluation, proven cryptographic techniques, review of content inputs)."2

HELM (Liang et al., 2022; TMLR 2023) gives the measurement rationale. It evaluated models on 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) across 16 core scenarios, with the stated aim of "ensuring that metrics beyond accuracy don't fall to the wayside".3 Its "multi-metric measurement" element holds that evaluation should be "evaluating every desideratum for each scenario considered", and the authors argue that "it is integral that all of these metrics be evaluated in the same contexts where we expect to deploy models."3 A user-side test set is the smallest version of that idea: one scenario (your task), several criteria, in your deployment context.

2.2 Construction

Let the test set be T = {(x_i, a_i)} for i = 1..n, where x_i is a realistic input and a_i is a reference: either a known good output or the set of facts a correct output must contain, written or approved by a domain expert. This course's construction rules:

2.3 Rubric and scoring

Write the rubric R = {r_1..r_k} before any output is seen, as binary criteria a person can apply without judgment calls. Split them into correctness criteria (required facts present and exact; no fabricated entity, number, cause or citation; abstains when the input lacks the answer) and usability criteria (length, format, required action stated). Mark criteria whose failure is critical (for example a fabricated citation or a wrong safety value): any critical failure fails the item regardless of the rest.

Score s_ij in {0, 1} for output i on criterion j. Report per-criterion pass rates, the critical-failure count and the failing items themselves, with notes. An aggregate score alone hides what matters: a summary set can score 95% on average while every failure is a fabricated cause.

2.4 What a small clean run does and does not show

The following is this course's arithmetic, not a sourced result. Suppose outputs fail independently with true per-item failure probability p. The chance that n items all pass is (1 - p)^n. With p = 0.10 and n = 10, that is 0.9^10 ≈ 0.35; with p = 0.05 and n = 20, 0.95^20 ≈ 0.36. So a clean run of 10 or 20 items is entirely consistent with a tool that fails one time in ten or one in twenty. A small test set is good at finding kinds of failure, especially on deliberate hard cases; it cannot certify a low failure rate. That is why 200.4-c still sets checking for live outputs after a test passes.

At Big Widget Company

Ana's quality team evaluates the approved AI tool for one-paragraph defect-report summaries. R: (1) part number and defect exact [critical]; (2) affected-unit count exact [critical]; (3) cause stated only if the report states one, else "cause not yet known" [critical]; (4) ≤ 80 words, ending with the requested action. T: 12 synthetic reports with engineer-written references: 8 typical; 4 hard (two defects; missing count; technician's guess marked as a guess; duplicated part number with a typo). Result: 8/8 typical pass; 3/4 hard fail critically (second defect dropped; count fabricated as "approximately 40 units"; guess promoted to cause). The record names the failure types, which become the checking targets in section 4.

3. LLM-as-a-judge: agreement and biases

Zheng et al. (2023; NeurIPS 2023 Datasets and Benchmarks) studied strong LLMs as judges of chat responses. Headline: "strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well", "achieving over 80% agreement, the same level of agreement between humans."4 The same paper examines "position, verbosity, and self-enhancement biases," "as well as limited reasoning ability".4 All figures below are for judges tested in 2023 (GPT-4, GPT-3.5, Claude-v1); they show that the failure modes exist, not their size in current models.

Position bias — "when an LLM exhibits a propensity to favor certain positions over others."4 Table 2 reports consistency, "the percentage of cases where a judge gives consistent results when swapping the order of two assistants".4

Judge (2023)Consistency, default promptBiased toward first, default promptConsistency, "rename" prompt
Claude-v123.8%75.0%56.2%
GPT-3.546.2%50.0%51.2%
GPT-465.0%30.0%66.2%

"Only GPT-4 outputs consistent results in more than 60% of cases," and "Most LLM judges favor the first position."4

Verbosity bias — when a judge "favors longer, verbose responses," "even if they are not as clear, high-quality, or accurate as shorter alternatives."4 In the "repetitive list" attack, 23 MT-bench answers containing a numbered list were made "unnecessarily verbose by asking GPT-4 to rephrase the list without adding any new information"; an attack succeeds "if an LLM judge thinks the new response is better than the old response." Table 3 failure rates: Claude-v1 91.3%, GPT-3.5 91.3%, GPT-4 8.7% (that is, 21/23, 21/23 and 2/23). The authors conclude "all LLMs may be prone to verbosity bias though GPT-4 defends significantly better than others."4

Self-enhancement bias — "the effect that LLM judges may favor the answers generated by themselves."4 Observed: "GPT-4 favors itself with a 10% higher win rate; Claude-v1 favors itself with a 25% higher win rate." But: "they also favor other models and GPT-3.5 does not favor itself." The authors' own limit: "Due to limited data and small differences, our study cannot determine whether the models exhibit a self-enhancement bias," partly because "we cannot easily rephrase a response to fit the style of another model without changing the quality."4 Treat it as an unquantified risk; design it out rather than estimate it.

Limited reasoning — GPT-4 as judge "shows limitations in grading basic math problems which it is capable of solving."4 On 10 math questions (Table 4), failures fell from 14/20 with the default prompt to 6/20 with chain-of-thought and 3/20 with a reference-guided prompt, "from 70% to 15%" in the authors' words.4 In the paper the reference is not ground truth: it is the judge's own answer, generated independently and then shown in the judge prompt; and the larger drop came from chain-of-thought (14 to 6), with the reference-guided prompt cutting failures further (6 to 3).4 Giving the judge a worked answer to compare against helped, which is one reason this course gives a judge the test set's expert-written references (section 3.1).

Mitigations reported. For position bias, "A conservative approach is to call a judge twice by swapping the order of two answers" and "only declare a win when an answer is preferred in both orders"; "If the results are inconsistent after swapping, we can call it a tie."4 Few-shot prompting raised GPT-4's consistency "from 65.0% to 77.5%".4 Note that 77.5% still leaves more than one comparison in five order-dependent.

3.1 Designing a judge-assisted test

This course's protocol, built on the findings above:

  1. Pointwise rubric grading over pairwise preference. Give the judge the rubric and the reference a_i and ask for a 0/1 per criterion. This removes position from the decision and makes length a scored criterion rather than a latent preference.
  2. If pairwise is unavoidable, run both orders and score inconsistent pairs as ties.4
  3. No self-grading. The judge model is not the generator model. (In Mata the generator vouched for its own citations, while his pre-filing external lookup had come back empty.)1
  4. No judge-only scoring of facts it cannot verify (citations, external records) or of arithmetic, given the reasoning limits above; a person checks those against the source.4
  5. Calibrate against people. Have a person score a sample of the same items and compute the judge's agreement with that person per criterion. Zheng et al. validated judges in the same way, by agreement with human judgments.4 If agreement on a criterion is poor, that criterion goes to people.
  6. The judge advises; a person decides. Judge scores feed a named person's decision on the tool; they are not the decision.

3.2 When a person must check instead

Human evaluation is required, not optional, when any of these hold: the output's error cost is high or falls on third parties (section 4); the criterion is factual or source-based and the judge has no reliable access to ground truth; the judge and generator are the same model; or judge-human agreement on this task and criterion has not been measured. Zheng et al. themselves describe human evaluation as "the gold standard", limited by being "exceptionally slow and costly".4 The judge reduces cost; it does not move the gold standard.

Misconception corrected: that an AI second opinion is equivalent to a human check. High average agreement coexists with systematic, directional errors (order, length, possibly self), and those are exactly the errors a test needs to be immune to.4

At Big Widget Company

Comparing two prompts, Ana's pairwise judge prefers prompt B. B's outputs are longer. She switches to pointwise rubric grading with the reference summaries, runs any remaining pairwise comparisons in both orders, and has an engineer score 20 items to measure judge-human agreement per criterion. Under the rubric, B fails criterion 4 (length) and criterion 3 (adds an unstated cause); A is selected. Criterion 3 stays human-checked because it is a factual judgment against the source report.

4. Checking intensity from error cost and incidence

NIST AI 600-1 uses the AI RMF definition of risk: "the composite measure of an event's probability (or likelihood) of occurring and the magnitude or degree of the consequences of the corresponding event."2 It says organisations "may additionally wish to allocate risk management resources relative to the severity and likelihood of negative impacts".2 GV-3.2-001 asks for independent evaluations "where the type and robustness of evaluations are proportional to the identified risks", and GV-1.3-001 lists factors for risk tiers including "Harm to fundamental rights or public safety" and "Presentation of ... invalid or untruthful output".2 MP-2.3-003 adds: "Deploy and document fact-checking techniques to verify the accuracy and veracity of information generated by GAI systems".2

4.1 A simple decision model

The following is this course's simplification, not a NIST formula. For one task, let p be the per-output probability of a material error (estimated from your test set's failure types, knowing from 2.4 that a small set bounds it only loosely), C the cost of an error that reaches its audience, and c the cost of a person checking one output. Checking every output is worth it when roughly p × C > c. Two adjustments matter more than the arithmetic:

4.2 The checking ladder

RungHuman checkingUse when
Spot checkPeriodic sample of outputsC small, borne internally, errors easy to notice
Targeted checkEvery output, on the criteria the test set showed to failC moderate; failure types known
Full checkEvery output, every criterion, against sourceC high, or borne by third parties
Dual control or no useFull check plus a second named sign-off, or the tool is not used for the taskSafety, health, legal rights or broad reach

A court filing belongs on the top rung, dual control: legal rights are at stake and the cost falls on third parties, so every citation is verified against the primary source and a second named person signs off. That placement is this course's; the court's framing is the attorneys' "gatekeeping role".1 In Mata the second signatory existed: the signing attorney's review covered style, not authorities.1 Re-run the test set after any model, prompt or configuration change, and move a task down a rung only on evidence from that re-run.

Misconception corrected: uniform checking. Uniform effort over-checks cheap outputs and under-checks costly ones; the court's harm list in Mata shows how an under-checked output's cost propagates to people who never saw the draft.1

At Big Widget Company

5. The three yeses as gate points

The Foundation's three movements are Get to Yes ("before a single action runs"), Stay at Yes ("while it runs, in real time") and Recover to Yes ("after — on demand, on incident"), and a gate decides "go ahead · only this far · wait for a person · no".5 Mapping them onto tool evaluation is this course's application, not a quotation.

MovementEvaluation controlsGate outcomes (decided by a named person, never by the judge model)
Get to Yes (before reliance on a task)Rubric with critical criteria; frozen, stratified test set; scored run; judge-human agreement if a judge is used. Unknown tool version or configuration is recorded as "unknown", with who can resolve itAll critical criteria pass, including hard cases: go ahead at the chosen rung. Some criteria fail: only this far (use the tool for the passing criteria; a person checks the rest on every output). Ambiguous results or poor judge-human agreement: wait for a person, named. Critical failures on a top-rung task: no
Stay at Yes (each live output)Checking at the chosen rung; re-run T after any model, setting or prompt change; a rung moves down only on re-run evidenceA re-run that fails a critical criterion re-opens Get to Yes
Recover to Yes (after a wrong output was relied on)Stop relying on it; notify the tool owner and those affected; add the failing input to T as a hard case; fix prompt, rung or toolClaim recovery only after the re-run passes. Withdrawn is stopped, not recovered

Evidence versus receipts. The evaluation record holds inputs, outputs and scores. Even with synthetic or redacted inputs, it is a controlled work record kept under the organisation's data rules. The Foundation's signed receipt is a separate object that "holds no private content";5 it carries "hashes, decision, tier, policy hash, key id" and never "prompts, outputs, payloads, personal data".6 A receipt shows that a decision was made under a policy; it does not show that the tool's outputs were correct.

6. Synthesis

The artifact this module produces is a dated evaluation record: task, tool and configuration (as far as the user can see), rubric with critical criteria, frozen test set (synthetic or approved and redacted), per-criterion results and failing items, judge protocol and judge-human agreement if a judge was used, the chosen checking rung with its reasoning, the gate answer and who gave it, and anything recorded as "unknown". Level 200's project builds on this record; Level 300 adds stage-level traces to explain failures the record only detects.

Terms: ground truth, reference answer, test set stratification, abstention, rubric, critical criterion, pointwise vs pairwise judging, position consistency, verbosity bias, self-enhancement bias, judge-human agreement, checking ladder, dual control, gate, signed receipt.

Further learning (optional, never a citation for this module):

Sources

  1. s4 P. Kevin Castel, U.S. District Court for the Southern District of New York (2023). Mata v. Avianca, Inc., No. 22-cv-1461 (PKC), Opinion and Order on Sanctions (docket entry 54). Record. Checked 2026-11-18 by source-checker role, Devin session f9ae423f (did not write, revise or red-team this module), remaining-sources check, 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  2. s3 Chloe Autio et al. (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). Standard. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the writer and reviser), 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s2 Percy Liang et al. (2022). Holistic Evaluation of Language Models. Paper. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the writer and reviser), 18 Nov 2026. ↩ ↩
  4. s1 Lianmin Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Paper. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the writer and reviser), 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  5. s5 AiGovOps Foundation (n.d.). AiGovOps Foundation Manifesto. Official doc. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the writer and reviser), 18 Nov 2026; page opened 18 Nov 2026, labelled "Fall 2026 Release", no numeric version or date shown. ↩ ↩
  6. s6 AiGovOps Foundation (n.d.). AiGovOps Foundation Practice (home page). Official doc. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the writer and reviser), 18 Nov 2026; page opened 18 Nov 2026, footer "practice v1.0.0-rc.1", "Fall 2026 Release". ↩

Next: the quiz