Level 500: Governance, safety and research · Module 500.1

Two colleagues compare two printouts while a third takes notes.
What was the baseline?

Reading and critiquing primary research · Technical gear

You will be able to: audit an AI paper's headline claim against its reported evidence by specifying the comparison baseline, the evaluation sample, the metric and its uncertainty, and the observation that would falsify the claim; classify a source as peer-reviewed paper, preprint, vendor report or system card, and state the epistemic warrant each status carries and lacks; and diagnose leakage, test-set contamination and construct-validity failures in a reported benchmark result.

This module is as of November 2026. Reviewed and published, version 1.0.0. It is not a credential. Sources and their publication status are in sources.yml. Of the nine research sources, five were read as preprints: s1, s5, s6 (arXiv v1; a Findings of EMNLP 2023 version exists and was not opened), s9 (arXiv abstract page only; s1 lists an ICLR 2018 version, not opened) and s8 (developer-authored). s2 is a publication record only. s10 and s11 are AiGovOps Foundation pages, cited only for the Foundation's own terms.

1. Auditing a headline claim

The claim is an inference, not an observation

An empirical ML paper reports measurements of the form "metric M of system A on sample S under protocol P", and then makes a claim that generalises beyond (M, S, P). The audit asks whether the reported measurements support the generalisation.

Lipton and Steinhardt (2018) identify four patterns that weaken that link: "Failure to distinguish between explanation and speculation"; "Failure to identify the sources of empirical gains"; "Mathiness: the use of mathematics that obfuscates or impresses rather than clarifies"; and "Misuse of language", including "suggestive definitions", "overloaded terminology" and "suitcase words".1 On the first, they observe that speculation offered "in the guise of explanations" is "interpreted as authoritative due to the trappings of a scientific paper".1 The paper is a position piece written for the ICML Machine Learning Debates workshop; it does "not purport to offer a full or balanced viewpoint".1

Four audit fields

FieldQuestionFailure it catches
BaselineIs the comparator tuned and evaluated under the same protocol?Gains from unequal tuning effort; missing trivial baselines
SampleWhat population is S drawn from, how large is it, and how was it split?Leakage; distribution mismatch; small-n noise
MetricHow is M computed, under which prompt/decoding/scoring/aggregation choices, with what variance?Protocol-sensitive scores; single-run point estimates
FalsifierWhat observation in (M, S, P) would contradict the claim, stated before reading the result?Unfalsifiable or post-hoc claims

The falsifier field is this course's addition, not a recommendation taken from the sources.

Baseline: equal tuning and trivial comparators

Melis, Dyer and Blunsom re-evaluated recurrent language-model architectures that had been "evaluated using differing code bases and limited computational resources", which they call "uncontrolled sources of experimental variation". Using "large-scale automatic black-box hyperparameter tuning", they conclude that "standard LSTM architectures, when properly regularised, outperform more recent models" on "language modelling benchmarks", including the Penn Treebank and Wikitext-2 corpora.2 Only the arXiv abstract page (v2, 20 November 2017) was read; it lists no journal reference.2 Lipton and Steinhardt's reference list cites the work at the International Conference on Learning Representations (ICLR), 2018.1 That version's text and review record were not opened: not checked. Scope the finding accordingly: recurrent architectures, those corpora, that tuning budget. It is not evidence that tuning explains claimed gains in general.

Lipton and Steinhardt interpret the case: published improvements "originally attributed to complex innovations in network architectures, were actually due to better hyper-parameter tuning".1 They also report a reading-comprehension dataset on which "73% of the questions can be answered by looking at a single sentence, while only 2% require looking at multiple sentences", and on which "simpler neural networks and linear classifiers outperformed complicated neural architectures".1 The remedy they name is the ablation study; they add that ablation "is neither necessary nor sufficient for understanding a method".1

Kapoor and Narayanan's civil-war reproduction shows the trivial-comparator check. In the four papers whose main findings they found invalid, correcting leakage left complex models (random forests, AdaBoost, gradient-boosted trees) not "substantively better than decades-old Logistic Regression models".3 For one paper, the AUC difference between AdaBoost and logistic regression "drops from 0.14 to 0.01"; for another, none of the models outperform "a baseline model that predicts the outcome of the previous year".3 The errors were: imputing training and test data together (L1.2), reusing a dataset imputed that way, and using proxies for the outcome as predictors (L2).3 The authors state that "none of these errors could have been caught by reading the papers".3

Metric: protocol sensitivity and variance

Biderman et al., developers of the lm-eval evaluation harness, report that "the precise string used for a given prompt, even down to whitespace usage, can significantly impact performance" and that the prompt "affects the model rankings and research conclusions".4 Their starkest case: on ARC-Easy, one open model "achieves an accuracy of 72.4±1.80%" with a cloze setup but "scores 26.5±1.78%" with an MMLU-style setup.4 The two setups differ in presentation and scoring: cloze scores the likelihood of each candidate answer as a completion, while the MMLU-style setup presents lettered options and scores the answer letter.4 The gap is therefore a protocol effect, not an isolated wording effect. They also note that MMLU computed by micro-averaging versus macro-averaging over its 57 subjects "affects results by several percentage points".4

On variance, they show that single-run point estimates and 95% confidence intervals over ten runs can lead "to very different conclusions about the relative performance of models", and that HumanEval "contains only 164 examples".4 Their harness reports bootstrapped standard errors by default; they note this measures a different kind of variation from re-running a stochastic model.4

Worked illustration (this course's arithmetic, not a figure from the sources). For accuracy $\hat p$ on $n$ independent items, the binomial standard error is

\mathrm{SE}(\hat p) = \sqrt{\frac{\hat p(1-\hat p)}{n}}.

At $\hat p = 0.5$ and $n = 164$, $\mathrm{SE} \approx 0.039$, so an approximate 95% interval for one model's accuracy is about $\pm 7.7$ percentage points. This treats the items as a sample from a wider population of similar problems; it says nothing about run-to-run sampling variance.

That band is the wrong yardstick for a gap between two models scored on the same items. Their errors are correlated. Let $b$ be the items only model A gets right and $c$ the items only model B gets right. The paired difference is $\hat d = (b-c)/n$, with

\mathrm{SE}(\hat d) \approx \frac{1}{\sqrt n}\sqrt{\frac{b+c}{n} - \hat d^{\,2}}.

It depends on the discordant items, not on either model's accuracy. Two models that disagree on few items can have a far narrower interval for their gap than $\pm 7.7$ points; models that disagree on most items can have a wider one. Use a paired analysis (McNemar's test, or a paired bootstrap over items).

Two different conclusions follow from a paired interval. If it includes zero, the evaluation has not established a gain. That is not evidence against a gain. Evidence against a meaningful gain needs a margin $\delta$ fixed in advance and an interval for $d$ that lies entirely below $\delta$. Even then the gain is rejected at the chosen confidence level, not ruled out with certainty. A wide interval means this evaluation did not resolve the claim; a small $n$ is one cause, but clustered items or an unpaired design can also widen it, so adding items alone need not fix it.

Falsifier: state it before reading the table

For each claim, write the predicate the result would have to satisfy if the claim were false, then check the table. State it with a margin and a precision, as above. For "complex models beat logistic regression", the falsifying observation is "after a leakage-free split, the paired interval for the AUC difference lies entirely below a meaningful margin $\delta$". A near-zero point estimate on its own meets only the weaker condition: the claimed gain is not established. Kapoor and Narayanan's corrected results report near-zero differences ("drops from 0.14 to 0.01" in one paper) and, in another, no model beating a previous-year baseline.3 This module has not checked the uncertainty around those corrected figures. Read them as removing the support for the claim; whether they rule out every meaningful gain is not checked. A claim whose falsifier cannot be stated in terms of (M, S, P) ("a milestone toward general AI") is not an empirical claim about the table; see section 3.

The misconception this corrects: "A result in a paper is a fact; the abstract says what the paper showed." A result is a measurement under a protocol; the abstract is the authors' generalisation of it. Audit baseline, sample, metric and falsifier against the tables and methods.

Worked example at Big Widget Company

Synthetic example for practice. A supplier's paper claims: "Our inspection model outperforms human inspectors at detecting weld defects." Its table:

SystemImagesAccuracy
Model500 (supplier plant)96%
Human inspectors (2)50 each91%

Logged as: "Vendor claim; table does not support 'outperforms human inspectors'; not checked on Big Widget data; next check: paired evaluation on Big Widget images." Owner and decision-maker: the quality lead. Gate answer until the paired evaluation is run and recorded: wait for a person (the claim is held, not relied on).

2. Source status and epistemic warrant

Peer-reviewed paper

Warrant: the manuscript passed a venue's review and acceptance process. Not warranted: correctness, reproducibility or that the reviewers ran anything.

Pineau et al. report the 2014 consistency experiment by the then programme chairs: 10% of submissions were reviewed by two independent committees, which "disagreed on 25.9% of papers" overall and, at a 22.5% acceptance rate, "on 57% of the list of accepted papers".5 The 57% has the accepted list as its denominator, not all submissions. The original 2014 write-up was not opened. Under the NeurIPS 2019 policy, code was expected "only for accepted papers, and only by the camera-ready deadline", and was "not expected to be used during the review process to decide on the soundness of the work".5 That is a norm, not an observation of reviewer behaviour. Of reviewer responses, 5,298 reported code provided; 2,255 reported looking at it; 1,315 found it useful in guiding the review.5 The survey measures looking, not execution; it gives no rerun rate. Lipton and Steinhardt add a selection effect: "The machine learning peer review process places a premium on technical novelty."1 Kapoor and Narayanan's invalidated civil-war papers were "published in top political science journals".3

Preprint

Warrant: a dated, versioned public claim by the authors (for example, "arXiv:1807.03341v2 [stat.ML] 26 Jul 2018").1 Not warranted: external review. This course treats a preprint as not peer-reviewed until a peer-reviewed version is located and opened.

Version identity matters. Status attaches to a version, not to a title:

WorkVersionsStatus handling
Lipton and SteinhardtarXiv 1807.03341v2 (2018, ICML debates workshop)1; ACM Queue 17(1):45–77, February 20196Only the Queue Crossref record was opened: container Queue, type "journal-article", nothing on review.6 Its genre and review status are not checked. Do not transfer status between versions.
Kapoor and NarayananarXiv 2207.07048v1 (14 July 2022); Patterns 4(9), September 20233Figures in this module are from v1. The journal version was not opened.
Sainz et al.arXiv 2310.18018v1 (27 October 2023); Findings of the Association for Computational Linguistics: EMNLP 2023, December 2023, pp. 10776–107877Quotes are from v1. Only the ACL Anthology record of the Findings version was opened; its text may differ.
Biderman et al.arXiv 2405.14782v3 (31 May 2026)4Cite the version read; earlier versions may differ.
Melis et al.arXiv 1707.05589v2 (20 November 2017)2; ICLR 2018, as listed in s1's references1Abstract page only; it lists no journal reference. The ICLR version's text and review record were not opened: not checked.

Vendor report

Warrant: the developer's disclosure. Not warranted: independent measurement. The developer report cited here (s8) "contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar".8 Separate two kinds of reproduction. Training reproduction: the report alone does not disclose enough to attempt it. Output-level retests may be: if the same model version is accessible, an outsider can re-run some evaluations. But Biderman et al. note that many APIs "do not expose output log-probabilities", forcing generation-based evaluation with "additional variance".4 Each retest needs a pinned model version, prompts, decoding settings and items, and supports only a claim about that version under that protocol.

Biderman et al. describe the structural problem: for models never released in any form, the only evaluation numbers are those the developer chose to publish, and "the field has, in effect, adopted a norm where model developers grade their own homework."4 They also report that published MMLU and ARC numbers for several model families appear to use an MMLU-style prompt that is "not documented to the best of our knowledge in technical reports".4

System card

Warrant: the developer's account of hazards, evaluations and mitigations. Not warranted: that the mitigations are effective, or external audit. The system card appended to s8 highlights "safety challenges presented by the model's limitations ... and capabilities", gives "a high-level overview of the safety processes" the developer adopted, and states that its mitigations "are limited and remain brittle in some cases".8

The misconception this corrects: "Peer review, or an arXiv number, means the claim is true." Each status certifies a process (acceptance, posting, disclosure), not the result. The checks in sections 1 and 3 apply to every status.

At Big Widget Company

The governance team's source register records, per source: identifier and version, status (one of the four above), author affiliation relative to the system evaluated (developer, independent, unknown), whether code, prompts and outputs are available, what was not checked, and the named owner of the decision that relies on it. The weld paper is logged "vendor report; developer-authored; no code or data; not independently replicated".

3. Diagnosing benchmark results

Raji et al. define a benchmark as "a particular combination of a dataset or sets of datasets (at least test data, sometimes also training data), and a metric", taken up by a research community "as a shared framework for the comparison of methods".9

Leakage

Kapoor and Narayanan: "Data leakage is a spurious relationship between the independent variables and the target variable that arises as an artifact of the data collection, sampling, or pre-processing strategy", which "usually leads to inflated estimates of model performance."3 Their survey found "20 papers from 17 fields that outline errors in ML-based science in their field, collectively affecting 329 papers", with leakage "a pitfall in every single case"; they present this as "a lower bound", not "a systematic meta-review".3

Their taxonomy:3

CodeTypeMechanism
L1.1No test setTraining and evaluating on the same data
L1.2Pre-processing on training and test setE.g. imputation or oversampling before the split
L1.3Feature selection on training and test setSelection uses test-set performance
L1.4Duplicates in datasetsSame records in both splits
L2Illegitimate featuresE.g. a proxy for the outcome (anti-hypertensive drug use as a feature for predicting hypertension)
L3.1Temporal leakageTest data from before the training data when predicting the future
L3.2Non-independence between train and testE.g. the same patients in both splits
L3.3Sampling bias in test distributionE.g. spatial or selection bias in the test set

They note that mitigations from modelling competitions do not transfer: in competitions, impartial third parties hold the evaluation set, whereas "in most ML-based science the researcher has access to the entire dataset while creating the ML models."3 Their proposed control is the model info sheet, which requires "precise arguments to justify that models used towards making scientific claims do not suffer from leakage".3

Contamination

Contamination is leakage of evaluation data into a language model's training data. Sainz et al.: "The worst kind of data contamination happens when a Large Language Model (LLM) is trained on the test split of a benchmark, and then evaluated in the same benchmark"; it "causes an overestimation of the performance of a contaminated model", and "the extent of the problem is unknown, as it is not straightforward to measure."7 They distinguish:7

They locate contamination at pre-training, supervised fine-tuning and post-deployment iteration, and call for a registry of contamination cases and for venues to flag affected papers.7

The s8 report's Appendix C is a developer-run detector you can audit. Evaluation and training text are normalised by removing spaces and symbols; for each evaluation example, three random 50-character substrings are sampled (or the whole example if shorter); a match on any one marks the example contaminated; contaminated examples are discarded and scores recomputed.8 The report states that the method "can result in false negatives ... as well as false positives", that answers and sometimes options are excluded from matching, and that the post-training (RLHF) data was not explicitly checked.8 For exams, the main text runs a variant with flagged questions removed and reports "the lower score of the two".8 Keep three quantities apart:

Separately, "portions of BIG-bench were inadvertently mixed into the training set", and training data from MATH and GSM-8K was mixed in deliberately.8 Sainz et al. draw the consequence for GSM-8K: those results "cannot be taken as zero-shot results when compared to other models".7

Audit questions for any contamination claim: which corpus was searched (pre-training only, or also fine-tuning and post-training data); which fields of each test item were matched; what matching rule and threshold; what the false-negative modes are (paraphrase, translation, formatting) and what the false-positive modes are; and who ran the check. Distinguish an exposure risk (the item is public), a detector flag and verified exposure.

Construct validity

Raji et al. define construct validity for ML as "how well the benchmark dataset, and associated metrics of evaluation, represents a task".9 They argue that "general" benchmarks such as ImageNet and GLUE/SuperGLUE, being "inherently specific, finite and contextual", "cannot possibly capture anything representative of the claims to general applicability being made about them".9 Their marketing example: a large technology company's January 2021 statement that its model "surpassing human performance on SuperGLUE marks an important milestone toward general AI."9 They quote Bowman and Dahl that "we know of no simple test that would allow one to determine if a benchmark presents a valid measure of model ability".9

They recommend scoping benchmarks "to evaluate concrete, well-scoped and contextualized tasks", and, for broader claims, alternative methods: systematic test suites, audits and adversarial testing; error, disaggregated and counterfactual analysis; ablation; and measurement of properties such as energy use and stability under training-data perturbation.9

Biderman et al. reach the same conclusion from the evaluator's side: benchmarks "are typically proxies for some broader notion of capability", and "the appropriate conclusions to draw about a model's behavior are informed by, not constituted by, these numerical scores."4 They describe the "Key Problem": the best automated tools for judging semantic equivalence "are the very models we are seeking to evaluate". One response is to restrict the answer space, most often by recasting questions as multiple choice, so correctness can be checked mechanically; they note that popular knowledge benchmarks' set-ups are much more like human exam questions "than how people actually use language models".4

Lipton and Steinhardt's dermatology example is the same failure in an applied claim: the classifier "only achieves low error on i.i.d. test data", while dermatologists face shifting real-world conditions.1

Diagnostic checklist

FailureEvidence to requestRed flag
LeakageSplit unit and time order; pipeline order (fit transforms on train only); feature provenanceRandom row-level split of grouped or temporal data; preprocessing before split; features recorded after the outcome
ContaminationCorpora searched; fields matched; matching rule; per-item detector flag rate, stated as a detector output with its known false-negative and false-positive modes; full, filtered and reported scores"No contamination found" without method; only pre-training data checked; a flag rate presented as verified exposure, or a filtered score presented as clean
Construct validityTask definition; similarity of items, conditions and scoring to deploymentClaims of general ability from one benchmark; "human-level" without the human protocol

This checklist is this course's synthesis of the sources above.

The misconception this corrects: "A higher benchmark score means the model is better at the real job." A score difference is evidence about the job only to the extent that the test is held out, uncontaminated and valid for that job, and the difference exceeds its (paired) uncertainty.

At Big Widget Company

The weld paper's methods section, read against the checklist:

4. Controls you can require

ControlSource recommendation
Evaluation code"Release of the exact evaluation code used"4
Prompts and settingsPrompts "clearly reported within a publication"; full prompts are "necessary but not sufficient" for reproducibility4
Uncertainty"Reporting variance over several runs, or reporting statistical significance"4
Leakage argumentModel info sheets3
ReproductionDistinguish reproducible (same data, same tools), replicable (different data), robust (same data, different analysis) and generalisable (different data and analysis)5

Two cautions. Pineau et al. note that a copy of the original code "is not a guarantee that this code is correct", and report that they "do not have concluding evidence" the NeurIPS 2019 programme improved paper quality.5 And Lipton and Steinhardt's reviewer test applies to anyone relying on a paper: "Might I have accepted this paper if the authors had done a worse job?"1

The AiGovOps Foundation's manifesto maps its gates to three movements, Get to Yes, Stay at Yes and Recover to Yes, and gives a gate four answers: "go ahead · only this far · wait for a person · no".10 The mapping below onto research reading is this course's application. The reader audits and recommends; a named owner decides whether, and how far, a result is relied on.

What you keep. Audit notes (versions opened, checks run, what is not checked, who decided) belong in a separately controlled work record. The Foundation's receipt is gate metadata: it carries "hashes, decision, tier, policy hash, key id" and never carries "prompts, outputs, payloads, personal data".11 It evidences that a decision was recorded, not that the research it relied on is correct. Where a gap remains, say so: on the Foundation's Practice site, "I don't know yet" is always an answer, and it always pairs you with someone.11

Further learning (optional, not citations)

Links only; nothing is copied from these courses and they support no claim above.

Sources

  1. s1 Zachary C. Lipton and Jacob Steinhardt (2018). Troubling Trends in Machine Learning Scholarship. Paper. Checked 2026-11-20 by source-checker agent (Claude Opus 5.5, round-4 re-check session separate from the lesson writer and item writer), 20 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  2. s9 Gábor Melis, Chris Dyer and Phil Blunsom (2017). On the State of the Art of Evaluation in Neural Language Models. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-3 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩
  3. s3 Sayash Kapoor and Arvind Narayanan (2022). Leakage and the Reproducibility Crisis in ML-based Science. Paper. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, session separate from the lesson writer and item writer), 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  4. s5 Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Baber Abbasi, Jessica Zosa Forde, Leo Gao, Jonathan Tow, et al. (30 authors; full roster in the source pack) (2024). Lessons from the Trenches on Reproducible Evaluation of Language Models. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  5. s7 Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché-Buc, Emily Fox and Hugo Larochelle (2021). Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩ ↩ ↩
  6. s2 Zachary C. Lipton and Jacob Steinhardt (2019). Troubling Trends in Machine Learning Scholarship. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩
  7. s6 Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle and Eneko Agirre (2023). NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩ ↩ ↩
  8. s8 OpenAI et al. (2023). GPT-4 Technical Report (including the appended GPT-4 System Card). Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩
  9. s4 Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton and Alex Hanna (2021). AI and the Everything in the Whole Wide World Benchmark. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩
  10. s10 AiGovOps Foundation (n.d.). AiGovOps Foundation Manifesto. Official doc. Checked 2026-11-20 by source-checker agent (Claude Opus 5.5, round-4 re-check session separate from the lesson writer and item writer), 20 Nov 2026; live page as served that day ("Fall 2026 Release", no date on the page). ↩
  11. s11 AiGovOps Foundation (n.d.). AiGovOps Foundation Practice (home page). Official doc. Checked 2026-11-20 by source-checker agent (Claude Opus 5.5, round-4 re-check session separate from the lesson writer and item writer), 20 Nov 2026; live page as served that day (footer "practice v1.0.0-rc.1", "Fall 2026 Release"). ↩ ↩

Next: the quiz