Level 500: Governance, safety and research · Module 500.1

Two colleagues compare two printouts while a third takes notes.
What was the baseline?

Reading and critiquing primary research · Standard gear

You will be able to: test an AI paper's headline claim against its own evidence by naming the baseline, the sample, the metric and what the result would look like if the claim were false; say whether a source is a peer-reviewed paper, a preprint, a vendor report or a system card, and what each status does and does not establish; and spot leakage, test-set contamination and construct-validity problems in a benchmark result before you rely on it.

This module is as of November 2026. Reviewed and published, version 1.0.0. It is not a credential. Every source and its status is listed in sources.yml. Where we cite a preprint, we say so.

1. The headline is a claim, not a finding

A paper's abstract is the authors' summary of what they think they showed. It is written by the people with the most to gain from the result. Your job at this level is to read the claim, then find the evidence for it inside the paper, and check that the two match.

Two machine learning researchers, Zachary Lipton and Jacob Steinhardt, described in 2018 how this goes wrong. They wrote that "papers often offer speculation in the guise of explanations, which are then interpreted as authoritative due to the trappings of a scientific paper and the presumed expertise of the authors."1 They named four patterns they saw spreading in the field: failing to separate explanation from speculation; failing to identify where empirical gains came from; "mathiness", maths that "obfuscates or impresses rather than clarifies"; and misuse of language.1 Their paper is itself a position piece, written for a workshop debate (section 2 explains why that matters).1

Four questions for every headline claim

Before you repeat a result, ask four questions of the paper's own tables and methods.

  1. The baseline: better than what? "Our model is better" only means something against a fair comparison. Was the comparison model tuned as carefully as the new one?
  2. The sample: measured on what? Which data, how many examples, drawn from where, and how much like the setting the claim is about?
  3. The metric: scored how? Which number, computed how, out of how many, from how many runs, with what uncertainty?
  4. The falsifier: what would "false" look like? Before you look at the result, say what the table would show if the claim were wrong. If no possible table could show that, the claim is not being tested.

The fourth question is this course's working rule, not a quotation from the sources. It is the one that turns reading into checking.

Real case: a weak baseline

Gábor Melis, Chris Dyer and Phil Blunsom re-tested several recurrent neural network designs for language modelling. Earlier results, they note, had been "evaluated using differing code bases and limited computational resources", which are "uncontrolled sources of experimental variation". After "large-scale automatic black-box hyperparameter tuning" of each design, they concluded that "standard LSTM architectures, when properly regularised, outperform more recent models" on the language modelling benchmarks they used, including the Penn Treebank and Wikitext-2 corpora.2 We read only the paper's arXiv abstract page (version 2, 20 November 2017), which lists no journal or conference version.2 Lipton and Steinhardt's reference list, however, cites the paper as appearing at the International Conference on Learning Representations (ICLR) in 2018.1 We have not opened that version, so its text and its review record are not checked. The version we quote is the arXiv preprint.

Lipton and Steinhardt use this case in their critique: a series of published improvements, "originally attributed to complex innovations in network architectures, were actually due to better hyper-parameter tuning."1 Keep the finding inside its bounds. It is about the recurrent models, benchmarks and tuning budget that Melis and colleagues tested. It does not show that tuning explains every claimed gain, for every design or task. Lipton and Steinhardt's wider complaint: "Too frequently, authors propose many tweaks absent proper ablation studies, obscuring the source of empirical gains."1 An ablation removes one change at a time to see which one actually made the difference.

Real case: the claim that failed to reproduce

In 2022, Sayash Kapoor and Arvind Narayanan of Princeton re-examined published papers on predicting civil wars. The field believed complex machine learning models "vastly outperform" logistic regression, a decades-old statistical method.3 Run the four questions on what they found.

These papers had been published in top political science journals. The authors' sharpest line: "none of these errors could have been caught by reading the papers".3 Reading the abstract was never going to be enough. Neither, in this case, was reading the whole paper; the errors were found in the code and data.

Real case: the same model, two ways of running the test

The metric question covers how a number was produced, not just what it is called. Stella Biderman and colleagues, who build and maintain a popular evaluation library, report that one open model scored 72.4% on a multiple-choice benchmark (ARC-Easy) under one evaluation setup and 26.5% under another.4 The model and the questions were the same. What changed was how the questions were presented and how answers were scored. In the "cloze" setup, each candidate answer is scored as a completion of the question. In the "MMLU-style" setup, the options are listed with letters and the model's answer letter is scored.4 So the gap is not a pure effect of wording. It shows that the result depends on how the test was run.

They also warn that "the precise string used for a given prompt, even down to whitespace usage, can significantly impact performance", and that the choice of prompt changes how models rank against each other.4 And they warn against single runs: a benchmark such as HumanEval has "only 164 examples", and small "improvements" on it "might be washed out simply via sampling again".4 Their paper is a preprint (version 3, May 2026).4

The misconception this corrects: "A result in a paper is a fact; the abstract says what the paper showed." A result is a measurement under particular conditions. The abstract is the authors' reading of it. Check the baseline, the sample, the metric and the falsifier yourself.

At Big Widget Company

Made-up example for practice. A supplier sends the quality team a paper. Its abstract says: "Our inspection model outperforms human inspectors at detecting weld defects." The results table shows the model at 96% accuracy on 500 photographs from the supplier's own plant, and "human baseline: 91%", with a footnote saying two inspectors each labelled 50 of the photographs.

Be careful with a near tie. A small gap with wide uncertainty does not show the model is no better. It means this test did not resolve the claim. The reason is not always too few photos: photos that come in batches from the same line, or a comparison that is not paired, can also leave the range wide. Only a test whose range sits below a gain that matters counts as evidence against the claim.

The team's note: "Vendor claim. The table does not support 'outperforms human inspectors'. Not checked on our welds. Next check: model and inspectors on the same photos." The quality lead owns the next check and the decision. Until the check is run, the claim waits for that person; it is not used. The reader recommends; the owner decides.

2. What the label on a source does and does not tell you

Sources arrive with labels: a conference paper, an arXiv number, a technical report, a system card. Each label tells you something about how the document came to exist. None of them tells you the claim is true.

Peer-reviewed paper

What it establishes: reviewers chosen by a journal or conference read the paper and the venue accepted it.

What it does not establish: that the result is right, or that anyone re-ran it.

Review is not a rerun and not proof. It is one check, by a few people, under time pressure.

Preprint

What it establishes: the authors made this claim publicly, in this version, on this date. On arXiv, the version and date are stamped on the paper: Lipton and Steinhardt's reads "arXiv:1807.03341v2 [stat.ML] 26 Jul 2018".1

What it does not establish: that anyone outside the author team has reviewed it. This course treats a preprint as not peer-reviewed unless you find and open a peer-reviewed version.

Versions are different documents. The status belongs to the version, not the title. The same title can exist in several forms:

Vendor report

What it establishes: what the developer chose to publish about its own system.

What it does not establish: independent measurement. Two things limit what you can check:

A vendor report can still describe its own checks. The same 2023 report describes a contamination check and how it chose which exam scores to report.8 Section 3 looks at that check. The point is who did the checking: the developer.

System card

What it establishes: the developer's own account of a model's risks and of the safety work it did. The system card appended to the same 2023 report says it highlights "safety challenges" and gives "a high-level overview of the safety processes" the developer adopted.8

What it does not establish: that those processes work, or that an outsider checked them. The same system card says its mitigations "are limited and remain brittle in some cases".8 A system card is evidence of what the developer says it tested and did. It is not an audit.

Quick reference

StatusTells youDoes not tell you
Peer-reviewed paperReviewers read it; the venue accepted itThat it is true, or that anyone re-ran it
PreprintThe authors claimed this, in this version, on this dateThat anyone outside the team reviewed it
Vendor reportWhat the developer chose to publishIndependent measurement; withheld details
System cardThe developer's account of risks and safety workThat the safeguards work; outside audit

This table is this course's summary of the sources above.

The misconception this corrects: "Peer review, or an arXiv number, means the claim is true." Peer review means some reviewers accepted it. An arXiv number means it was posted. Neither re-runs the experiment. You still check the evidence.

At Big Widget Company

The governance team keeps a source log. Each entry records the exact version opened (arXiv version number, journal issue or document date), its status from the table above, who wrote it, and what was not checked. The weld supplier's paper is logged as "vendor report, not peer-reviewed, not independently tested", even though it is formatted like a conference paper. The log entry names the person who will decide whether the paper is relied on.

3. Three ways a benchmark score misleads

A benchmark is a fixed test: a dataset and a scoring rule that a community agrees to use to compare models.9 A score on it can be high for reasons that have nothing to do with the real job. Check for three problems before you rely on one.

Leakage: the test was not really held out

Kapoor and Narayanan define data leakage as "a spurious relationship between the independent variables and the target variable that arises as an artifact of the data collection, sampling, or pre-processing strategy". Because that relationship will not exist in the real world, leakage "usually leads to inflated estimates of model performance."3

Their survey found 20 papers that documented errors in machine-learning-based science across 17 fields, together affecting 329 papers, and leakage appeared "in every single case".3 They call this a lower bound, not a systematic count.3 Their taxonomy has eight types under three headings.3

HeadingTypesWhat goes wrong
No clean split between training and test dataNo test set; pre-processing on both; feature selection on both; duplicatesThe model sees test information before it is tested
Illegitimate features(judged case by case)The model uses something it would not have in real use, such as a proxy for the answer
Test set not drawn from the population the claim is aboutTemporal leakage; non-independence; sampling biasThe test does not match the claim

Two examples from the paper: a hypertension model that used anti-hypertensive drug use as an input, and a histopathology study that put different observations of the same patients in both the training and test sets.3

Contamination: a kind of leakage where the model may have seen the test

A specific kind of leakage hits large language models. Sainz and colleagues write: "The worst kind of data contamination happens when a Large Language Model (LLM) is trained on the test split of a benchmark, and then evaluated in the same benchmark." They add that "the extent of the problem is unknown, as it is not straightforward to measure."7 Models are trained on huge crawls of the internet, which makes it very hard to know whether a given benchmark was in the training data.7

They distinguish three kinds of contamination: the annotation guidelines were seen; the raw text was seen before it was labelled; or the annotations (the answers) were seen.7 If the test split's answers were seen, "the experiment is completely invalidated". If only the training or development split was seen, comparisons with models trained on those splits survive, but claims of zero-shot or few-shot performance do not.7

The 2023 developer report from section 2 shows what a developer's contamination check looks like, and its limits:

So "we checked for contamination" is itself a claim. Ask how the check worked, what it could miss and who ran it. Keep three things apart: a risk of exposure (the test is public), a flag from a detector (which can be wrong), and confirmed exposure (which needs evidence that the item was in the training data).

Construct validity: the test does not measure the job

Inioluwa Deborah Raji and colleagues define construct validity, for machine learning, as "how well the benchmark dataset, and associated metrics of evaluation, represents a task".9 They argue that benchmarks such as ImageNet and GLUE are treated as measures of general ability ("visual understanding", "language understanding") that, being "inherently specific, finite and contextual", they cannot be.9 Their image: a children's book in which Grover visits a museum of "everything in the whole wide world", walks through a few arbitrary rooms, and opens the last door, "Everything Else", to find himself outside.9

Their example of the problem in marketing: in January 2021, a large technology company said that its model "surpassing human performance on SuperGLUE marks an important milestone toward general AI."9 A score on one fixed set of tasks became a claim about general intelligence.

Biderman and colleagues make the same point about language models: "It is almost never the case that one cares about the actual score of a language model on a particular benchmark." Benchmarks are proxies, and "the appropriate conclusions to draw about a model's behavior are informed by, not constituted by, these numerical scores."4

Lipton and Steinhardt give an example from medicine: a "dermatologist-level" skin-cancer classifier. Dermatologists meet a wide variety of cases and changing conditions; "the machine classifier, however, only achieves low error on i.i.d. test data" (data drawn the same way as its training data).1

A checklist before you rely on a score

ProblemAsk
LeakageWas every step (cleaning, filling gaps, choosing features) done on training data only? Are the same people, units or time periods in both sets? Does any input give away the answer?
ContaminationCould the model have seen these test items, or their answers, in training? How was that checked, what could the check miss, and is the result a risk, a flag or a confirmed exposure?
Construct validityWhat job do we care about? How much does this test look like it, in data, conditions and scoring?

This checklist is this course's synthesis of the sources above.

The misconception this corrects: "A higher benchmark score means the model is better at the real job." A higher score means a higher score on that test, under those conditions. It says something about the real job only if the test was held out, unseen and built to match the job.

At Big Widget Company

Back to the weld supplier. The quality team reads the method section.

The team asks the supplier for a test on welds the model has never seen, split by weld, using Big Widget's own camera images. Until that test is in, the claim stays "not checked", and the quality lead does not rely on it.

4. Reading as a governance habit

The authors of these sources suggest fixes for their own fields, and each gives you something to ask for.

The Foundation names three yeses: Get to Yes, Stay at Yes, Recover to Yes.10 They are a cycle, not a one-time approval. Its gates decide "go ahead · only this far · wait for a person · no".10 Here they are applied to reading research; that application is this course's, not a quotation. Reading is one check inside a yes, not the whole of it. A named person who owns the decision gives the answer; the reader's job is to say what the evidence does and does not show.

"Not checked" is an honest answer, not a failure. The Foundation's own line is that "I don't know yet" is always an answer, and it always pairs you with someone.11 Pair every "not checked" with the person who will check it next.

What you keep. Keep your reading notes (which version you opened, which checks you ran, what is not checked, who decided) in your own work record, under your workplace's controls. That is not the Foundation's receipt. A receipt records a gate's decision as metadata: it carries "hashes, decision, tier, policy hash, key id" and never carries "prompts, outputs, payloads, personal data".11 A receipt shows that a decision was made and recorded. It does not show the research was right.

Words to know: baseline, ablation, sample, metric, falsifier, peer review, preprint, version, vendor report, system card, leakage, contamination, construct validity, reproducible, replicable, not checked, not established, gate, receipt.

Further learning (optional, not citations)

Free public university course materials for learners who want more. Links only; nothing here is copied from them, and they support no claim in this module.

Sources

  1. s1 Zachary C. Lipton and Jacob Steinhardt (2018). Troubling Trends in Machine Learning Scholarship. Paper. Checked 2026-11-20 by source-checker agent (Claude Opus 5.5, round-4 re-check session separate from the lesson writer and item writer), 20 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  2. s9 Gábor Melis, Chris Dyer and Phil Blunsom (2017). On the State of the Art of Evaluation in Neural Language Models. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-3 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩
  3. s3 Sayash Kapoor and Arvind Narayanan (2022). Leakage and the Reproducibility Crisis in ML-based Science. Paper. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, session separate from the lesson writer and item writer), 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  4. s5 Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Baber Abbasi, Jessica Zosa Forde, Leo Gao, Jonathan Tow, et al. (30 authors; full roster in the source pack) (2024). Lessons from the Trenches on Reproducible Evaluation of Language Models. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  5. s7 Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché-Buc, Emily Fox and Hugo Larochelle (2021). Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩ ↩ ↩
  6. s2 Zachary C. Lipton and Jacob Steinhardt (2019). Troubling Trends in Machine Learning Scholarship. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩
  7. s6 Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle and Eneko Agirre (2023). NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩
  8. s8 OpenAI et al. (2023). GPT-4 Technical Report (including the appended GPT-4 System Card). Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  9. s4 Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton and Alex Hanna (2021). AI and the Everything in the Whole Wide World Benchmark. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩ ↩ ↩
  10. s10 AiGovOps Foundation (n.d.). AiGovOps Foundation Manifesto. Official doc. Checked 2026-11-20 by source-checker agent (Claude Opus 5.5, round-4 re-check session separate from the lesson writer and item writer), 20 Nov 2026; live page as served that day ("Fall 2026 Release", no date on the page). ↩ ↩
  11. s11 AiGovOps Foundation (n.d.). AiGovOps Foundation Practice (home page). Official doc. Checked 2026-11-20 by source-checker agent (Claude Opus 5.5, round-4 re-check session separate from the lesson writer and item writer), 20 Nov 2026; live page as served that day (footer "practice v1.0.0-rc.1", "Fall 2026 Release"). ↩ ↩

Next: the quiz