
Reading and critiquing primary research · Standard gear
You will be able to: test an AI paper's headline claim against its own evidence by naming the baseline, the sample, the metric and what the result would look like if the claim were false; say whether a source is a peer-reviewed paper, a preprint, a vendor report or a system card, and what each status does and does not establish; and spot leakage, test-set contamination and construct-validity problems in a benchmark result before you rely on it.
This module is as of November 2026. Reviewed and published, version 1.0.0. It is not a credential. Every source and its status is listed in sources.yml. Where we cite a preprint, we say so.
1. The headline is a claim, not a finding
A paper's abstract is the authors' summary of what they think they showed. It is written by the people with the most to gain from the result. Your job at this level is to read the claim, then find the evidence for it inside the paper, and check that the two match.
Two machine learning researchers, Zachary Lipton and Jacob Steinhardt, described in 2018 how this goes wrong. They wrote that "papers often offer speculation in the guise of explanations, which are then interpreted as authoritative due to the trappings of a scientific paper and the presumed expertise of the authors."1 They named four patterns they saw spreading in the field: failing to separate explanation from speculation; failing to identify where empirical gains came from; "mathiness", maths that "obfuscates or impresses rather than clarifies"; and misuse of language.1 Their paper is itself a position piece, written for a workshop debate (section 2 explains why that matters).1
Four questions for every headline claim
Before you repeat a result, ask four questions of the paper's own tables and methods.
- The baseline: better than what? "Our model is better" only means something against a fair comparison. Was the comparison model tuned as carefully as the new one?
- The sample: measured on what? Which data, how many examples, drawn from where, and how much like the setting the claim is about?
- The metric: scored how? Which number, computed how, out of how many, from how many runs, with what uncertainty?
- The falsifier: what would "false" look like? Before you look at the result, say what the table would show if the claim were wrong. If no possible table could show that, the claim is not being tested.
The fourth question is this course's working rule, not a quotation from the sources. It is the one that turns reading into checking.
Real case: a weak baseline
Gábor Melis, Chris Dyer and Phil Blunsom re-tested several recurrent neural network designs for language modelling. Earlier results, they note, had been "evaluated using differing code bases and limited computational resources", which are "uncontrolled sources of experimental variation". After "large-scale automatic black-box hyperparameter tuning" of each design, they concluded that "standard LSTM architectures, when properly regularised, outperform more recent models" on the language modelling benchmarks they used, including the Penn Treebank and Wikitext-2 corpora.2 We read only the paper's arXiv abstract page (version 2, 20 November 2017), which lists no journal or conference version.2 Lipton and Steinhardt's reference list, however, cites the paper as appearing at the International Conference on Learning Representations (ICLR) in 2018.1 We have not opened that version, so its text and its review record are not checked. The version we quote is the arXiv preprint.
Lipton and Steinhardt use this case in their critique: a series of published improvements, "originally attributed to complex innovations in network architectures, were actually due to better hyper-parameter tuning."1 Keep the finding inside its bounds. It is about the recurrent models, benchmarks and tuning budget that Melis and colleagues tested. It does not show that tuning explains every claimed gain, for every design or task. Lipton and Steinhardt's wider complaint: "Too frequently, authors propose many tweaks absent proper ablation studies, obscuring the source of empirical gains."1 An ablation removes one change at a time to see which one actually made the difference.
Real case: the claim that failed to reproduce
In 2022, Sayash Kapoor and Arvind Narayanan of Princeton re-examined published papers on predicting civil wars. The field believed complex machine learning models "vastly outperform" logistic regression, a decades-old statistical method.3 Run the four questions on what they found.
- Baseline. The claim was "complex model beats logistic regression". Once the errors were fixed, the complex models did not "perform substantively better than decades-old Logistic Regression models".3 In one paper, the gap in AUC (the score that paper reported) between the boosted model, AdaBoost, and logistic regression fell from 0.14 to 0.01. In another, none of the models beat a simple baseline that predicts the previous year's outcome.3
- Sample. The errors were in how the test data was built: data was filled in (imputed) using the training and test data together, or a filled-in dataset was reused, or the model was given proxies for the very outcome it was meant to predict.3 Section 3 names these problems.
- Metric. The headline numbers were real numbers. They measured something other than what the claim needed.
- Falsifier. If "complex models are better" were false, a correctly built test would show the complex model roughly level with logistic regression. That is what the corrected test showed.3 Be exact about what that shows. A near-zero gap means the claimed advantage was not established. It counts against any meaningful advantage only if the test is precise enough to rule one out. This module has not checked the uncertainty around the corrected figures.
These papers had been published in top political science journals. The authors' sharpest line: "none of these errors could have been caught by reading the papers".3 Reading the abstract was never going to be enough. Neither, in this case, was reading the whole paper; the errors were found in the code and data.
Real case: the same model, two ways of running the test
The metric question covers how a number was produced, not just what it is called. Stella Biderman and colleagues, who build and maintain a popular evaluation library, report that one open model scored 72.4% on a multiple-choice benchmark (ARC-Easy) under one evaluation setup and 26.5% under another.4 The model and the questions were the same. What changed was how the questions were presented and how answers were scored. In the "cloze" setup, each candidate answer is scored as a completion of the question. In the "MMLU-style" setup, the options are listed with letters and the model's answer letter is scored.4 So the gap is not a pure effect of wording. It shows that the result depends on how the test was run.
They also warn that "the precise string used for a given prompt, even down to whitespace usage, can significantly impact performance", and that the choice of prompt changes how models rank against each other.4 And they warn against single runs: a benchmark such as HumanEval has "only 164 examples", and small "improvements" on it "might be washed out simply via sampling again".4 Their paper is a preprint (version 3, May 2026).4
The misconception this corrects: "A result in a paper is a fact; the abstract says what the paper showed." A result is a measurement under particular conditions. The abstract is the authors' reading of it. Check the baseline, the sample, the metric and the falsifier yourself.
At Big Widget Company
Made-up example for practice. A supplier sends the quality team a paper. Its abstract says: "Our inspection model outperforms human inspectors at detecting weld defects." The results table shows the model at 96% accuracy on 500 photographs from the supplier's own plant, and "human baseline: 91%", with a footnote saying two inspectors each labelled 50 of the photographs.
- Baseline: two humans on 50 photos each, against the model on 500. Not the same test.
- Sample: one plant's photographs. Big Widget's welds come from different lines and cameras.
- Metric: accuracy. If 95 of every 100 welds are sound, a model that calls everything "sound" scores 95%. Ask for the defect catch rate and the false-alarm rate, each out of how many welds.
- Falsifier: first decide what "better" must mean. For example: a higher defect catch rate at the same false-alarm rate, by a margin the quality lead sets before the test. Then score the model and the inspectors on the same photos. Two different results are possible. If the whole uncertainty range for the gap sits below that margin, the test rules out a gain that large, at the uncertainty level the team chose for that test. That is strong evidence against the claim, not a guarantee. If the gap is small and its uncertainty still reaches the margin, the claim is not established, which is not the same as false. This table cannot show either result, because the model and the inspectors took different tests.
Be careful with a near tie. A small gap with wide uncertainty does not show the model is no better. It means this test did not resolve the claim. The reason is not always too few photos: photos that come in batches from the same line, or a comparison that is not paired, can also leave the range wide. Only a test whose range sits below a gain that matters counts as evidence against the claim.
The team's note: "Vendor claim. The table does not support 'outperforms human inspectors'. Not checked on our welds. Next check: model and inspectors on the same photos." The quality lead owns the next check and the decision. Until the check is run, the claim waits for that person; it is not used. The reader recommends; the owner decides.
2. What the label on a source does and does not tell you
Sources arrive with labels: a conference paper, an arXiv number, a technical report, a system card. Each label tells you something about how the document came to exist. None of them tells you the claim is true.
Peer-reviewed paper
What it establishes: reviewers chosen by a journal or conference read the paper and the venue accepted it.
What it does not establish: that the result is right, or that anyone re-ran it.
- Reviewers disagree. Pineau and colleagues, reporting on the NeurIPS reproducibility programme, describe an experiment the NeurIPS programme chairs ran in 2014: 10% of submissions were reviewed independently by two separate groups of reviewers. The groups disagreed on 25.9% of those papers overall. At the 22.5% acceptance rate they used, they disagreed on 57% of the list of accepted papers.5 So the 57% is about the accepted list, not about all papers. This module gives their percentages; the original 2014 write-up was not opened here.
- Reviewers are not expected to run the code. NeurIPS 2019's code policy "expects code only for accepted papers, and only by the camera-ready deadline", and the code "is not expected to be used during the review process to decide on the soundness of the work".5 That is a statement of what was expected, not of what reviewers did. In the reviewer survey, 5,298 answers said code was provided, 2,255 said the reviewer had looked at it, and 1,315 said it was useful in guiding the review.5 The survey counts looking at code. It gives no figure for re-running it.
- Reviewers reward some things over others. Lipton and Steinhardt: "The machine learning peer review process places a premium on technical novelty."1
- Peer-reviewed papers can still fail. The civil-war papers in section 1 were published in top journals.3
Review is not a rerun and not proof. It is one check, by a few people, under time pressure.
Preprint
What it establishes: the authors made this claim publicly, in this version, on this date. On arXiv, the version and date are stamped on the paper: Lipton and Steinhardt's reads "arXiv:1807.03341v2 [stat.ML] 26 Jul 2018".1
What it does not establish: that anyone outside the author team has reviewed it. This course treats a preprint as not peer-reviewed unless you find and open a peer-reviewed version.
Versions are different documents. The status belongs to the version, not the title. The same title can exist in several forms:
- Lipton and Steinhardt's 2018 preprint was written for the ICML Machine Learning Debates workshop.1 A piece with the same title appeared in ACM's Queue (volume 17, issue 1, February 2019).6 We opened only its publication record, which names the container Queue and gives the type "journal-article". The record says nothing about review, and we have not opened the text. Its review status and genre are not checked.6 Do not borrow one version's status for the other.
- Kapoor and Narayanan's paper was first posted to arXiv in July 2022 and later published in the journal Patterns (volume 4, issue 9, September 2023).3 The numbers in this module come from the arXiv version we opened (version 1). The journal version may differ.
- Oscar Sainz and colleagues' paper on contamination (section 3) is cited here from its arXiv preprint, version 1, dated 27 October 2023, which is the version we read. A version with the same title was published in Findings of the Association for Computational Linguistics: EMNLP 2023 (December 2023, pages 10776–10787).7 We opened only that publication's record, not its text, so the quotes here are from the preprint.
- Biderman and colleagues' evaluation paper is cited as version 3 of its arXiv posting, dated 31 May 2026.4 Cite the version you read.
- Melis and colleagues' paper (section 1) is cited from its arXiv abstract page, version 2.2 Lipton and Steinhardt list an ICLR 2018 version, which we have not opened.1
Vendor report
What it establishes: what the developer chose to publish about its own system.
What it does not establish: independent measurement. Two things limit what you can check:
- Withheld details. One developer's 2023 technical report on its large language model says it "contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar".8 That report alone does not give an outsider enough to reproduce the training. Narrower checks may still be possible. If outsiders can reach the same model version, they can re-run some tests of its answers. But Biderman and colleagues note that many APIs "do not expose output log-probabilities", so such tests must score generated answers, which adds variance.4 Each narrower retest needs its own inputs: the exact model version, the prompts, the settings and the test items. It supports a narrower claim than the report makes.
- Self-graded scores. Biderman and colleagues write that for some models never released in any form, the only evaluation numbers are the ones the developer chose to publish, and that the field "has, in effect, adopted a norm where model developers grade their own homework."4 They add: "Evaluations are not just scorecards: they are advertisements."4
A vendor report can still describe its own checks. The same 2023 report describes a contamination check and how it chose which exam scores to report.8 Section 3 looks at that check. The point is who did the checking: the developer.
System card
What it establishes: the developer's own account of a model's risks and of the safety work it did. The system card appended to the same 2023 report says it highlights "safety challenges" and gives "a high-level overview of the safety processes" the developer adopted.8
What it does not establish: that those processes work, or that an outsider checked them. The same system card says its mitigations "are limited and remain brittle in some cases".8 A system card is evidence of what the developer says it tested and did. It is not an audit.
Quick reference
| Status | Tells you | Does not tell you |
|---|---|---|
| Peer-reviewed paper | Reviewers read it; the venue accepted it | That it is true, or that anyone re-ran it |
| Preprint | The authors claimed this, in this version, on this date | That anyone outside the team reviewed it |
| Vendor report | What the developer chose to publish | Independent measurement; withheld details |
| System card | The developer's account of risks and safety work | That the safeguards work; outside audit |
This table is this course's summary of the sources above.
The misconception this corrects: "Peer review, or an arXiv number, means the claim is true." Peer review means some reviewers accepted it. An arXiv number means it was posted. Neither re-runs the experiment. You still check the evidence.
At Big Widget Company
The governance team keeps a source log. Each entry records the exact version opened (arXiv version number, journal issue or document date), its status from the table above, who wrote it, and what was not checked. The weld supplier's paper is logged as "vendor report, not peer-reviewed, not independently tested", even though it is formatted like a conference paper. The log entry names the person who will decide whether the paper is relied on.
3. Three ways a benchmark score misleads
A benchmark is a fixed test: a dataset and a scoring rule that a community agrees to use to compare models.9 A score on it can be high for reasons that have nothing to do with the real job. Check for three problems before you rely on one.
Leakage: the test was not really held out
Kapoor and Narayanan define data leakage as "a spurious relationship between the independent variables and the target variable that arises as an artifact of the data collection, sampling, or pre-processing strategy". Because that relationship will not exist in the real world, leakage "usually leads to inflated estimates of model performance."3
Their survey found 20 papers that documented errors in machine-learning-based science across 17 fields, together affecting 329 papers, and leakage appeared "in every single case".3 They call this a lower bound, not a systematic count.3 Their taxonomy has eight types under three headings.3
| Heading | Types | What goes wrong |
|---|---|---|
| No clean split between training and test data | No test set; pre-processing on both; feature selection on both; duplicates | The model sees test information before it is tested |
| Illegitimate features | (judged case by case) | The model uses something it would not have in real use, such as a proxy for the answer |
| Test set not drawn from the population the claim is about | Temporal leakage; non-independence; sampling bias | The test does not match the claim |
Two examples from the paper: a hypertension model that used anti-hypertensive drug use as an input, and a histopathology study that put different observations of the same patients in both the training and test sets.3
Contamination: a kind of leakage where the model may have seen the test
A specific kind of leakage hits large language models. Sainz and colleagues write: "The worst kind of data contamination happens when a Large Language Model (LLM) is trained on the test split of a benchmark, and then evaluated in the same benchmark." They add that "the extent of the problem is unknown, as it is not straightforward to measure."7 Models are trained on huge crawls of the internet, which makes it very hard to know whether a given benchmark was in the training data.7
They distinguish three kinds of contamination: the annotation guidelines were seen; the raw text was seen before it was labelled; or the annotations (the answers) were seen.7 If the test split's answers were seen, "the experiment is completely invalidated". If only the training or development split was seen, comparisons with models trained on those splits survive, but claims of zero-shot or few-shot performance do not.7
The 2023 developer report from section 2 shows what a developer's contamination check looks like, and its limits:
- It says "a minority of the problems in the exams were seen by the model during training". For each exam it ran a variant with the flagged questions removed and reported "the lower score of the two".8 So a reported exam score is whichever was lower, the full score or the filtered one. Neither is certified clean.
- It flagged questions by matching short substrings. The report says this "can result in false negatives ... as well as false positives".8 Some overlap may have been missed, and some flagged questions may never have been seen. The matching ignored answers, and in some cases the options, and the developer says it did not explicitly check its post-training (RLHF) data.8
- It found "portions of BIG-bench were inadvertently mixed into the training set" and excluded that benchmark from its results.8
- It mixed in training data from two maths benchmarks, MATH and GSM-8K, on purpose.8 Sainz and colleagues point out that the model's GSM-8K results therefore "cannot be taken as zero-shot results".7
So "we checked for contamination" is itself a claim. Ask how the check worked, what it could miss and who ran it. Keep three things apart: a risk of exposure (the test is public), a flag from a detector (which can be wrong), and confirmed exposure (which needs evidence that the item was in the training data).
Construct validity: the test does not measure the job
Inioluwa Deborah Raji and colleagues define construct validity, for machine learning, as "how well the benchmark dataset, and associated metrics of evaluation, represents a task".9 They argue that benchmarks such as ImageNet and GLUE are treated as measures of general ability ("visual understanding", "language understanding") that, being "inherently specific, finite and contextual", they cannot be.9 Their image: a children's book in which Grover visits a museum of "everything in the whole wide world", walks through a few arbitrary rooms, and opens the last door, "Everything Else", to find himself outside.9
Their example of the problem in marketing: in January 2021, a large technology company said that its model "surpassing human performance on SuperGLUE marks an important milestone toward general AI."9 A score on one fixed set of tasks became a claim about general intelligence.
Biderman and colleagues make the same point about language models: "It is almost never the case that one cares about the actual score of a language model on a particular benchmark." Benchmarks are proxies, and "the appropriate conclusions to draw about a model's behavior are informed by, not constituted by, these numerical scores."4
Lipton and Steinhardt give an example from medicine: a "dermatologist-level" skin-cancer classifier. Dermatologists meet a wide variety of cases and changing conditions; "the machine classifier, however, only achieves low error on i.i.d. test data" (data drawn the same way as its training data).1
A checklist before you rely on a score
| Problem | Ask |
|---|---|
| Leakage | Was every step (cleaning, filling gaps, choosing features) done on training data only? Are the same people, units or time periods in both sets? Does any input give away the answer? |
| Contamination | Could the model have seen these test items, or their answers, in training? How was that checked, what could the check miss, and is the result a risk, a flag or a confirmed exposure? |
| Construct validity | What job do we care about? How much does this test look like it, in data, conditions and scoring? |
This checklist is this course's synthesis of the sources above.
The misconception this corrects: "A higher benchmark score means the model is better at the real job." A higher score means a higher score on that test, under those conditions. It says something about the real job only if the test was held out, unseen and built to match the job.
At Big Widget Company
Back to the weld supplier. The quality team reads the method section.
- Leakage: photographs were taken in bursts of five per weld, and the split was by photograph, not by weld. The same weld is in both training and test sets: non-independence.
- Contamination: the model was pre-trained on a public image collection. The paper does not say whether that collection includes the public weld-defect dataset the supplier also reports a score on. That is a risk, not a finding. Status: not checked. The quality lead asks the supplier how it checked.
- Construct validity: the test photos are well lit and centred. Big Widget's line cameras are not. The job is "catch defects on our line", and the test does not look like it.
The team asks the supplier for a test on welds the model has never seen, split by weld, using Big Widget's own camera images. Until that test is in, the claim stays "not checked", and the quality lead does not rely on it.
4. Reading as a governance habit
The authors of these sources suggest fixes for their own fields, and each gives you something to ask for.
- Code, prompts and outputs. Biderman and colleagues recommend releasing "the exact evaluation code used" and reporting prompts within the publication. They note that reporting full prompts is "necessary but not sufficient" for reproducibility.4
- Uncertainty. They recommend reporting variance over several runs or statistical significance "instead of reporting isolated numbers".4
- Leakage arguments. Kapoor and Narayanan propose model info sheets, which require researchers "to provide precise arguments to justify that models used towards making scientific claims do not suffer from leakage".3
- Reproduction is not proof either. Pineau and colleagues note that having the original code "is not a guarantee that this code is correct", and that after NeurIPS 2019's programme they did "not have concluding evidence" that it improved the quality of papers.5 They distinguish reproducible (same data, same tools), replicable (different data), robust (same data, different analysis) and generalisable (different data and different analysis).5 Ask which one a claim has actually met.
- Ask the reviewer's question. Lipton and Steinhardt suggest reviewers ask: "Might I have accepted this paper if the authors had done a worse job?"1
The Foundation names three yeses: Get to Yes, Stay at Yes, Recover to Yes.10 They are a cycle, not a one-time approval. Its gates decide "go ahead · only this far · wait for a person · no".10 Here they are applied to reading research; that application is this course's, not a quotation. Reading is one check inside a yes, not the whole of it. A named person who owns the decision gives the answer; the reader's job is to say what the evidence does and does not show.
- Get to Yes: before a research result shapes a decision, log its status and exact version, run the four questions and the three-problem checklist, and write "not checked" against anything you could not confirm. Name the person who decides whether the result is relied on, and what it is relied on for ("only this far" is a real answer: relied on for a pilot, not for a purchase). A result with an important gap waits for that person; it does not go into the decision by default.
- Stay at Yes: re-check when a new version, an erratum or a failed reproduction appears. Versions change: Biderman and colleagues' paper reached version 3 in 2026.4
- Recover to Yes: if a decision rested on a result that later fails, stop relying on it, say so, name who owns the fix, and revisit the decision. A stated plan to re-check is not yet a recovery; you are back at yes when the new check has actually been run and recorded.
"Not checked" is an honest answer, not a failure. The Foundation's own line is that "I don't know yet" is always an answer, and it always pairs you with someone.11 Pair every "not checked" with the person who will check it next.
What you keep. Keep your reading notes (which version you opened, which checks you ran, what is not checked, who decided) in your own work record, under your workplace's controls. That is not the Foundation's receipt. A receipt records a gate's decision as metadata: it carries "hashes, decision, tier, policy hash, key id" and never carries "prompts, outputs, payloads, personal data".11 A receipt shows that a decision was made and recorded. It does not show the research was right.
Words to know: baseline, ablation, sample, metric, falsifier, peer review, preprint, version, vendor report, system card, leakage, contamination, construct validity, reproducible, replicable, not checked, not established, gate, receipt.
Further learning (optional, not citations)
Free public university course materials for learners who want more. Links only; nothing here is copied from them, and they support no claim in this module.
- Stanford University, CS336: Language Modeling from Scratch (Spring 2026): <https://cs336.stanford.edu/>. Current lecture schedule with scaling and evaluation materials. Full exercises may need paid GPU compute.
- University of Washington, CSE 599: Generative Models (Autumn 2020): <https://courses.cs.washington.edu/courses/cse599i/20au/>. Historical background; a research reading list with lecture notes.
- University of Michigan, EECS 598: Unsupervised Feature Learning: <https://web.eecs.umich.edu/~honglak/teaching/eecs598/>. Historical background; a public paper-reading schedule, useful practice for the four questions.