Level 500: Governance, safety and research · Module 500.1

Two colleagues compare two printouts while a third takes notes.
What was the baseline?

Reading and critiquing primary research · Plain gear

You will be able to: check whether a research paper's big claim is backed by its own evidence; tell a reviewed paper from a draft, a company report and a safety card, and say what each one can and cannot prove; and spot three ways a test score can fool you.

This module is as of November 2026. Reviewed and published, version 1.0.0. It is not a credential.

1. The headline is a claim, not a fact

Think of a film trailer. It shows the best bits. It is made by the people who want you to buy a ticket. A research paper's summary, called the abstract, is a bit like that. The authors wrote it. They chose what to put in it. To know what they really showed, you have to look inside.

Two researchers, Zachary Lipton and Jacob Steinhardt, warned about this in 2018. They said papers often dress up guesses as explanations. Readers then believe the guesses, because they appear in a scientific paper written by experts.1

Four questions to ask

  1. Better than what? A claim that something is "better" needs a fair comparison. Was the old way given the same care as the new one?
  2. Tested on what? Which examples? How many? Are they like the real situation?
  3. Scored how? What number was used? Out of how many? Was the test run once, or many times?
  4. What would "wrong" look like? Before you look at the results, picture what they would show if the claim were false. If no result could ever prove it wrong, it is not really being tested.

The fourth question is this course's own rule. It is the one that turns reading into checking.

Real case: the unfair race

Imagine a race where one runner trains for months and the other does not train at all. The trained runner wins. Does that prove their new shoes are better? No.

Something like this happened in AI research. In 2017, three researchers re-tested several newer designs for language models against an older design called an LSTM. They said the earlier tests had used different code and limited computer power. This time, every design was tuned in the same careful, automatic way. The older design, set up properly, beat the newer ones on the tests they used.2 So some earlier "improvements" had come from better tuning, not from the new designs.1

That is true for those models and those tests. It does not mean tuning explains every claimed gain. (We read only the summary page of their paper, posted on a site called arXiv. That page lists no other version.2 But Lipton and Steinhardt's list of sources names a version at a conference, ICLR, in 2018.1 We have not opened that version, so whether it was reviewed is not checked. What we quote here comes from the arXiv page.)

Real case: predicting wars

Some researchers had published papers saying clever computer models could predict civil wars much better than a simple, old method. In 2022, two researchers at Princeton, Sayash Kapoor and Arvind Narayanan, checked those papers again.3

They found mistakes in how the tests were set up. In some, information from the test had leaked into the training. In one, the model was given clues that were close to the answer itself.3 When the mistakes were fixed, the clever models did not do much better than the simple, old method.3

These papers had appeared in top journals. And the researchers said none of the mistakes could have been caught just by reading the papers.3 They had to dig into the data and code.

Real case: same test, run two ways

A team that builds a popular tool for testing AI language models tried something simple. They gave one model the same multiple-choice questions, run in two different ways. In one, each possible answer was scored directly. In the other, the choices were listed as A, B, C and D, and only the letter the model picked was scored. With the first way, it got about 72 out of every 100 right. With the second, it got about 26.4 Same model. Same questions. Only the way the test was run changed. They warn that even tiny changes, down to spaces in the question, can change the score.4

The mistake to avoid: "If it is in a paper, it is a fact. The summary tells me what they found." A result is what happened in one test, set up one way. Check the four questions yourself.

At Big Widget Company

Made-up example. A supplier says its camera system "beats human inspectors at finding bad welds". The quality team looks inside the report. The system was tested on 500 photos from the supplier's own factory. The "human" score came from two people who each looked at only 50 photos.

Watch out for a near tie. If a fair test comes out almost even, that does not prove the claim wrong. It means the test has not shown the system is better. To count against the claim, the test has to be sure enough to rule out a gain that matters. Even then it is strong evidence, not a promise. A close test that is not that sure gives an honest answer: "we don't know yet".

The team writes: "Supplier claim. Not checked on our welds. Next check: system and inspectors on the same photos." The quality lead owns that next check. Until it is done, the claim waits. It is not used in any decision.

2. What the label on a source can and cannot prove

Sources come with labels. Each label tells you how the document was made. None of them tells you the claim is true.

A reviewed paper

Other experts read it before a journal or conference accepted it. That is useful. But it is not proof.

A draft (preprint)

A preprint is a paper the authors posted online themselves, often on a site called arXiv. It shows what the authors claimed, and when. In this course we treat it as not reviewed, unless you find a reviewed version.

Papers also come in versions, like drafts of an essay. Lipton and Steinhardt's 2018 draft was written for a workshop debate.1 A piece with the same title came out in ACM's Queue in 2019.6 We have only seen its listing, not the piece itself. So we do not know if it was reviewed: not checked. They are different documents. Always note which version you read.

A company report

A company writes about its own AI. That tells you what the company chose to share. One AI company's 2023 report on its own model said it left out details such as the model's size and how it was trained.7 So that report alone does not give outsiders enough to rebuild the model the same way. People outside may still be able to test some of its answers, if they can use the same version of the model.4 A test like that checks those answers. It does not check how the model was built. One team of researchers put the bigger problem simply: AI makers often "grade their own homework".4

A safety card (system card)

A system card is the company's own account of the risks it found and the safety steps it took. The same company's system card says its safety measures "are limited and remain brittle in some cases".7 A system card tells you what the company says it did. It is not an outside check.

The mistake to avoid: "It was reviewed, or it has an arXiv number, so it must be true." Reviewed means some experts accepted it. An arXiv number means it was posted. Neither one re-ran the test.

At Big Widget Company

The team keeps a list of every source they rely on. For each one they write: what kind it is, which version they read, who wrote it, and what they could not check. The supplier's weld report looks like a science paper. It goes on the list as "company report, not reviewed, not tested by anyone else".

3. Three ways a test score can fool you

A benchmark is a fixed test that lots of researchers use to compare AI models.8 A high score can come from the wrong reasons. Look for three problems.

1. Leakage: the test was not really kept apart

Imagine you want to know if a student can name birds from photos. You give them a photo of a robin to practise on. Then, in the test, you show a second photo of the same robin, taken a second later. The student gets it right. Did they learn robins, or just remember that one bird? You cannot tell.

In AI, this is called leakage. Information from the test gets into the training, or the model is given a clue that would not exist in real life. Kapoor and Narayanan found leakage in every one of the reviews of problem papers they collected, from 17 different fields of science.3 One example: a model meant to predict high blood pressure was given, as a clue, whether the patient took blood pressure pills.3

2. Contamination: the model may have already seen the test

Contamination is one kind of leakage. Big AI language models learn from huge amounts of text taken from the internet. Test questions are often online too. So the model may have read the test, and even the answers, before it is tested.9 How big the problem is "is unknown", researchers say, because it is hard to measure.9

A test being online is a risk, not proof that the model read it. Keep those apart.

The company's 2023 report describes how it checked its own exams. It used a simple matching tool to find questions the model might have seen. It ran each exam again without those questions and reported whichever score was lower.7 It also found that parts of one test set had slipped into its training data by mistake, so it left that test out.7 But the company did the checking itself. It says its matching tool can miss questions the model did see, and can flag questions it did not see.7 So even the lower score is not proven clean.

3. The test does not match the job

A driving test in an empty car park does not show you can drive in city traffic. Researchers call this a construct validity problem: the test does not measure the real thing you care about.8

Inioluwa Deborah Raji and colleagues compare some famous AI tests to a children's story. In it, Grover visits a museum that claims to hold "everything in the whole wide world". It has a few odd rooms. When he opens the last door, marked "Everything Else", he is outside.8 No fixed test can hold everything. In 2021, a big tech company said its model beating humans on one language test was "an important milestone toward general AI".8 That is a huge claim from one test.

The mistake to avoid: "A higher test score means the AI is better at the real job." A higher score means it did better on that test. It tells you about the real job only if the test was kept apart from the model and looks like the real job.

At Big Widget Company

The team looks again at the weld report.

The team asks the supplier to test the system on our own camera photos of welds it has never seen. Until then, the claim stays "not checked".

4. Make it a habit

The Foundation has three yeses: Get to Yes, Stay at Yes, Recover to Yes.10 They are a cycle, not a one-time okay. At a Foundation gate, the gate decides, and there are four answers: "go ahead · only this far · wait for a person · no".10 You, the reader, do not decide. Your job is to say what the evidence shows, and what it does not.

Even checking the original computer code is not a guarantee. Researchers who ran a big checking programme point out that the code can have mistakes too.5

The Foundation says "I don't know yet" is always an answer, and it always pairs you with someone.11 So when you write "not checked", also write who will check it next.

What you keep. Keep your notes (what you read, which version, what you could not check, who decided) in your own work record, where your workplace controls who can see them. The Foundation's receipt is different. It is the record of a gate's decision, and it never holds "prompts, outputs, payloads, personal data".11 So what you typed, what the AI wrote, and anyone's private details stay out of it.

Words to know: abstract, claim, fair comparison, reviewed paper, preprint, version, company report, system card, benchmark, leakage, contamination, construct validity, not checked, gate, receipt.

Sources

  1. s1 Zachary C. Lipton and Jacob Steinhardt (2018). Troubling Trends in Machine Learning Scholarship. Paper. Checked 2026-11-20 by source-checker agent (Claude Opus 5.5, round-4 re-check session separate from the lesson writer and item writer), 20 Nov 2026. ↩ ↩ ↩ ↩
  2. s9 Gábor Melis, Chris Dyer and Phil Blunsom (2017). On the State of the Art of Evaluation in Neural Language Models. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-3 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩
  3. s3 Sayash Kapoor and Arvind Narayanan (2022). Leakage and the Reproducibility Crisis in ML-based Science. Paper. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, session separate from the lesson writer and item writer), 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩
  4. s5 Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Baber Abbasi, Jessica Zosa Forde, Leo Gao, Jonathan Tow, et al. (30 authors; full roster in the source pack) (2024). Lessons from the Trenches on Reproducible Evaluation of Language Models. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩ ↩ ↩
  5. s7 Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché-Buc, Emily Fox and Hugo Larochelle (2021). Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩ ↩
  6. s2 Zachary C. Lipton and Jacob Steinhardt (2019). Troubling Trends in Machine Learning Scholarship. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩
  7. s8 OpenAI et al. (2023). GPT-4 Technical Report (including the appended GPT-4 System Card). Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩ ↩ ↩
  8. s4 Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton and Alex Hanna (2021). AI and the Everything in the Whole Wide World Benchmark. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩ ↩ ↩
  9. s6 Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle and Eneko Agirre (2023). NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, round-2 re-check session separate from the lesson writer and item writer), 19 Nov 2026. ↩ ↩
  10. s10 AiGovOps Foundation (n.d.). AiGovOps Foundation Manifesto. Official doc. Checked 2026-11-20 by source-checker agent (Claude Opus 5.5, round-4 re-check session separate from the lesson writer and item writer), 20 Nov 2026; live page as served that day ("Fall 2026 Release", no date on the page). ↩ ↩
  11. s11 AiGovOps Foundation (n.d.). AiGovOps Foundation Practice (home page). Official doc. Checked 2026-11-20 by source-checker agent (Claude Opus 5.5, round-4 re-check session separate from the lesson writer and item writer), 20 Nov 2026; live page as served that day (footer "practice v1.0.0-rc.1", "Fall 2026 Release"). ↩ ↩

Next: the quiz