Level 500: Governance, safety and research · Module 500.1

Try each exercise first, then open its model answer.

Reading and critiquing primary research · Workbook

Three exercises, one for each thing this module teaches. The research cases come from the lesson. The Big Widget Company material is made up, and so are its people, products and numbers. Use only what the lesson taught. You do not need to look anything up.

Exercise 1 (500.1-a)

Made-up example. A vendor sends Big Widget's planning team a paper about its demand-forecasting model. The abstract says:

"Our model cuts forecasting error in half compared with existing methods, and will improve planning for any manufacturer."

The results table shows:

MethodAverage errorTest data
Our model4.1%12 months of orders from one toy maker
Last year's sales, copied forward8.3%Same 12 months

A footnote says the model's settings were tuned many times, and the comparison method was not tuned at all. Each number comes from a single run.

  1. Write down the four questions from the lesson (baseline, sample, metric, falsifier) and answer each one for this paper.
  2. Name the part of the abstract that the table does not support, and say why.
  3. Write a one-line note for Big Widget's source log, and say who should own the follow-up.
Model answer

A good answer works through all four questions and ties each one to the table:

  • Baseline (better than what?): the comparison is copying last year's sales forward, a very simple method that was given no tuning, while the new model was tuned many times. That is not a fair race. The lesson's real case shows why it matters: when Melis and colleagues tuned every recurrent design in the same careful way, the older LSTM, properly set up, beat newer designs on the benchmarks they used. That finding is about those models and tests; it does not settle this vendor's claim either way.
  • Sample (tested on what?): twelve months of orders from one toy maker. Big Widget makes complex, car-like widgets, not toys. One company and one year say little about "any manufacturer".
  • Metric (scored how?): average error from a single run, with no spread or uncertainty given. The lesson's sources warn that one run can mislead and recommend reporting variance over several runs.
  • Falsifier (what would "wrong" look like?): first fix what "cuts error in half" must mean, and against which fairly tuned methods. The comparison method's error is uncertain too, since it comes from the same sample, so test the gap, not one number against the other. Score both methods on the same data over several runs, and work out one number for each run: the model's error minus half the comparison method's error. A result counts against the claim if the whole uncertainty range for that difference sits above zero, at the uncertainty level chosen for that test. That is strong evidence, not a guarantee. If the range includes zero, the test did not resolve the claim; it did not show it false. This table cannot show either result.

The unsupported part is "will improve planning for any manufacturer". The table covers one toy maker over one year. (A strong answer also notes that "existing methods" is broader than the one untuned method in the table.)

A good log note: "Vendor claim. Baseline untuned; one company, one year, one run. Does not support 'any manufacturer'. Not checked on our orders." It names one person, such as the planning lead, as the owner of the follow-up.

What to look for:

  • The answer treats the abstract as the vendor's claim, not as a finding.
  • It names a falsifier as a result that could actually appear, not just "the model could be wrong".
  • It writes "not checked" rather than guessing how the model would do on Big Widget's data.

Exercise 2 (500.1-b)

Big Widget's governance team is building its source log. Classify each source below as a peer-reviewed paper, a preprint, a vendor report or a system card. For each one, write one sentence on what its status establishes and one sentence on what it does not.

  1. Kapoor and Narayanan's paper on leakage, in the arXiv version 1 the lesson used (July 2022).
  2. The later version of the same paper in the journal Patterns (volume 4, issue 9, September 2023). For this exercise, take as given that it went through the journal's peer review; the lesson did not open this version.
  3. The 2023 technical report one developer published on its own large language model (the report the lesson quotes).
  4. The system card appended to that report.

Then answer: a colleague says "the arXiv version and the journal version are the same paper, so I'll quote the arXiv numbers as peer-reviewed". What do you say?

Model answer
SourceStatusEstablishesDoes not establish
1. arXiv v1, 2022PreprintThe authors made these claims, in this version, on this dateThat anyone outside the team reviewed it
2. Patterns, 2023Peer-reviewed paperReviewers read it and the journal accepted itThat the result is right, or that anyone re-ran it
3. Developer's 2023 technical reportVendor reportWhat the developer chose to publish about its own modelIndependent measurement. It withholds details such as model size, training compute and dataset construction, so the report alone does not give an outsider enough to reproduce the training; narrower retests of the model's answers may be possible on the same model version
4. Its system cardSystem cardThe developer's own account of risks and the safety work it didThat the safeguards work, or that an outsider audited them; the card itself calls its mitigations "limited" and "brittle in some cases"

Reply to the colleague: they are different documents. The lesson took its numbers from the arXiv version 1, and the journal version was not opened, so it may differ. Quote the version you read and give it that version's status. If you want to cite the peer-reviewed version, open it and check the numbers there. Even then, peer review is not proof. In the 2014 NeurIPS experiment that Pineau and colleagues report, two independent review groups disagreed on 25.9% of the papers they both reviewed, and on 57% of the list of accepted papers at a 22.5% acceptance rate. And reviewers were not expected to use code to decide whether the work was sound.

What to look for:

  • The answer does not treat an arXiv number as a sign of review, or peer review as a sign of truth.
  • It does not move a status from one version to another.
  • It treats the vendor report and the system card as the developer's own account, careful or not.

Exercise 3 (500.1-c)

Made-up example. A supplier offers Big Widget a language model to answer questions from its repair technicians. The supplier's report says:

"Our model scores 92% on RepairQA, a public benchmark of multiple-choice questions about car-engine repair. The model was pre-trained on a large crawl of the public web. RepairQA's test questions and answers have been on a public website since 2022. We fine-tuned the model on RepairQA's training split. To pick the best version of the model, we tried 40 settings and kept the one that scored highest on the RepairQA test split."

For each of the three problems (leakage, contamination, construct validity):

  1. Say whether the description gives you reason to worry, and point to the sentence that does.
  2. Say what you would ask the supplier for before relying on the score.
Model answer
  • Leakage: yes. Choosing among 40 settings by their score on the test split means test information shaped the model before the final test. The test was not really held out, and leakage usually inflates measured performance. Ask for a score on a test set that was never used to choose settings, or for the setting to be chosen on separate validation data.
  • Contamination: a risk, not checked. The model was pre-trained on a web crawl, and the test questions and answers have been public since 2022. So the model may have seen the test answers. That is a risk, not a finding: a page being public does not show it was in the training data. Researchers say the extent of the problem "is unknown" because it is hard to measure, and if the test answers were seen, the experiment is invalidated. Ask how contamination was checked, what that check could miss, and whether any result is a risk, a detector's flag or confirmed exposure. A developer's check is itself a claim: the developer report in the lesson says its own matching method can miss questions the model saw and flag questions it did not.
  • Construct validity: yes. The job is answering Big Widget technicians' questions about Big Widget's widgets, in their own words. The test is multiple-choice questions about car engines. A score on a fixed, finite test does not show the model can do a different, open-ended job. Ask for a test on real, unseen questions from Big Widget technicians, scored by people who know the repairs.

Fine-tuning on RepairQA's training split is not a problem on its own. Comparisons with other models trained on the same split can still stand, but it would rule out any claim that the model scores this well without such training (a "zero-shot" claim).

What to look for:

  • All three problems are named and each is tied to a sentence in the description.
  • The answer does not say a higher score means the model is better at the real job.
  • It asks for evidence rather than guessing the true score, and marks contamination as a risk, "not checked", rather than as a confirmed failure.

Next module: 500.2 What research can and cannot yet say about how models work