Level 500: Governance, safety and research · Module 500.2

Try each exercise first, then open its model answer.

What research can and cannot yet say about how models work · Workbook

Four exercises, one for each thing this module teaches. Each one uses the three boxes from the lesson: supported, not established and unresolved. Use only what the lesson taught. You do not need to look anything up, train a model or run a benchmark.

The setting is Big Widget Company, a made-up company. The vendor, the people and the claims are made up too. The charts and the sketch are the course-made examples from the lesson. The studies are the real ones the lesson cites.

Exercise 1 (500.2-a)

Big Widget's AI review board has two summaries in front of it. In both, loss measures how badly a language model predicts the next token (a small piece of text, often a word or part of a word) in text it has not seen.

A vendor writes: "Scaling laws prove our next model, ten times bigger, will handle warranty claims for custom parts."

Write a short note to the board (four to six sentences) that:

  1. Says in one sentence how the two summaries differ.
  2. States one prediction about loss that the summaries support, with its conditions attached.
  3. Names the vendor's claim as an inference about a specific ability, and says which box it goes in and why.
  4. Says what evidence the board should ask for instead.
Model answer

For example:

"The two summaries disagree on how to spend extra computing power: Summary A says grow the model much faster than the data, Summary B says grow them at about the same rate. They support a prediction about loss, with conditions: within the range tested, and when training text and compute are not the limit, a model twice as big should reach a loss about 5% lower. At a fixed budget, Summary B shows that bigger is not automatically better; a smaller model trained on more text reached lower loss. The vendor's claim is about a specific ability, handling custom-part claims, and no loss fit names a task, so it is not established. Kaplan et al. say themselves that it still had to be checked whether lower loss means better performance on real tasks, and the 70B model's better test results were measured after training, not forecast by the fit. We should ask for the new model's measured results on a course-made set of custom-part claims, with the uncertainty and the 'out of how many?'"

What to look for:

  • A loss prediction, with its conditions. The note ties the supported prediction to loss, to the tested range, and to data and compute not being the limit. A note that says "make it bigger and the loss falls", with no conditions, or "bigger is better at everything", has kept the misconception.
  • Loss is not a test score. A 5% lower loss is not 5% more right answers on any task.
  • The ability claim is not established. The note does not call the vendor's claim false. It says this evidence does not reach it.
  • The disagreement is read correctly. Two fits giving different advice shows that a scaling law is a fit to the runs a team trained, under one setup. It does not show that either team was wrong or that scaling laws are useless.
  • A request for measured evidence, with its uncertainty.

Exercise 2 (500.2-b)

The vendor sends a chart with the caption: "Arithmetic ability emerges at our largest size." Big Widget asks for the same models scored a second way. Here are both, using the lesson's course-made numbers for five models, A (smallest) to E (largest). These numbers are invented for the course; they are not any paper's data. They are the average scores worked out from the lesson's pretend rules: each digit is right with a chance that rises steadily with size, and digits are right or wrong separately.

ModelChart 1: all or nothing (share of answers with all 5 digits right)Chart 2: partial credit (digits right out of 5)
A0.2%1.5
B1.8%2.25
C7.8%3.0
D23.7%3.75
E59.0%4.5

Both charts come from the same made-up models and the same rules; only the marking differs. Model A took a 20-question test. Treat each question as separate: getting one right or wrong does not change the chances on the next.

Write a critique of four to six sentences that:

  1. Describes what each chart shows, and names the metric sensitivity: what changes between the charts and what does not.
  2. Explains, in your own words or with a spelling-test comparison, why all-or-nothing scoring can make a jump.
  3. Says what model A's score on its 20-question test does and does not tell you.
  4. Ends with what stays unresolved, rather than calling the ability "real" or "fake".
Model answer

For example:

"Chart 1 sits near zero for A and B, then climbs fast to 59% at E. Chart 2 rises by the same step, 0.75 of a digit, at every size. Both charts describe the same models; only the scoring changed, so the jump in Chart 1 is sensitive to the metric. All-or-nothing scoring works like a spelling test where one wrong letter costs the whole word: a model can get more digits right each step, but its score stays near zero until it gets whole answers right, then climbs fast. Model A gets a whole answer right only about 2 times in 1,000, so on 20 separate questions it will most likely score zero, about 95 times in 100 by the lesson's arithmetic; a zero on a short test is not a true zero rate. Still, a steady partial-credit chart does not by itself prove the scoring was the only cause of the jump. Whether some skills really change in kind as models grow is unresolved by this comparison, and the researchers who ran this kind of test say their work does not claim models cannot have emergent abilities."

What to look for:

  • Same models, different scoring. The critique says only the marking changed between the charts. (In the real study, the researchers scored the same actual answers both ways.)
  • Why the jump happens. Steady per-digit gains multiply when every digit must be right.
  • Resolution. A zero observed on a small test is not a true zero rate. The 95-in-100 figure assumes the questions are separate. (Other good questions: how many tasks and metrics were looked at before this chart was chosen.)
  • A bounded ending. A critique that ends "so the ability is fake" or "so the ability is real" has forced a binary the evidence does not settle. A good one ends with what is unresolved.

Exercise 3 (500.2-c)

The vendor's tool writes step-by-step reasoning before each warranty decision. The vendor says: "Every decision comes with its reasoning, so you can always see why it decided."

Here is a summary of the experiment from the lesson. Researchers tested two models from 2023. They added a nudge to questions, such as "I think the answer is A", or made every example answer option A. The nudge should not have mattered. It often changed the models' answers, sometimes to a wrong one. The researchers reviewed 426 explanations that went along with a nudged answer; only 1 of the 426 mentioned the nudge. In one example, a model reasoned its way from "plausible" to "implausible" about a soccer phrase and never mentioned the nudge.

Write three to five sentences that:

  1. State the conclusion this experiment supports.
  2. State one limit of the experiment, preferably one the authors state themselves.
  3. Say which box the vendor's claim goes in, and one practical step Big Widget could take.
Model answer

For example:

"The experiment supports this: a model's written reasoning can leave out the thing that changed its answer and still read as a sensible argument. That is a finding about cause, not about intent. One limit, which the authors state, is that their test can catch reasoning that leaves out a cause but cannot prove that reasoning is faithful; passing is necessary, not sufficient. It also only tested small changes to the input, on two models from 2023. So the vendor's claim is not established: seeing the reasoning does not mean seeing the real reason. We could test the tool with planted cues on course-made claims, re-run that test on each new version, and keep a person deciding the claims that matter."

What to look for:

  • The supported conclusion. Reasoning can omit a factor that changed the answer while still reading well. A reply that says "the written reasoning is the real reason" has kept the misconception.
  • No intent. A reply that says the model "lied" or "meant to mislead" claims more than the test shows.
  • A real limit. Necessary but not sufficient; only small changes to the input; only two 2023 models. A reply that says the study proves all reasoning is always unfaithful has overstated it.
  • No safe shortcut. A reply that says "just check the reasoning for mistakes" misses that, in the authors' hand-annotated sample of 104 unfaithful explanations, 15% had no obvious errors.

Exercise 4 (500.2-d)

The lesson's course-made sketch shows a toy network ending in a ReLU that must store five features of different importance in only two dimensions. Each stored feature is an arrow (a direction), not a neuron of its own.

The sketch assumes each feature switches on by chance, separately from the others.

Part A. In two or three sentences, use the sketch to explain why the network can live with the interference in the sparse case but not in the dense case.

Part B. Sort each statement into supported, not established or unresolved, using only this study. Give a one-line reason for each.

  1. In these toy networks, superposition happens, and whether it happens depends on how sparse and how important the features are.
  2. Superposition explains why the vendor's tool turned down warranty claim 4471.
  3. The geometry and learning behaviour seen in the toy networks also appear in large, real networks.
  4. The toy study showed that, in large language models, each neuron stands for one clean idea.
Model answer

Part A, for example: "When each feature is rarely on, and switches on separately from the others, two of them are rarely on at the same time, so most of the time a feature's neighbours are off and there is little to leak. The paper reports that the network tolerates the interference that remains and filters it with its ReLU. When features are often on, neighbours are often on together, so the leaks would add up; the network keeps the two most important at right angles and drops the rest."

Why rarely-on features rarely clash, and why the leaks would add up in the dense case, is the course's reasoning, not a quote from the paper. What the paper reports is the outcome in each case. The reasoning needs the "separately" part: two features can each be rare and still nearly always switch on together, and then the leak is there every time. A reply that says rarity alone keeps features apart, that the leaks cancel out to zero, or that each feature gets its own neuron, has misread the sketch.

Part B:

  1. Supported. This is what the toy experiments directly show; the authors call it a direct demonstration that superposition occurs given a relatively natural setup.
  2. Not established. The study used only small networks trained on made-up data. It never looked inside a deployed system or traced any of its decisions.
  3. Unresolved. The authors say it is "very unclear" what generalises to real networks, and call the geometry and learning dynamics "much more uncertain" than superposition itself.
  4. Not established. The toy study did not examine large language models. Separately, the authors observe that in large language models it seems rare for neurons to correspond to clean features. That is their observation, not a result of the toy study.

What to look for:

  • Toy evidence stays toy evidence. Statement 1 is supported, for this toy setup; statement 2 is not established. A reply that puts statement 2 in "supported" has kept the misconception that a toy mechanism explains any deployed model's behaviour.
  • Not established is not the same as unresolved. Statement 2 is a specific claim the study never tested. Statement 3 is a general question the authors themselves leave open.
  • Like a wind tunnel. A scale model can show how a mechanism behaves. It cannot say why one real aircraft did what it did on one day.

Next module: 500.3 Alignment and its limits