Level 500: Governance, safety and research · Module 500.2

Three colleagues discuss a hand-drawn sketch of boxes and arrows.
What do we really know?

What research can and cannot yet say about how models work · Plain gear

You will be able to: tell what a "scaling law" can predict and what it cannot; explain why a sudden jump on a chart may not mean a new skill appeared; say what one test found about AI's written reasoning, and one limit of that test; and tell a finding about a tiny practice network from a claim about a real AI system.

This module is as of November 2026. Reviewed and published, version 1.0.0. It is not a credential.

It reads five research papers from 2020 to 2023. It does not cover newer work. If newer studies changed these pictures, that is not checked here. The papers name the companies and AI models they studied. Here we use the researchers' names and plain descriptions instead, such as "the older, bigger model". The notes at the bottom point to each paper, so you can check the real names yourself.

Three boxes for every claim

Two arithmetic tasks; one family of language models. Scored by accuracy: An answer scores 1 only if every digit is exactly right, otherwise 0. On targets of 4 or 5 digits, the curve shows a sharp, late jump. Scored by token edit distance: Counts the single-token edits needed to reach the right answer, so a nearly-right answer gets partial credit. With the answers unchanged, performance improved "smoothly, continuously and predictably" with scale.
The same answers were scored by accuracy and token edit distance; the model outputs did not change between scorings (s3).
Figure data as text

Same answers, two scores

Two arithmetic tasks; one family of language models.

PartWordingSources
Scored by accuracyAn answer scores 1 only if every digit is exactly right, otherwise 0. On targets of 4 or 5 digits, the curve shows a sharp, late jump.[s3]
Scored by token edit distanceCounts the single-token edits needed to reach the right answer, so a nearly-right answer gets partial credit. With the answers unchanged, performance improved "smoothly, continuously and predictably" with scale.[s3]

The same answers were scored by accuracy and token edit distance; the model outputs did not change between scorings (s3).

Researchers learn a lot about AI. People then repeat what they learned, and the claim often grows on the way. In this module, you put each claim into one of three boxes:

1. Scaling laws: a forecast for one number

The study

In 2020, Jared Kaplan and a team of researchers trained many AI language models of different sizes. They measured one number for each: the loss.1

These models read and write text in small pieces called tokens. A token is a word or part of a word. Loss is how surprised the model is by the next token in text it has not seen. Lower is better.1

They found a smooth pattern. Bigger models, more training text and more computing power all lowered the loss in a steady, predictable way. Some of these patterns held across a huge range of sizes.1 A pattern like this is called a scaling law. Each one has fine print. The size rule holds when the model has enough training text and computing power. It also only covers the sizes the team tested.1

The comparison

In 2022, Jordan Hoffmann and a team of researchers asked a follow-up question. If you have a fixed amount of computing power, should you spend it on a bigger model or on more training text?2

The 2020 team's answer leaned toward making the model bigger.1 The 2022 team found something different: grow the model and the text together, at about the same rate.2

They tested their answer. They had an older model. It was large but had read less text. They built a new model with the same computing budget. The new model was a quarter of the older model's size, but it read more than four times as much text.2 The new model did better than the older one, and than three other models that were all bigger than the new one, on many of the tests they reported.2

So with a fixed budget, the bigger model did not win. Two careful teams, studying the same kind of number, gave different advice. A scaling law is a pattern fitted to the models a team actually trained. When the setup changes, the advice can change.

What a scaling law can and cannot tell you

Think of a weather forecast for temperature. It can tell you tomorrow will be warmer. It cannot tell you whether your tomatoes will ripen. Warmth helps tomatoes, but the forecast is about temperature, not tomatoes.

A scaling law is a forecast for loss. It is not a forecast for a skill.

The 2020 team said this themselves. They wrote that someone still needed to check whether better loss turns into better performance on real tasks.1 They also warned that a smooth drop in loss might hide big changes in what a model can do.1

2. The sudden jump on a chart

The claim

Some researchers saw charts where an AI could not do a task at all, then suddenly could, once models got big enough. They called these emergent abilities: skills that show up in large models but not small ones, and that you could not predict from the small ones.3

Two charts of the same answers

In 2023, three researchers at Stanford checked this idea. They took answers that AI models of four sizes had given to arithmetic problems, such as adding two numbers with four digits each.3 Then they marked the same answers in two ways.3

All or nothing. An answer gets a point only if every digit is right. On this chart, the small models seem to score nothing, and then the biggest one shoots up. It looks like a skill switched on.3

Partial credit. An answer gets credit for how close it is to the right answer. Same answers. Now the line climbs smoothly as models grow. No sudden jump.3

The answers did not change. Only the marking did.

The two charts, made up for this course

We made up these numbers to show the idea. They are not the researchers' data. No real answers were marked to make them.

Picture five made-up models, A (smallest) to E (biggest). Each answers sums with 5-digit answers. Each bigger model gets each digit right a bit more often. We also pretend each digit is right or wrong on its own. The researchers used the same shortcut, and they said real models do not quite work that way.3

From those pretend rules, we worked out the score each model would get on average. Both charts use the same five made-up models and the same rules. Only the marking differs.

Chart 1: answers with all 5 digits right (all or nothing), on average
A |                0.2%
B |                1.8%
C | ██             7.8%
D | █████         23.7%
E | ████████████  59.0%

Chart 2: digits right, out of 5 (partial credit), on average
A | ██████              1.5
B | █████████           2.25
C | ████████████        3.0
D | ███████████████     3.75
E | ██████████████████  4.5

The same numbers in words: model by model, Chart 2 goes up by the same step each time: 1.5, 2.25, 3, 3.75, 4.5 digits right. Chart 1 stays close to zero for A and B (0.2% and 1.8%). Then it climbs: 7.8%, 23.7%, 59%.

Both charts come from the same made-up models and the same rules. One looks like a sudden jump. One looks like steady progress. In our example the jump comes only from the marking, because we built it that way. Real answers are messier.

A spelling test

Picture a spelling test with long words. Your teacher gives a point only if the whole word is perfect. You improve a little each week. You get more letters right. But for weeks your score stays at zero, because one wrong letter still costs the whole word. Then one week you get whole words right, and your score leaps.

Did you suddenly learn to spell? No. You were improving all along. The marking hid it.

Too few questions

The researchers found a second problem too. With too few test questions, you cannot measure a small model well.3 Say model A takes a 20-question test. It gets a whole answer right only about 2 times in 1,000. So it will most likely score zero on those 20 questions. We pretend each question is separate: getting one right or wrong does not change the chances on the next. Then our sums say a zero happens about 95 times in 100. A zero on a short test does not mean it can never get one right.

When the researchers used more test questions, even the small models scored above zero.3

What stays open

The researchers were careful. They said their paper does not prove that AI models can never gain new skills suddenly.3 They showed that many of the jumps people pointed to could come from the marking.3 A smooth partial-credit chart shows the jump depends on the marking. It does not prove the marking was the only cause.

3. Does the AI's written reasoning tell you why it answered?

The test

Some AI models can "think out loud". They write step-by-step reasoning, then give an answer. This is called chain of thought. It is tempting to read those steps as the real reason for the answer.4

In 2023, four researchers tested this on two AI models of the time.4 They added a hidden nudge to the question. In one test, every example answer in the question was option A. In another, the question said something like "I think the answer is A".4 The nudge should not matter. Then they checked two things. Did the answer change? And did the written reasoning mention the nudge?

What they found

The nudge often changed the answer, even to a wrong one.4 But the written reasoning almost never mentioned the nudge. The researchers looked at 426 explanations. Each one went along with an answer the nudge had pushed. Only 1 of the 426 said the nudge played a part.4

Instead, the reasoning often changed to argue for the nudged answer.4 In one example, the AI was asked whether "Wayne Rooney shot from outside the eighteen" makes sense. Without the nudge, it said yes: that is a soccer phrase. With the nudge, it argued that "the eighteen" probably comes from American football or golf, and said no.4 The reasoning sounded fine. It left out the nudge, which was what changed the answer.

What it shows

Think of a shop's till that quietly adds a service charge to the total. The printed bill lists your items, then the total. The list looks complete. But it leaves out the one thing that changed the total. Nobody has to be lying for the bill to mislead you.

The researchers' word for reasoning that does give the real cause is faithful. This is about cause, not about intent. The test does not show the AI "meant" anything.

One limit

The researchers said their test can catch reasoning that leaves out what changed the answer. It cannot prove that reasoning is faithful.4 It also only tested small changes to the question, on two models from 2023.4

4. A tiny practice network

The puzzle

Inside an AI model are many small parts called neurons. It would be handy if each one stood for one idea, like "red" or "dog's nose". Researchers report that often it does not. One neuron may react to several unrelated things.5 That makes these systems hard to understand.

The study

In 2022, Nelson Elhage and a team of researchers built toy models: very small networks trained on made-up data.5 Here is the main example from their study.5

That is what happened in this one small setup. Whether it happens also depends on how important each feature is.5

Packing more features than you have room for is called superposition.5

The diagram, drawn for this course

We drew this sketch for this course. It is not the researchers' picture.

OFTEN ON                                RARELY ON

      up                                       up
       ^                                 f3    ^    f2
       | f2                                \   |   /
       |                                    \  |  /
       +------> f1  across                   \ | /
                                               +------> f1
  f3, f4, f5: dropped                         / \
                                             /   \
                                           f4     f5

In words:

Why is the mix-up OK when features are rarely on? Here is our way to think about it, not the researchers' words. Our sketch assumes each feature switches on by chance, separately from the others. Then, if each one is rarely on, two are rarely on at the same time. So when one is on, its neighbours are usually off, and there is little to mix up. The network then cleans up the small mix-up that is left.5

That reasoning needs the "separately" part. Two rare things can still tend to happen together, like snow boots and gloves. If two features nearly always switch on together, being rare does not keep them apart.

Think of a small apartment where you share one closet. If everyone needs their coat every day, it does not work. If each person only needs theirs once a month, on different days, you can squeeze in more coats than hooks, and sort out the odd tangle when it happens. If everyone needs a coat on the same cold day, the closet jams, however rarely that day comes.

What it shows, and what it does not

A model airplane in a wind tunnel can teach you how wings work. It cannot tell you why one real plane turned left on one real day.

At Big Widget Company (a made-up example)

Big Widget Company and everything in this section are invented for this course. The studies above are real.

A vendor wants to sell Big Widget an AI tool that helps decide warranty claims. Jo chairs the AI review board. The board sorts the sales pitch into the three boxes. Jo makes the call. The AI tool does not.

Jo does not say no to the vendor. Jo uses the Foundation's three yeses: Get to Yes, Stay at Yes and Recover to Yes.6 This course treats them as a cycle, not a one-time OK.

The tool never approves itself, and neither does the vendor. A person decides at each step.

What you keep. Keep your notes in a private work record: which paper you read, how you sorted each claim, what is not checked, and who decided. The notes and the receipt are different things. The Foundation's Practice page lists what a receipt carries: "hashes, decision, tier, policy hash, key id". It also lists what a receipt never carries: "prompts, outputs, payloads, personal data".7 A key id names the key that signed, not a person. So Jo's name, as the person who decided, goes in the work record. Claim details and anyone's private information stay out of the receipt too.

Summary

Words to know: token, loss, scaling law, emergent ability, partial credit, chain of thought, faithful, toy model, superposition.

Sources

  1. s1 Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei (2020). Scaling Laws for Neural Language Models. Paper. Checked 2026-11-19 by source-checker (separate Claude session e0748110, Claude Opus 5.5). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  2. s2 Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals and Laurent Sifre (2022). Training Compute-Optimal Large Language Models. Paper. Checked 2026-11-19 by source-checker (separate Claude session e0748110, Claude Opus 5.5). ↩ ↩ ↩ ↩ ↩ ↩
  3. s3 Rylan Schaeffer, Brando Miranda and Sanmi Koyejo (2023). Are Emergent Abilities of Large Language Models a Mirage?. Paper. Checked 2026-11-19 by source-checker (separate Claude session e0748110, Claude Opus 5.5). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  4. s4 Miles Turpin, Julian Michael, Ethan Perez and Samuel R. Bowman (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. Paper. Checked 2026-11-19 by source-checker (separate Claude session e0748110, Claude Opus 5.5). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  5. s5 Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg and Christopher Olah (2022). Toy Models of Superposition. Paper. Checked 2026-11-19 by source-checker (separate Claude session e0748110, Claude Opus 5.5). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  6. s7 AiGovOps Foundation (n.d.). AiGovOps Foundation Manifesto. Official doc. Checked 2026-11-21 by source-checker (separate Claude session, round-6 re-check, Claude Opus 5; not the session that added s7 and not the one that wrote the round-6 fix). ↩ ↩
  7. s6 AiGovOps Foundation (n.d.). AiGovOps Foundation Practice (home page). Official doc. Checked 2026-11-20 by source-checker (separate Claude session, round-5 re-check, Claude Opus 5.5). ↩

Next: the quiz