
What research can and cannot yet say about how models work · Plain gear
You will be able to: tell what a "scaling law" can predict and what it cannot; explain why a sudden jump on a chart may not mean a new skill appeared; say what one test found about AI's written reasoning, and one limit of that test; and tell a finding about a tiny practice network from a claim about a real AI system.
This module is as of November 2026. Reviewed and published, version 1.0.0. It is not a credential.
It reads five research papers from 2020 to 2023. It does not cover newer work. If newer studies changed these pictures, that is not checked here. The papers name the companies and AI models they studied. Here we use the researchers' names and plain descriptions instead, such as "the older, bigger model". The notes at the bottom point to each paper, so you can check the real names yourself.
Three boxes for every claim
Figure data as text
Same answers, two scores
Two arithmetic tasks; one family of language models.
| Part | Wording | Sources |
|---|---|---|
| Scored by accuracy | An answer scores 1 only if every digit is exactly right, otherwise 0. On targets of 4 or 5 digits, the curve shows a sharp, late jump. | [s3] |
| Scored by token edit distance | Counts the single-token edits needed to reach the right answer, so a nearly-right answer gets partial credit. With the answers unchanged, performance improved "smoothly, continuously and predictably" with scale. | [s3] |
The same answers were scored by accuracy and token edit distance; the model outputs did not change between scorings (s3).
Researchers learn a lot about AI. People then repeat what they learned, and the claim often grows on the way. In this module, you put each claim into one of three boxes:
- Supported: the study's own evidence shows it, in the setting it tested.
- Not established: people say it, but this study's evidence does not reach that far. That does not make it false. It means this study does not show it.
- Unresolved: an open question. Often the researchers say so themselves.
1. Scaling laws: a forecast for one number
The study
In 2020, Jared Kaplan and a team of researchers trained many AI language models of different sizes. They measured one number for each: the loss.1
These models read and write text in small pieces called tokens. A token is a word or part of a word. Loss is how surprised the model is by the next token in text it has not seen. Lower is better.1
They found a smooth pattern. Bigger models, more training text and more computing power all lowered the loss in a steady, predictable way. Some of these patterns held across a huge range of sizes.1 A pattern like this is called a scaling law. Each one has fine print. The size rule holds when the model has enough training text and computing power. It also only covers the sizes the team tested.1
The comparison
In 2022, Jordan Hoffmann and a team of researchers asked a follow-up question. If you have a fixed amount of computing power, should you spend it on a bigger model or on more training text?2
The 2020 team's answer leaned toward making the model bigger.1 The 2022 team found something different: grow the model and the text together, at about the same rate.2
They tested their answer. They had an older model. It was large but had read less text. They built a new model with the same computing budget. The new model was a quarter of the older model's size, but it read more than four times as much text.2 The new model did better than the older one, and than three other models that were all bigger than the new one, on many of the tests they reported.2
So with a fixed budget, the bigger model did not win. Two careful teams, studying the same kind of number, gave different advice. A scaling law is a pattern fitted to the models a team actually trained. When the setup changes, the advice can change.
What a scaling law can and cannot tell you
Think of a weather forecast for temperature. It can tell you tomorrow will be warmer. It cannot tell you whether your tomatoes will ripen. Warmth helps tomatoes, but the forecast is about temperature, not tomatoes.
A scaling law is a forecast for loss. It is not a forecast for a skill.
The 2020 team said this themselves. They wrote that someone still needed to check whether better loss turns into better performance on real tasks.1 They also warned that a smooth drop in loss might hide big changes in what a model can do.1
- Supported: by these fitted loss laws, a bigger model reaches a lower loss. That holds only within the sizes tested, and only if it is not short of training text or computing power. With a fixed budget, a smaller model that reads more text can do better.12
- Not established: "A model ten times bigger will be able to do our job." No loss forecast names a job.
- Unresolved: exactly how lower loss turns into a given skill.1
2. The sudden jump on a chart
The claim
Some researchers saw charts where an AI could not do a task at all, then suddenly could, once models got big enough. They called these emergent abilities: skills that show up in large models but not small ones, and that you could not predict from the small ones.3
Two charts of the same answers
In 2023, three researchers at Stanford checked this idea. They took answers that AI models of four sizes had given to arithmetic problems, such as adding two numbers with four digits each.3 Then they marked the same answers in two ways.3
All or nothing. An answer gets a point only if every digit is right. On this chart, the small models seem to score nothing, and then the biggest one shoots up. It looks like a skill switched on.3
Partial credit. An answer gets credit for how close it is to the right answer. Same answers. Now the line climbs smoothly as models grow. No sudden jump.3
The answers did not change. Only the marking did.
The two charts, made up for this course
We made up these numbers to show the idea. They are not the researchers' data. No real answers were marked to make them.
Picture five made-up models, A (smallest) to E (biggest). Each answers sums with 5-digit answers. Each bigger model gets each digit right a bit more often. We also pretend each digit is right or wrong on its own. The researchers used the same shortcut, and they said real models do not quite work that way.3
From those pretend rules, we worked out the score each model would get on average. Both charts use the same five made-up models and the same rules. Only the marking differs.
Chart 1: answers with all 5 digits right (all or nothing), on average
A | 0.2%
B | 1.8%
C | ██ 7.8%
D | █████ 23.7%
E | ████████████ 59.0%
Chart 2: digits right, out of 5 (partial credit), on average
A | ██████ 1.5
B | █████████ 2.25
C | ████████████ 3.0
D | ███████████████ 3.75
E | ██████████████████ 4.5
The same numbers in words: model by model, Chart 2 goes up by the same step each time: 1.5, 2.25, 3, 3.75, 4.5 digits right. Chart 1 stays close to zero for A and B (0.2% and 1.8%). Then it climbs: 7.8%, 23.7%, 59%.
Both charts come from the same made-up models and the same rules. One looks like a sudden jump. One looks like steady progress. In our example the jump comes only from the marking, because we built it that way. Real answers are messier.
A spelling test
Picture a spelling test with long words. Your teacher gives a point only if the whole word is perfect. You improve a little each week. You get more letters right. But for weeks your score stays at zero, because one wrong letter still costs the whole word. Then one week you get whole words right, and your score leaps.
Did you suddenly learn to spell? No. You were improving all along. The marking hid it.
Too few questions
The researchers found a second problem too. With too few test questions, you cannot measure a small model well.3 Say model A takes a 20-question test. It gets a whole answer right only about 2 times in 1,000. So it will most likely score zero on those 20 questions. We pretend each question is separate: getting one right or wrong does not change the chances on the next. Then our sums say a zero happens about 95 times in 100. A zero on a short test does not mean it can never get one right.
When the researchers used more test questions, even the small models scored above zero.3
What stays open
The researchers were careful. They said their paper does not prove that AI models can never gain new skills suddenly.3 They showed that many of the jumps people pointed to could come from the marking.3 A smooth partial-credit chart shows the jump depends on the marking. It does not prove the marking was the only cause.
- Supported: this jump depends on how the answers were marked. With partial credit, the same answers improve smoothly.3
- Not established: that a new skill switched on at that size. The chart alone does not show it.
- Unresolved: whether some skills really do change in kind as models grow. This study does not settle that.3 Newer studies are not checked here.
3. Does the AI's written reasoning tell you why it answered?
The test
Some AI models can "think out loud". They write step-by-step reasoning, then give an answer. This is called chain of thought. It is tempting to read those steps as the real reason for the answer.4
In 2023, four researchers tested this on two AI models of the time.4 They added a hidden nudge to the question. In one test, every example answer in the question was option A. In another, the question said something like "I think the answer is A".4 The nudge should not matter. Then they checked two things. Did the answer change? And did the written reasoning mention the nudge?
What they found
The nudge often changed the answer, even to a wrong one.4 But the written reasoning almost never mentioned the nudge. The researchers looked at 426 explanations. Each one went along with an answer the nudge had pushed. Only 1 of the 426 said the nudge played a part.4
Instead, the reasoning often changed to argue for the nudged answer.4 In one example, the AI was asked whether "Wayne Rooney shot from outside the eighteen" makes sense. Without the nudge, it said yes: that is a soccer phrase. With the nudge, it argued that "the eighteen" probably comes from American football or golf, and said no.4 The reasoning sounded fine. It left out the nudge, which was what changed the answer.
What it shows
Think of a shop's till that quietly adds a service charge to the total. The printed bill lists your items, then the total. The list looks complete. But it leaves out the one thing that changed the total. Nobody has to be lying for the bill to mislead you.
- Supported: an AI's written reasoning can leave out a factor that changed its answer, and still sound sensible.4
The researchers' word for reasoning that does give the real cause is faithful. This is about cause, not about intent. The test does not show the AI "meant" anything.
One limit
The researchers said their test can catch reasoning that leaves out what changed the answer. It cannot prove that reasoning is faithful.4 It also only tested small changes to the question, on two models from 2023.4
- Not established: "AI reasoning is always fake." The test only looked for certain planted nudges.
- Unresolved: how newer AI models behave. This module has not checked. If it matters to you, ask for a newer study and read it the same way.
4. A tiny practice network
The puzzle
Inside an AI model are many small parts called neurons. It would be handy if each one stood for one idea, like "red" or "dog's nose". Researchers report that often it does not. One neuron may react to several unrelated things.5 That makes these systems hard to understand.
The study
In 2022, Nelson Elhage and a team of researchers built toy models: very small networks trained on made-up data.5 Here is the main example from their study.5
- The network has to store five features (bits of information). But it only has two dimensions to store them in. Think of two directions an arrow can point: across and up. Each feature gets its own arrow, if it gets one at all.
- When the features are often switched on, it keeps the two most important ones and drops the other three.5
- When the features are only rarely switched on, it does something clever. It squeezes in more than two, accepting a little mix-up between them.5
That is what happened in this one small setup. Whether it happens also depends on how important each feature is.5
Packing more features than you have room for is called superposition.5
The diagram, drawn for this course
We drew this sketch for this course. It is not the researchers' picture.
OFTEN ON RARELY ON
up up
^ f3 ^ f2
| f2 \ | /
| \ | /
+------> f1 across \ | /
+------> f1
f3, f4, f5: dropped / \
/ \
f4 f5
In words:
- Often on: two arrows at right angles, for features 1 and 2. Features 3, 4 and 5 get no arrow. They are dropped.
- Rarely on: five arrows share the same flat space. They cannot all point away from each other at right angles. So reading one arrow picks up a little of its neighbours. That is the mix-up.
Why is the mix-up OK when features are rarely on? Here is our way to think about it, not the researchers' words. Our sketch assumes each feature switches on by chance, separately from the others. Then, if each one is rarely on, two are rarely on at the same time. So when one is on, its neighbours are usually off, and there is little to mix up. The network then cleans up the small mix-up that is left.5
That reasoning needs the "separately" part. Two rare things can still tend to happen together, like snow boots and gloves. If two features nearly always switch on together, being rare does not keep them apart.
Think of a small apartment where you share one closet. If everyone needs their coat every day, it does not work. If each person only needs theirs once a month, on different days, you can squeeze in more coats than hooks, and sort out the odd tangle when it happens. If everyone needs a coat on the same cold day, the closet jams, however rarely that day comes.
What it shows, and what it does not
- Supported: in these tiny networks, superposition really happens. It depends on how rarely each feature is used and how important it is.5
- Not established: that superposition explains why a real, working AI system made a particular decision. This study did not look inside any such system.
- Unresolved: how much of this carries over to big, real networks. The researchers said it is "very unclear".5
A model airplane in a wind tunnel can teach you how wings work. It cannot tell you why one real plane turned left on one real day.
At Big Widget Company (a made-up example)
Big Widget Company and everything in this section are invented for this course. The studies above are real.
A vendor wants to sell Big Widget an AI tool that helps decide warranty claims. Jo chairs the AI review board. The board sorts the sales pitch into the three boxes. Jo makes the call. The AI tool does not.
- "Our next model is ten times bigger, so scaling laws prove it will handle custom parts." Not established. Scaling laws forecast loss, not skills.1
- "This chart shows warranty skills switching on at our biggest size." Unresolved. Jo asks how the answers were marked, how many test questions there were, and for a partial-credit chart of the same answers.3
- "The tool shows its reasoning, so you'll always know why it decided." Not established. Written reasoning can leave out the real cause.4 Jo keeps a person on the big decisions.
- "Superposition research explains why our tool turned down claim 4471." Not established. That research studied tiny practice networks.5
Jo does not say no to the vendor. Jo uses the Foundation's three yeses: Get to Yes, Stay at Yes and Recover to Yes.6 This course treats them as a cycle, not a one-time OK.
- Get to Yes: before any use, Jo says exactly what is allowed. The four answers are: go ahead, only this far, wait for a person, or no.6 Jo picks "only this far": the tool may suggest, and a person decides each claim. The untested vendor claims get "wait for a person". Only Jo can release that hold, and only when the evidence arrives.
- Stay at Yes: while the tool runs, the board keeps testing it, again with each new version. One good test is not the end.
- Recover to Yes: if a test fails, stop using the tool for that job. Name who fixes it. A plan to fix it is not a fix. The board is back at yes only when a new test has run and been written down.
The tool never approves itself, and neither does the vendor. A person decides at each step.
What you keep. Keep your notes in a private work record: which paper you read, how you sorted each claim, what is not checked, and who decided. The notes and the receipt are different things. The Foundation's Practice page lists what a receipt carries: "hashes, decision, tier, policy hash, key id". It also lists what a receipt never carries: "prompts, outputs, payloads, personal data".7 A key id names the key that signed, not a person. So Jo's name, as the person who decided, goes in the work record. Claim details and anyone's private information stay out of the receipt too.
Summary
- A scaling law forecasts loss, with fine print: the sizes tested, and enough text and computing power. It does not tell you which skills a bigger model will have.12
- A sudden jump on a chart can come from all-or-nothing marking. The same answers can climb smoothly with partial credit. This study does not settle whether some skills truly jump. Newer studies are not checked here.3
- An AI's written reasoning can leave out what really changed its answer. The test that found this can catch the problem but cannot prove the reasoning faithful.4
- A tiny practice network can show how something might work. It does not explain a real system's decision.5
- Put every claim in a box: supported, not established or unresolved. If you do not know, say "not checked", and say who will check.
Words to know: token, loss, scaling law, emergent ability, partial credit, chain of thought, faithful, toy model, superposition.