Level 400: Engineering AI systems · Module 400.1

Two coworkers at a café table compare two stacks of plain paper, one tall and one short.
Test the job, not just the loss

How training scales: loss, compute and tokens · Plain gear

You will be able to: say what the "loss" number measures while a model learns, and why a lower loss does not prove the model is better at your job; decide whether a study points to a bigger model or more training text for the same budget, and say when that answer might not apply to you; and explain how a model chops text into pieces, and why that can cost some languages and part numbers more.

This lesson is as of November 2026. Big Widget Company is made up. Its plans in this lesson are made up too. When we have not checked something, we say so.

The guessing game, and its score

This lesson is about one kind of model: the kind the studies here trained. It writes text one piece at a time. We do not claim every AI model works this way.

A model like this learns by playing one game, billions of times. It sees some text and guesses the next piece. GPT-3, a well-known model from 2020, works this way. It writes one piece at a time, and each piece follows the ones before it.1 By 2022, researchers were describing the big models of the day, GPT-3 among them, as built on a design from 2017.2 That design came from a paper on translation. It also wrote one piece at a time and turned its output into chances for the next piece.3

So each time, the model gives a chance to every piece that could come next. Then the real next piece is revealed. If the model gave the real piece a high chance, it scores well. If it gave it a low chance, it scores badly. The loss is that score turned around: low loss means good guesses. It is averaged over a huge number of guesses. Researchers check it on text the model did not train on.4

Think of a weather forecaster who is graded every day. Each day, the grade depends on how much chance they gave to the weather that really happened. A forecaster with a better grade gave more chance, on average, to what really happened in that town. That is all the grade says. It does not say whether their forecasts helped anyone plan a picnic.

Two things are worth knowing about the loss.

A lower loss is not the same as a better job

Lower loss does mean something. The GPT-3 team found that as loss went down, results went up on a wide range of language tasks.1 So a falling loss is a good sign. The model is learning something.

But it does not tell you about your job.

At Big Widget. The quality team has two versions of a model. Version A has the lower loss. Version B writes better summaries of the plant's defect reports. Engineers judged both on a set of the plant's defect reports.

Before you believe either result, ask "out of how many?" How many reports? How many engineers judged them? This made-up story does not say, and a real report must. The team needs defect summaries. So the summary test is the evidence that counts. The loss answers a different question. The quality lead, a person, reads the evidence and decides.

The mistake this fixes: "A lower loss means the model is better at the job I need." It means the model guesses the next piece of text better. To know about your job, test your job.

Big brain or more reading?

Compute is about 6 times parameters times tokens. Kaplan's fit puts most extra compute into model size (about C to the 0.73); Chinchilla's puts it equally into size and tokens (about C to the 0.5). On the same compute, Gopher had 280 billion parameters and 300 billion tokens; Chinchilla had 70 billion parameters and 1.4 trillion tokens. The Chinchilla fit holds only for its own conditions: training compute, not serving; less than an epoch of data; and fitted runs up to just over 16 billion parameters.
C ≈ 6ND is a rule of thumb both studies use (s2, s3). Kaplan's fit has model size growing as about C^0.73 (s2); the Chinchilla study's as about C^0.5, tested with Gopher and Chinchilla at the same compute (s3). A scaling law tells you what was true under its experiments' conditions; the Chinchilla authors list counting training compute only, less than an epoch of data and fitted runs up to just over 16 billion parameters (s3).
Figure data as text

Same compute, two answers

Study or ruleWhat it shows
RULE OF THUMB [s2, s3]C ≈ 6ND. Training compute is about 6 operations per parameter (N) per training token (D).
KAPLAN 2020 [s2]Size grows as about C^0.73. Most of any extra compute goes to a bigger model.
CHINCHILLA 2022 [s3]Size grows as about C^0.5. Not C^0.73: model size and tokens grow in step.
SAME COMPUTE [s3]Gopher vs Chinchilla. Gopher: 280 billion parameters, 300 billion tokens. Chinchilla: 70 billion parameters, 1.4 trillion tokens.
CHINCHILLA'S CONDITIONS [s3]Chinchilla's fit holds for its runs. Training compute, not serving; less than an epoch of data; fitted runs up to just over 16 billion parameters.

Training costs computing power. Picture a fixed budget of it. You can spend that budget two ways.

Researchers worked out a rough rule for the cost. It grows with the size of the model times the amount of text it reads.4 So for the same budget, bigger means less reading, and smaller means more.

A 2020 study found that loss goes down in a smooth, predictable way as you add model size, text and computing power.4 That study said to spend most of any extra budget on a bigger model.4

Then a 2022 study tested it again with over 400 models. Its answer was different. Model size and the amount of text "should be scaled equally". Each time you double the model, you should double the text.2

They checked it with a big test. They had an older model called Gopher. Gopher was very big but had read only about one piece of text for each of its settings. They built a new model, Chinchilla, on the same budget. Chinchilla was four times smaller and read about four times more text. Chinchilla beat Gopher, and several other big models, on a wide range of tests.2

So, for the same budget, the study points away from "build the biggest model you can afford". If a model is big but has not read much, a smaller model that reads more can do better.

When the answer might not fit your case

A study like this is a set of experiments. It tells you what happened under its conditions. The Chinchilla team wrote down its own limits.2 Here are three that matter.

At Big Widget. The research team has a fixed budget for a model of its own. Its first plan is like Gopher: a big model that reads about one piece of text per setting.2 Under the study's conditions, the better plan is a smaller model that reads more. Then the team checks the conditions.

The study helps. It does not decide. The research lead decides.

The mistake this fixes: "A scaling study is a rule for everyone, and the biggest model you can afford is always best." On a fixed training budget, the study found the opposite. And its answer depends on conditions you have to check.

How a model chops up text

A model like these does not see text the way you do. First, a tool called a tokenizer chops the text into pieces called tokens. Differences between languages start right here, before the model sees anything.5 A token can be a whole word, part of a word or a single letter. It depends on what the tokenizer learned.6 For GPT-3, a token was on average a bit less than a whole word.1

How does the tokenizer decide where to cut? One common method works like this.6

  1. Start with single letters.
  2. Look through a huge pile of text. Find the pair of pieces that sits side by side most often. Glue that pair into one new piece.
  3. Do it again, thousands of times.

Common words end up as one piece. Rare words stay split into smaller pieces.6

Think of a box of fridge magnets. The box-maker looked at lots of English and made whole-word magnets for common words. For everything else you have to spell it out from smaller magnets. The magnets you get depend on what text the maker looked at.

That is true of real tokenizers. One researcher team found a tokenizer with a single token for the odd shop phrase "BuyableInstoreAndOnline". Yet the Bulgarian word for "why" took 6 tokens for its 4 letters. The researchers point out that a tokenizer reflects the text it was built from.5

Who pays for the chopping

Other languages. The same team tested the same 2,000 sentences in 200 languages.5 With the tokenizer used by ChatGPT and GPT-4 in 2023, the same text took about 1.6 times as many tokens in Italian as in English. It took 3 times as many in Arabic. For Shan, a language of Myanmar, it took up to 15 times as many.5 That can matter in three ways.5

Quality can suffer too. GPT-3 was far behind earlier translation systems of a similar kind at turning English into Romanian. It stood out from its other results. Its makers thought the reason could be its tokenizer, which was built from almost all English text. They said "could": it was a guess, not a proof.1

Part numbers. Anything rare may get chopped up too. A made-up Big Widget part number like BWX-40417-QR is not common on the internet. So a tokenizer built from internet text may well chop it into several pieces, the way it chops rare words.6 We have not checked how any one tool splits it. So treat it as a risk to test, not a fact. Count its tokens with the tool you will use. And check that the model gives it back exactly, because one wrong character means a different part.

At Big Widget. The service team writes the same instruction in English and in the languages of its plants abroad. The token counts can come out very different. The team counts tokens in each language, checks how it is billed, and times the model before it sets prices or limits. It also checks that part numbers come back out of the model exactly as they went in.

The mistake this fixes: "The model sees whole words and letters, like people do." It sees tokens, chosen from someone else's pile of text. A token might be a word, part of a word or one letter. That choice has a cost.

The three yeses

None of these numbers is permission to act. Loss, a study and a token count are evidence. The model does not decide. A named person does: the quality lead, the research lead or the service lead. And a yes is not a yes to everything. It covers one job.

Keep the decision, not the secret. Keep the test results, token counts and who decided in the team's own work record. Do not paste defect reports, customer text or manuals into anything shared beyond it. A receipt of a decision holds no private content: what was decided, who decided and where the evidence is kept. Never the questions, the answers or anyone's details. (This is a course rule. Module 100.P shows how.)

Not sure? "I don't know yet" is an honest answer. Say it, say who will find out, and wait for the owner.

Words to know

loss, next-token guess, parameters, training text, computing budget, scaling study, tokenizer, token.

Further learning (optional, not sources)

These free university courses go much deeper. They need programming and university maths. They are pointers only. Nothing in this lesson is copied from them.

Sources

  1. s6 Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al. (2020). Language Models are Few-Shot Learners. Paper. Checked 2026-11-19 by source-checker role, separate Claude session (Claude Opus 5.5), round-3 re-check 2026-11-19; see checks/source-check.md. ↩ ↩ ↩ ↩ ↩ ↩
  2. s3 Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, et al. (2022). Training Compute-Optimal Large Language Models. Paper. Checked 2026-11-19 by source-checker role, separate Claude session (Claude Opus 5.5), round-3 re-check 2026-11-19; see checks/source-check.md. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s1 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin (2017). Attention Is All You Need. Paper. Checked 2026-11-19 by source-checker role, separate Claude session (Claude Opus 5.5), round-2 re-check 2026-11-19; see checks/source-check.md. ↩
  4. s2 Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, et al. (2020). Scaling Laws for Neural Language Models. Paper. Checked 2026-11-19 by source-checker role, separate Claude session (Claude Opus 5.5), round-2 re-check 2026-11-19; see checks/source-check.md. ↩ ↩ ↩ ↩ ↩ ↩
  5. s5 Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi (2023). Language Model Tokenizers Introduce Unfairness Between Languages. Paper. Checked 2026-11-19 by source-checker role, separate Claude session (Claude Opus 5.5), round-3 re-check 2026-11-19; see checks/source-check.md. ↩ ↩ ↩ ↩ ↩ ↩
  6. s4 Rico Sennrich, Barry Haddow, and Alexandra Birch (2016). Neural Machine Translation of Rare Words with Subword Units. Paper. Checked 2026-11-19 by source-checker role, separate Claude session (Claude Opus 5.5), round-2 re-check 2026-11-19; see checks/source-check.md. ↩ ↩ ↩ ↩

Next: the quiz