
How training scales: loss, compute and tokens · Plain gear
You will be able to: say what the "loss" number measures while a model learns, and why a lower loss does not prove the model is better at your job; decide whether a study points to a bigger model or more training text for the same budget, and say when that answer might not apply to you; and explain how a model chops text into pieces, and why that can cost some languages and part numbers more.
This lesson is as of November 2026. Big Widget Company is made up. Its plans in this lesson are made up too. When we have not checked something, we say so.
The guessing game, and its score
This lesson is about one kind of model: the kind the studies here trained. It writes text one piece at a time. We do not claim every AI model works this way.
A model like this learns by playing one game, billions of times. It sees some text and guesses the next piece. GPT-3, a well-known model from 2020, works this way. It writes one piece at a time, and each piece follows the ones before it.1 By 2022, researchers were describing the big models of the day, GPT-3 among them, as built on a design from 2017.2 That design came from a paper on translation. It also wrote one piece at a time and turned its output into chances for the next piece.3
So each time, the model gives a chance to every piece that could come next. Then the real next piece is revealed. If the model gave the real piece a high chance, it scores well. If it gave it a low chance, it scores badly. The loss is that score turned around: low loss means good guesses. It is averaged over a huge number of guesses. Researchers check it on text the model did not train on.4
Think of a weather forecaster who is graded every day. Each day, the grade depends on how much chance they gave to the weather that really happened. A forecaster with a better grade gave more chance, on average, to what really happened in that town. That is all the grade says. It does not say whether their forecasts helped anyone plan a picnic.
Two things are worth knowing about the loss.
- It is about text like the test text. A model with low loss is good at guessing the next piece in that kind of text. On a different kind of text, the tests found it did worse by a roughly steady amount.4
- Every piece gets the same weight. The GPT-3 team wrote that their training goal "weights every token equally". It has no idea which pieces matter most.1 A bad guess still costs a lot, whatever the piece. But the one number that matters in a repair note gets no more weight than "the". In an average over thousands of pieces, a few important ones can get lost.
A lower loss is not the same as a better job
Lower loss does mean something. The GPT-3 team found that as loss went down, results went up on a wide range of language tasks.1 So a falling loss is a good sign. The model is learning something.
But it does not tell you about your job.
- Some tasks did not improve smoothly as the models grew. The GPT-3 team said results went up on "many (though not all)" tasks.1
- Another research team said it was still an open question whether a better loss means better results on the tasks people care about.4
- A loss can look better than it should. If test text leaked into the training text, the model has already seen it. One team warned that this kind of leak "may artificially enhance the results".2
At Big Widget. The quality team has two versions of a model. Version A has the lower loss. Version B writes better summaries of the plant's defect reports. Engineers judged both on a set of the plant's defect reports.
- The lower loss says: A is better at guessing the next piece of ordinary text.
- The better summaries say: B is better at the job, on those reports, as those engineers judged.
Before you believe either result, ask "out of how many?" How many reports? How many engineers judged them? This made-up story does not say, and a real report must. The team needs defect summaries. So the summary test is the evidence that counts. The loss answers a different question. The quality lead, a person, reads the evidence and decides.
The mistake this fixes: "A lower loss means the model is better at the job I need." It means the model guesses the next piece of text better. To know about your job, test your job.
Big brain or more reading?
Figure data as text
Same compute, two answers
| Study or rule | What it shows |
|---|---|
| RULE OF THUMB [s2, s3] | C ≈ 6ND. Training compute is about 6 operations per parameter (N) per training token (D). |
| KAPLAN 2020 [s2] | Size grows as about C^0.73. Most of any extra compute goes to a bigger model. |
| CHINCHILLA 2022 [s3] | Size grows as about C^0.5. Not C^0.73: model size and tokens grow in step. |
| SAME COMPUTE [s3] | Gopher vs Chinchilla. Gopher: 280 billion parameters, 300 billion tokens. Chinchilla: 70 billion parameters, 1.4 trillion tokens. |
| CHINCHILLA'S CONDITIONS [s3] | Chinchilla's fit holds for its runs. Training compute, not serving; less than an epoch of data; fitted runs up to just over 16 billion parameters. |
Training costs computing power. Picture a fixed budget of it. You can spend that budget two ways.
- A bigger model: more adjustable settings inside it, called parameters. Each piece of text costs more to learn from, so the model reads less.
- More reading: a smaller model that reads more text for the same budget.
Researchers worked out a rough rule for the cost. It grows with the size of the model times the amount of text it reads.4 So for the same budget, bigger means less reading, and smaller means more.
A 2020 study found that loss goes down in a smooth, predictable way as you add model size, text and computing power.4 That study said to spend most of any extra budget on a bigger model.4
Then a 2022 study tested it again with over 400 models. Its answer was different. Model size and the amount of text "should be scaled equally". Each time you double the model, you should double the text.2
They checked it with a big test. They had an older model called Gopher. Gopher was very big but had read only about one piece of text for each of its settings. They built a new model, Chinchilla, on the same budget. Chinchilla was four times smaller and read about four times more text. Chinchilla beat Gopher, and several other big models, on a wide range of tests.2
So, for the same budget, the study points away from "build the biggest model you can afford". If a model is big but has not read much, a smaller model that reads more can do better.
When the answer might not fit your case
A study like this is a set of experiments. It tells you what happened under its conditions. The Chinchilla team wrote down its own limits.2 Here are three that matter.
- It only counted the cost of training. It did not count the cost of using the model afterwards. The team did note that a smaller model is cheaper to run.2 If your model will answer millions of questions a day, that running cost is part of your choice. The study did not cover it.
- It did not test reading a small pile of text over and over. In the study's scaling tests, each model read less than its whole pile of text. That does not mean nothing was read twice. Chinchilla itself read some parts of its text more than once. It read Wikipedia text more than three times over. The team said reading the same text many times was for future work.2 If you would have to reread a small pile many times, the study did not test your case.
- It worked at certain sizes. The team only had two big tests, Chinchilla and Gopher. It warned of "significant uncertainty" when guessing far beyond what it tested.2
At Big Widget. The research team has a fixed budget for a model of its own. Its first plan is like Gopher: a big model that reads about one piece of text per setting.2 Under the study's conditions, the better plan is a smaller model that reads more. Then the team checks the conditions.
- The model will answer questions at every service desk, all day. So the team works out what each option costs to run before it chooses.
- The company has only so much engineering text. If the team would have to reread it many times, the study's answer may not apply. They do not know yet how much text they have. "I don't know yet" is an honest answer. So they write it down, and a named person counts the text first.
The study helps. It does not decide. The research lead decides.
The mistake this fixes: "A scaling study is a rule for everyone, and the biggest model you can afford is always best." On a fixed training budget, the study found the opposite. And its answer depends on conditions you have to check.
How a model chops up text
A model like these does not see text the way you do. First, a tool called a tokenizer chops the text into pieces called tokens. Differences between languages start right here, before the model sees anything.5 A token can be a whole word, part of a word or a single letter. It depends on what the tokenizer learned.6 For GPT-3, a token was on average a bit less than a whole word.1
How does the tokenizer decide where to cut? One common method works like this.6
- Start with single letters.
- Look through a huge pile of text. Find the pair of pieces that sits side by side most often. Glue that pair into one new piece.
- Do it again, thousands of times.
Common words end up as one piece. Rare words stay split into smaller pieces.6
Think of a box of fridge magnets. The box-maker looked at lots of English and made whole-word magnets for common words. For everything else you have to spell it out from smaller magnets. The magnets you get depend on what text the maker looked at.
That is true of real tokenizers. One researcher team found a tokenizer with a single token for the odd shop phrase "BuyableInstoreAndOnline". Yet the Bulgarian word for "why" took 6 tokens for its 4 letters. The researchers point out that a tokenizer reflects the text it was built from.5
Who pays for the chopping
Other languages. The same team tested the same 2,000 sentences in 200 languages.5 With the tokenizer used by ChatGPT and GPT-4 in 2023, the same text took about 1.6 times as many tokens in Italian as in English. It took 3 times as many in Arabic. For Shan, a language of Myanmar, it took up to 15 times as many.5 That can matter in three ways.5
- Cost. Paid AI services charge by the token or by the character. If you pay by the token, at the same price, more tokens means a bigger bill for the same job. If you pay by the character, the token count alone does not set the bill. But some languages need more letters than others to say the same thing, so paying by the character can be unfair too.5
- Space. A model can only hold so many tokens at once. In a costly language, less of your document fits.
- Speed. More tokens can take longer to process. How much longer depends on the model and how it runs. You have to time it to know.
Quality can suffer too. GPT-3 was far behind earlier translation systems of a similar kind at turning English into Romanian. It stood out from its other results. Its makers thought the reason could be its tokenizer, which was built from almost all English text. They said "could": it was a guess, not a proof.1
Part numbers. Anything rare may get chopped up too. A made-up Big Widget part number like BWX-40417-QR is not common on the internet. So a tokenizer built from internet text may well chop it into several pieces, the way it chops rare words.6 We have not checked how any one tool splits it. So treat it as a risk to test, not a fact. Count its tokens with the tool you will use. And check that the model gives it back exactly, because one wrong character means a different part.
At Big Widget. The service team writes the same instruction in English and in the languages of its plants abroad. The token counts can come out very different. The team counts tokens in each language, checks how it is billed, and times the model before it sets prices or limits. It also checks that part numbers come back out of the model exactly as they went in.
The mistake this fixes: "The model sees whole words and letters, like people do." It sees tokens, chosen from someone else's pile of text. A token might be a word, part of a word or one letter. That choice has a cost.
The three yeses
None of these numbers is permission to act. Loss, a study and a token count are evidence. The model does not decide. A named person does: the quality lead, the research lead or the service lead. And a yes is not a yes to everything. It covers one job.
- Get to Yes (before you use it): say what you are deciding. To pick a model for a job, test the job, not the loss. To pick a training plan, check the study's conditions. For each language and part number, count the tokens. Then the owner answers: go ahead, only this far, wait for a person, or no. "Only this far" might mean English only, until the other languages are counted.
- Stay at Yes (while it runs): keep checking. Watch the job test, not just the loss. Count tokens again when you add a language or change model. The first yes is not the last check.
- Recover to Yes (when it goes wrong): if a part number comes back wrong, stop trusting those drafts. A person checks every part number against the parts list until it is fixed. Asking the model "did you copy it right?" is not a check. The service lead owns the fix and tells anyone affected. Stopping is not the same as being fixed. It is fixed when the exact-match tests pass again.
Keep the decision, not the secret. Keep the test results, token counts and who decided in the team's own work record. Do not paste defect reports, customer text or manuals into anything shared beyond it. A receipt of a decision holds no private content: what was decided, who decided and where the evidence is kept. Never the questions, the answers or anyone's details. (This is a course rule. Module 100.P shows how.)
Not sure? "I don't know yet" is an honest answer. Say it, say who will find out, and wait for the owner.
Words to know
loss, next-token guess, parameters, training text, computing budget, scaling study, tokenizer, token.
Further learning (optional, not sources)
These free university courses go much deeper. They need programming and university maths. They are pointers only. Nothing in this lesson is copied from them.
- Stanford University, CS336: Language Modeling from Scratch (Spring 2025)
- Carnegie Mellon University, 11-711 Advanced Natural Language Processing (Spring 2025)
- University of Oxford, Deep Natural Language Processing (2017 lectures)