
How training scales: loss, compute and tokens · Standard gear
You will be able to: explain what next-token cross-entropy loss measures during pre-training, and what a lower loss does and does not say about a Big Widget task; read a published scaling study's conditions and decide whether, for the same compute, it points to more parameters or more training tokens, and name a condition under which that answer would not carry over; and explain how a subword tokenizer splits text, and spot the cost and quality gaps it creates across languages and part numbers.
Facts about particular models, tokenizers and studies are dated in the text. This module is as of November 2026. Big Widget Company is the made-up company used across this course; its plans and numbers here are made up for teaching. Where we did not check something, we say "not checked".
1. What the loss measures
This module is about one kind of model: the autoregressive text models that the studies below trained, which read text as subword tokens. It does not claim that every language model works this way.
An autoregressive model is trained on one job: given the text so far, predict the next token. The GPT-3 paper (2020) describes GPT-3 as "an autoregressive language model with 175 billion parameters": it produces text one token at a time, each one following the tokens before it.1 The Transformer design was introduced in 2017 for translation. Its decoder already worked this way, "consuming the previously generated symbols as additional input when generating the next", and it turned its output into "predicted next-token probabilities".2 By 2022, the Chinchilla authors were describing the large language models of the day as "large autoregressive transformers", citing that 2017 paper.3
So at every position the model gives a probability to every token it could produce next. The cross-entropy loss looks only at the probability the model gave to the token that actually came next. If that probability was high, the loss at that position is small. If it was low, the loss is large. The training loss is this penalty averaged over a very large number of positions. Kaplan and colleagues, in a 2020 study of how loss scales, measured it in nats (a unit of information) averaged over a 1,024-token context. They recorded it on a held-out test set and on other kinds of text.4
Two things follow from that definition.
- Loss is about predicting text like the training text. A low loss on held-out text means the model is good at guessing the next token in text of that kind. Kaplan's team found that when a model is tested on a different kind of text, it pays a roughly constant penalty but otherwise improves in step with its training-set performance.4 The number still belongs to a particular kind of text.
- Every position gets the same weight in the average. The GPT-3 authors put it plainly: "Our current objective weights every token equally and lacks a notion of what is most important to predict and what is less important."1 Same weight is not same penalty. A token the model gave a low probability adds a large penalty, whatever it is. But the one torque value that matters in a repair note gets no extra weight over "the". Averaged over many thousands of positions, a few important tokens can be drowned out.
2. What a lower loss does and does not tell you
What it does tell you. Across many tasks, lower loss has gone with better results. Plotting validation loss against training compute, the GPT-3 authors found a smooth power-law trend. They raised the worry that "these improvements in cross-entropy loss come only from modeling spurious details of our training corpus", and answered that the improvements "lead to consistent performance gains across a broad spectrum of natural language tasks".1 So a falling loss is real evidence that the model is learning something general.
What it does not tell you. It does not tell you how well the model does your task.
- Not all tasks follow the loss. The same authors report "relatively smooth increases in many (though not all) downstream tasks", and name tasks "on which few-shot performance struggles, even at the scale of GPT-3".1
- The question was left open. In their discussion, Kaplan's team wrote that "it will be important to investigate whether continued improvement on the loss translates into improvement on relevant language tasks".4
- A low score can be flattered. When the Chinchilla authors reported their model's language-modelling results, they warned that because Chinchilla saw four times more data than Gopher, "train/test set leakage may artificially enhance the results".3 A loss is only as honest as the separation between the text the model trained on and the text it is tested on.
At Big Widget. The plant's quality team has two checkpoints of an in-house model. Checkpoint A has the lower validation loss on held-out general text. Checkpoint B scores higher on the job they care about: a test set of the plant's own defect reports, where engineers judged each model's summaries. Each number supports a different claim:
- The lower loss supports: "A predicts held-out text of that kind better."
- The higher task score supports: "B writes better defect-report summaries, on this test set, as these engineers judged them."
Before you believe either number, ask "out of how many?" How many defect reports were in the test set, and how many engineers judged them? This made-up example does not give those counts, and a real report must. If the decision is which checkpoint summarises defect reports, the task score is the evidence that answers it. The loss is not; it measures something else. Neither number decides on its own: the quality lead reads the evidence and makes the call. (Module 400.6 covers how to make a task score hold up, with error bars and a test set the model has not seen.)
The misconception this corrects: "A lower training loss means the model is better at the job I need." Lower loss means better next-token prediction on text like the test text, every position weighted the same. Whether that helps your job is a separate question, answered by testing the job.
3. Scaling laws: loss, size, data and compute
Figure data as text
Same compute, two answers
| Study or rule | What it shows |
|---|---|
| RULE OF THUMB [s2, s3] | C ≈ 6ND. Training compute is about 6 operations per parameter (N) per training token (D). |
| KAPLAN 2020 [s2] | Size grows as about C^0.73. Most of any extra compute goes to a bigger model. |
| CHINCHILLA 2022 [s3] | Size grows as about C^0.5. Not C^0.73: model size and tokens grow in step. |
| SAME COMPUTE [s3] | Gopher vs Chinchilla. Gopher: 280 billion parameters, 300 billion tokens. Chinchilla: 70 billion parameters, 1.4 trillion tokens. |
| CHINCHILLA'S CONDITIONS [s3] | Chinchilla's fit holds for its runs. Training compute, not serving; less than an epoch of data; fitted runs up to just over 16 billion parameters. |
Three things set how far training can push the loss down: the number of parameters in the model (N), the number of training tokens (D) and the compute spent (C). Kaplan's team found that "The loss scales as a power-law with model size, dataset size, and the amount of compute used for training": a smooth, predictable curve.4 They used a rule of thumb for compute: training costs about 6 floating-point operations per parameter per training token, so C ≈ 6ND.4 The Chinchilla study used the same approximation.3
That rule of thumb shows the trade-off this module is about. For a fixed compute budget, a bigger model means fewer training tokens, and a smaller model means more. Which is better?
Kaplan's team (2020, a preprint) concluded that "Larger models are significantly more sample-efficient", so that compute-efficient training means "training very large models on a relatively modest amount of data and stopping significantly before convergence".4 By their fit, as compute grows, model size should grow roughly as C^0.73: most of any extra compute goes to a bigger model.4 They also said the trends cannot go on for ever: "performance must flatten out eventually before reaching zero loss".4
4. The Chinchilla result: same compute, smaller model, more tokens
In 2022 Jordan Hoffmann and colleagues at DeepMind revisited the question, in a paper later published at NeurIPS 2022.3 They trained over 400 language models, from 70 million to over 16 billion parameters, and found that "the model size and the number of training tokens should be scaled equally": "for every doubling of model size the number of training tokens should also be doubled".3 Three different ways of fitting the data gave similar answers: model size should grow roughly as C^0.5, not C^0.73.3 By that analysis, the large models of the day were "significantly undertrained".3
They tested the prediction at full scale. Gopher, an earlier DeepMind model, had 280 billion parameters trained on 300 billion tokens. Chinchilla, with the same compute budget, had 70 billion parameters trained on 1.4 trillion tokens: four times smaller, with about four times more data, as the authors round it.3 Chinchilla "uniformly and significantly" outperformed Gopher, GPT-3, Jurassic-1 and Megatron-Turing NLG on a large range of downstream evaluation tasks.3
So for the same compute, the study points away from "biggest model you can afford" and towards a smaller model trained on more tokens, when the starting plan is undertrained for its size.
5. When the result might not carry over
A scaling law is a fit to a set of experiments. It tells you what was true under those experiments' conditions. The Chinchilla authors list several of their own limits, and two scaling studies already disagree. Treat a scaling law as evidence with conditions, not a universal rule.
- It counts training compute, not serving. The study asks which model gives the lowest loss for a fixed training budget.3 It does not set out to minimise the cost of running the model afterwards, though it notes that Chinchilla's "reduced model size reduces inference cost considerably" and that it "uses substantially less compute for fine-tuning and inference".3 If a model will answer millions of requests a day, the cost of serving it is part of the decision, and the study's answer leaves it out. The study alone cannot settle that; work it out with your own numbers.
- It did not test many passes over a small corpus. "The training runs for our analysis have all been trained on less than an epoch of data"; the authors say "future work may consider the multiple epoch regime".3 "Less than an epoch" means each run used fewer tokens than the whole corpus held.3 It does not mean no text was ever repeated. Chinchilla's own 1.4-trillion-token run sampled some parts of its data more than once: about 1.24 passes over its MassiveWeb subset and 3.40 over Wikipedia.3 What the scaling fits did not cover is a small corpus repeated many times over. If you would have to do that to reach the token count, the study did not test your situation. The authors also note, speculatively, that "scaling to larger and larger datasets is only beneficial when the data is high-quality".3
- It was tested at certain scales. The fitted runs went up to just over 16 billion parameters, and "we only have two comparable training runs at large scale (Chinchilla and Gopher)".3 They saw "some concavity" in the optimal model size at high compute budgets, and wrote that "there is significant uncertainty extrapolating out many orders of magnitude".3 A budget far outside the tested range is an extrapolation.
- It measured loss, on its own data. The study's target was loss, on its own training dataset, MassiveText.3 As section 2 showed, loss is not your task score.
At Big Widget. The research team has a fixed compute budget for a model trained on the company's service and engineering text. Their first plan uses roughly as many training tokens as parameters, much like Gopher's 300 billion tokens for 280 billion parameters, which the Chinchilla study found undertrained.3 Under the study's conditions (no small corpus repeated many times, a budget in its tested range, training cost only), the reallocation is: fewer parameters, more tokens, same compute. Then the team asks whether the conditions hold:
- The model will draft answers for every service desk in every region, all day. Serving cost was outside the study's question, so they work out the cost of running each option before choosing.
- The company has only so much engineering text. If reaching the token count means repeating it many times, the study's answer is untested for them. They do not know yet how much usable text they have. "I don't know yet" is an honest answer, so they write it down, and a named data engineer counts the text before the plan goes further.
The study is evidence for the decision, not the decision. The research lead owns the choice of plan.
The misconception this corrects: "A scaling law is a universal rule, and the biggest model you can afford is always the best choice." The Chinchilla study found the opposite of "biggest is best" for a fixed training budget, and its answer itself depends on conditions you must check.
6. Tokens: how a subword tokenizer splits text
Before any text reaches a model like these, a tokenizer cuts it into tokens from a fixed vocabulary, and the model works on those tokens. Differences between languages start "at the tokenization stage, well before the language model sees any data at all".5 A token can be a whole word, a piece of a word or a single character, depending on what the tokenizer learned.6 The GPT-3 authors note that their encoding "operates on significant fractions of a word (on average ∼0.7 words per token)". They give this as part of why tasks that need letter-by-letter work are hard for it: the model has to pull apart the pieces to get at the letters.1
"Subword tokenization is currently the preferred approach for state of the art language models", as Petrov and colleagues wrote in 2023.5 One common method is byte-pair encoding (BPE), adapted for splitting words by Sennrich, Haddow and Birch (2016).6 It learns its vocabulary from a training corpus:
- Start with single characters as the vocabulary.
- Count every pair of neighbouring symbols in the corpus, and merge the most frequent pair into a new symbol.
- Repeat. The final vocabulary is the starting characters plus the number of merges, and the number of merges is "the only hyperparameter of the algorithm".6
Because merges follow frequency, "Frequent character n-grams (or whole words) are eventually merged into a single symbol", while rare and unknown words are encoded "as sequences of subword units".6 The Transformer paper's English-German experiments already used byte-pair encoding, with a shared vocabulary of about 37,000 tokens.2
The key consequence: the tokenizer's vocabulary reflects the text it was built from. Petrov and colleagues (2023) found that the GPT-2 tokenizer has a token for the odd string "BuyableInstoreAndOnline", while a common Arabic word is split into one token per letter and the Bulgarian word for "why" (защо) takes 6 tokens for its 4 letters. As they put it: "If one uses non-natural inputs, log files, or specialist forums, the tokenizer vocabulary would reflect this."5
7. The gaps a tokenizer creates
Across languages. Petrov's team measured how many tokens different tokenizers need for the same 2,000 sentences translated into 200 languages (the FLORES-200 set).5 For the tokenizer used by ChatGPT and GPT-4 at the time (2023), the same text took about 1.6 times as many tokens in Italian as in English, 2.6 times in Bulgarian and 3 times in Arabic; for Shan, a language of Myanmar, the gap reached 15 times.5 They traced three effects. Each depends on how the service runs, so keep them apart.5
- Cost depends on how you are billed. "Commercial services charge users per token or Unicode character." When the price is per token and the same for every language, more tokens for the same text means a higher charge for that text; the authors found users of some languages "paying at least 2.5 times more for the same task as users of English". If the price is per character, the token count alone does not set the price. That does not remove the gap: the same text takes more characters in some languages than in others, so per-character billing has gaps of its own.5
- Context is fixed in tokens. A model's context holds a fixed number of tokens, so users of token-efficient languages can fit texts "more than an order of magnitude longer" than users of others.
- Latency has to be measured. More tokens can take longer to process; the authors report that some languages "can require twice the time to process the same content as English". How much longer depends on the model and how it is served. It is not a fixed ratio you can read off the token count.
Quality can suffer too. GPT-3's English-to-Romanian translation was "a noticeable outlier", over 10 BLEU worse than earlier unsupervised translation systems. Its authors suggested this "could be a weakness due to reusing the byte-level BPE tokenizer of GPT-2 which was developed for an almost entirely English training dataset".1 They offered it as a possible cause, not a proven one.
Part numbers and codes. The same mechanism applies to anything rare in the tokenizer's training text. A Big Widget part number or fault code (made up here, such as BWX-40417-QR) is not a common string on the public web. A tokenizer built from public text may well split it into several tokens, as BPE does with rare words.6 We have not run it through any tokenizer, so we do not know the split. Treat this as a risk to test, not a fact: count its tokens with the tokenizer you will deploy, and test that the model gives back the exact string, because one changed character makes it a different part.
At Big Widget. The service team writes the same instruction in English and in the languages of its plants abroad. Run through one tokenizer, the versions can come out with very different token counts. The team counts tokens in every language it supports, checks how its provider bills, and measures latency on its own setup before it sets prices, limits or context budgets. It also tests that part numbers survive a round trip through the model unchanged.
The misconception this corrects: "The model sees whole words and letters the way people do." A model like these sees tokens from a vocabulary learned from someone else's text. A token may be a word, a piece of one or a single character, and that choice has costs.
8. The three yeses for a training or tokenizer decision
None of the numbers in this module is permission to act. A loss, a scaling fit or a token count is evidence. The model does not decide, and neither does the metric: a named person does. A yes is not a blanket licence either. It covers one checkpoint, one plan or one language set, for one job. At Big Widget the owners are the quality lead for checkpoints, the research lead for the training plan and the service lead for per-language prices, limits and part-number use.
- Get to Yes (before you train or ship): say exactly what you are deciding and what evidence fits it. For a checkpoint, that is a task score on your own job, with its counts, not the loss. For a training plan, check each of the study's conditions against your setting. For a language or part number, count tokens with the tokenizer you will deploy. Then the owner gives one of four answers: go ahead, only this far (for example, English only until the other languages are measured), wait for a person, or no.
- Stay at Yes (while it runs): keep checking. Watch the task score, not just the loss, as the model changes. Re-count tokens when you add a language or change model, since a new tokenizer means new counts. Keep the exact-match test for part numbers running. The first yes is not the last check.
- Recover to Yes (when it goes wrong): if a part number comes back changed, hold that use. Drafts that quote part numbers wait for a person to check them against the parts list. Do not ask the model whether it copied the number right; asking the same model again is not a check. The service lead owns the fix and tells anyone affected. Holding the drafts is not the same as being back at yes. You are back at yes when the exact-match tests pass again on the fixed setup.
Keep the decision, not the evidence. Keep the evidence (test sets, scores, token counts, who decided and when) in the team's own controlled work record. Do not paste defect reports, customer text or manuals into anything shared beyond that record. A receipt of a gate decision holds no private content: it carries the decision, who made it and references to the evidence, never prompts, model outputs, customer text or personal details. It shows that a decision was recorded, not that the evidence behind it was right. (This is a course rule; module 100.P shows the format.)
Not sure? "I don't know yet" is an honest answer, such as for how much usable text the company has. Write it down, name who will find out, and let the decision wait for its owner.
Words to know
next-token prediction, autoregressive, cross-entropy loss, validation loss, task score, parameters (N), training tokens (D), compute (C ≈ 6ND), scaling law, compute-optimal, undertrained, epoch, inference cost, tokenizer, subword, byte-pair encoding (BPE), tokenization premium.
Further learning (optional, not sources)
These free university materials go deeper. They are pointers only; nothing in this module is copied from them, and we did not find a licence for any of them.
- Stanford University, CS336: Language Modeling from Scratch (Spring 2025): public schedule and archive, including tokenizer and scaling-law assignments; some files were not reachable when checked on 18 November 2026.
- Carnegie Mellon University, 11-711 Advanced Natural Language Processing (Spring 2025)
- University of Oxford, Deep Natural Language Processing (2017 lectures)