Level 400: Engineering AI systems · Module 400.1

Two coworkers at a café table compare two stacks of plain paper, one tall and one short.
Test the job, not just the loss

How training scales: loss, compute and tokens · Standard gear

You will be able to: explain what next-token cross-entropy loss measures during pre-training, and what a lower loss does and does not say about a Big Widget task; read a published scaling study's conditions and decide whether, for the same compute, it points to more parameters or more training tokens, and name a condition under which that answer would not carry over; and explain how a subword tokenizer splits text, and spot the cost and quality gaps it creates across languages and part numbers.

Facts about particular models, tokenizers and studies are dated in the text. This module is as of November 2026. Big Widget Company is the made-up company used across this course; its plans and numbers here are made up for teaching. Where we did not check something, we say "not checked".

1. What the loss measures

This module is about one kind of model: the autoregressive text models that the studies below trained, which read text as subword tokens. It does not claim that every language model works this way.

An autoregressive model is trained on one job: given the text so far, predict the next token. The GPT-3 paper (2020) describes GPT-3 as "an autoregressive language model with 175 billion parameters": it produces text one token at a time, each one following the tokens before it.1 The Transformer design was introduced in 2017 for translation. Its decoder already worked this way, "consuming the previously generated symbols as additional input when generating the next", and it turned its output into "predicted next-token probabilities".2 By 2022, the Chinchilla authors were describing the large language models of the day as "large autoregressive transformers", citing that 2017 paper.3

So at every position the model gives a probability to every token it could produce next. The cross-entropy loss looks only at the probability the model gave to the token that actually came next. If that probability was high, the loss at that position is small. If it was low, the loss is large. The training loss is this penalty averaged over a very large number of positions. Kaplan and colleagues, in a 2020 study of how loss scales, measured it in nats (a unit of information) averaged over a 1,024-token context. They recorded it on a held-out test set and on other kinds of text.4

Two things follow from that definition.

2. What a lower loss does and does not tell you

What it does tell you. Across many tasks, lower loss has gone with better results. Plotting validation loss against training compute, the GPT-3 authors found a smooth power-law trend. They raised the worry that "these improvements in cross-entropy loss come only from modeling spurious details of our training corpus", and answered that the improvements "lead to consistent performance gains across a broad spectrum of natural language tasks".1 So a falling loss is real evidence that the model is learning something general.

What it does not tell you. It does not tell you how well the model does your task.

At Big Widget. The plant's quality team has two checkpoints of an in-house model. Checkpoint A has the lower validation loss on held-out general text. Checkpoint B scores higher on the job they care about: a test set of the plant's own defect reports, where engineers judged each model's summaries. Each number supports a different claim:

Before you believe either number, ask "out of how many?" How many defect reports were in the test set, and how many engineers judged them? This made-up example does not give those counts, and a real report must. If the decision is which checkpoint summarises defect reports, the task score is the evidence that answers it. The loss is not; it measures something else. Neither number decides on its own: the quality lead reads the evidence and makes the call. (Module 400.6 covers how to make a task score hold up, with error bars and a test set the model has not seen.)

The misconception this corrects: "A lower training loss means the model is better at the job I need." Lower loss means better next-token prediction on text like the test text, every position weighted the same. Whether that helps your job is a separate question, answered by testing the job.

3. Scaling laws: loss, size, data and compute

Compute is about 6 times parameters times tokens. Kaplan's fit puts most extra compute into model size (about C to the 0.73); Chinchilla's puts it equally into size and tokens (about C to the 0.5). On the same compute, Gopher had 280 billion parameters and 300 billion tokens; Chinchilla had 70 billion parameters and 1.4 trillion tokens. The Chinchilla fit holds only for its own conditions: training compute, not serving; less than an epoch of data; and fitted runs up to just over 16 billion parameters.
C ≈ 6ND is a rule of thumb both studies use (s2, s3). Kaplan's fit has model size growing as about C^0.73 (s2); the Chinchilla study's as about C^0.5, tested with Gopher and Chinchilla at the same compute (s3). A scaling law tells you what was true under its experiments' conditions; the Chinchilla authors list counting training compute only, less than an epoch of data and fitted runs up to just over 16 billion parameters (s3).
Figure data as text

Same compute, two answers

Study or ruleWhat it shows
RULE OF THUMB [s2, s3]C ≈ 6ND. Training compute is about 6 operations per parameter (N) per training token (D).
KAPLAN 2020 [s2]Size grows as about C^0.73. Most of any extra compute goes to a bigger model.
CHINCHILLA 2022 [s3]Size grows as about C^0.5. Not C^0.73: model size and tokens grow in step.
SAME COMPUTE [s3]Gopher vs Chinchilla. Gopher: 280 billion parameters, 300 billion tokens. Chinchilla: 70 billion parameters, 1.4 trillion tokens.
CHINCHILLA'S CONDITIONS [s3]Chinchilla's fit holds for its runs. Training compute, not serving; less than an epoch of data; fitted runs up to just over 16 billion parameters.

Three things set how far training can push the loss down: the number of parameters in the model (N), the number of training tokens (D) and the compute spent (C). Kaplan's team found that "The loss scales as a power-law with model size, dataset size, and the amount of compute used for training": a smooth, predictable curve.4 They used a rule of thumb for compute: training costs about 6 floating-point operations per parameter per training token, so C ≈ 6ND.4 The Chinchilla study used the same approximation.3

That rule of thumb shows the trade-off this module is about. For a fixed compute budget, a bigger model means fewer training tokens, and a smaller model means more. Which is better?

Kaplan's team (2020, a preprint) concluded that "Larger models are significantly more sample-efficient", so that compute-efficient training means "training very large models on a relatively modest amount of data and stopping significantly before convergence".4 By their fit, as compute grows, model size should grow roughly as C^0.73: most of any extra compute goes to a bigger model.4 They also said the trends cannot go on for ever: "performance must flatten out eventually before reaching zero loss".4

4. The Chinchilla result: same compute, smaller model, more tokens

In 2022 Jordan Hoffmann and colleagues at DeepMind revisited the question, in a paper later published at NeurIPS 2022.3 They trained over 400 language models, from 70 million to over 16 billion parameters, and found that "the model size and the number of training tokens should be scaled equally": "for every doubling of model size the number of training tokens should also be doubled".3 Three different ways of fitting the data gave similar answers: model size should grow roughly as C^0.5, not C^0.73.3 By that analysis, the large models of the day were "significantly undertrained".3

They tested the prediction at full scale. Gopher, an earlier DeepMind model, had 280 billion parameters trained on 300 billion tokens. Chinchilla, with the same compute budget, had 70 billion parameters trained on 1.4 trillion tokens: four times smaller, with about four times more data, as the authors round it.3 Chinchilla "uniformly and significantly" outperformed Gopher, GPT-3, Jurassic-1 and Megatron-Turing NLG on a large range of downstream evaluation tasks.3

So for the same compute, the study points away from "biggest model you can afford" and towards a smaller model trained on more tokens, when the starting plan is undertrained for its size.

5. When the result might not carry over

A scaling law is a fit to a set of experiments. It tells you what was true under those experiments' conditions. The Chinchilla authors list several of their own limits, and two scaling studies already disagree. Treat a scaling law as evidence with conditions, not a universal rule.

At Big Widget. The research team has a fixed compute budget for a model trained on the company's service and engineering text. Their first plan uses roughly as many training tokens as parameters, much like Gopher's 300 billion tokens for 280 billion parameters, which the Chinchilla study found undertrained.3 Under the study's conditions (no small corpus repeated many times, a budget in its tested range, training cost only), the reallocation is: fewer parameters, more tokens, same compute. Then the team asks whether the conditions hold:

The study is evidence for the decision, not the decision. The research lead owns the choice of plan.

The misconception this corrects: "A scaling law is a universal rule, and the biggest model you can afford is always the best choice." The Chinchilla study found the opposite of "biggest is best" for a fixed training budget, and its answer itself depends on conditions you must check.

6. Tokens: how a subword tokenizer splits text

Before any text reaches a model like these, a tokenizer cuts it into tokens from a fixed vocabulary, and the model works on those tokens. Differences between languages start "at the tokenization stage, well before the language model sees any data at all".5 A token can be a whole word, a piece of a word or a single character, depending on what the tokenizer learned.6 The GPT-3 authors note that their encoding "operates on significant fractions of a word (on average ∼0.7 words per token)". They give this as part of why tasks that need letter-by-letter work are hard for it: the model has to pull apart the pieces to get at the letters.1

"Subword tokenization is currently the preferred approach for state of the art language models", as Petrov and colleagues wrote in 2023.5 One common method is byte-pair encoding (BPE), adapted for splitting words by Sennrich, Haddow and Birch (2016).6 It learns its vocabulary from a training corpus:

  1. Start with single characters as the vocabulary.
  2. Count every pair of neighbouring symbols in the corpus, and merge the most frequent pair into a new symbol.
  3. Repeat. The final vocabulary is the starting characters plus the number of merges, and the number of merges is "the only hyperparameter of the algorithm".6

Because merges follow frequency, "Frequent character n-grams (or whole words) are eventually merged into a single symbol", while rare and unknown words are encoded "as sequences of subword units".6 The Transformer paper's English-German experiments already used byte-pair encoding, with a shared vocabulary of about 37,000 tokens.2

The key consequence: the tokenizer's vocabulary reflects the text it was built from. Petrov and colleagues (2023) found that the GPT-2 tokenizer has a token for the odd string "BuyableInstoreAndOnline", while a common Arabic word is split into one token per letter and the Bulgarian word for "why" (защо) takes 6 tokens for its 4 letters. As they put it: "If one uses non-natural inputs, log files, or specialist forums, the tokenizer vocabulary would reflect this."5

7. The gaps a tokenizer creates

Across languages. Petrov's team measured how many tokens different tokenizers need for the same 2,000 sentences translated into 200 languages (the FLORES-200 set).5 For the tokenizer used by ChatGPT and GPT-4 at the time (2023), the same text took about 1.6 times as many tokens in Italian as in English, 2.6 times in Bulgarian and 3 times in Arabic; for Shan, a language of Myanmar, the gap reached 15 times.5 They traced three effects. Each depends on how the service runs, so keep them apart.5

Quality can suffer too. GPT-3's English-to-Romanian translation was "a noticeable outlier", over 10 BLEU worse than earlier unsupervised translation systems. Its authors suggested this "could be a weakness due to reusing the byte-level BPE tokenizer of GPT-2 which was developed for an almost entirely English training dataset".1 They offered it as a possible cause, not a proven one.

Part numbers and codes. The same mechanism applies to anything rare in the tokenizer's training text. A Big Widget part number or fault code (made up here, such as BWX-40417-QR) is not a common string on the public web. A tokenizer built from public text may well split it into several tokens, as BPE does with rare words.6 We have not run it through any tokenizer, so we do not know the split. Treat this as a risk to test, not a fact: count its tokens with the tokenizer you will deploy, and test that the model gives back the exact string, because one changed character makes it a different part.

At Big Widget. The service team writes the same instruction in English and in the languages of its plants abroad. Run through one tokenizer, the versions can come out with very different token counts. The team counts tokens in every language it supports, checks how its provider bills, and measures latency on its own setup before it sets prices, limits or context budgets. It also tests that part numbers survive a round trip through the model unchanged.

The misconception this corrects: "The model sees whole words and letters the way people do." A model like these sees tokens from a vocabulary learned from someone else's text. A token may be a word, a piece of one or a single character, and that choice has costs.

8. The three yeses for a training or tokenizer decision

None of the numbers in this module is permission to act. A loss, a scaling fit or a token count is evidence. The model does not decide, and neither does the metric: a named person does. A yes is not a blanket licence either. It covers one checkpoint, one plan or one language set, for one job. At Big Widget the owners are the quality lead for checkpoints, the research lead for the training plan and the service lead for per-language prices, limits and part-number use.

Keep the decision, not the evidence. Keep the evidence (test sets, scores, token counts, who decided and when) in the team's own controlled work record. Do not paste defect reports, customer text or manuals into anything shared beyond that record. A receipt of a gate decision holds no private content: it carries the decision, who made it and references to the evidence, never prompts, model outputs, customer text or personal details. It shows that a decision was recorded, not that the evidence behind it was right. (This is a course rule; module 100.P shows the format.)

Not sure? "I don't know yet" is an honest answer, such as for how much usable text the company has. Write it down, name who will find out, and let the decision wait for its owner.

Words to know

next-token prediction, autoregressive, cross-entropy loss, validation loss, task score, parameters (N), training tokens (D), compute (C ≈ 6ND), scaling law, compute-optimal, undertrained, epoch, inference cost, tokenizer, subword, byte-pair encoding (BPE), tokenization premium.

Further learning (optional, not sources)

These free university materials go deeper. They are pointers only; nothing in this module is copied from them, and we did not find a licence for any of them.

Sources

  1. s6 Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al. (2020). Language Models are Few-Shot Learners. Paper. Checked 2026-11-19 by source-checker role, separate Claude session (Claude Opus 5.5), round-3 re-check 2026-11-19; see checks/source-check.md. ↩ ↩ ↩ ↩ ↩ ↩
  2. s1 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin (2017). Attention Is All You Need. Paper. Checked 2026-11-19 by source-checker role, separate Claude session (Claude Opus 5.5), round-2 re-check 2026-11-19; see checks/source-check.md. ↩ ↩
  3. s3 Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, et al. (2022). Training Compute-Optimal Large Language Models. Paper. Checked 2026-11-19 by source-checker role, separate Claude session (Claude Opus 5.5), round-3 re-check 2026-11-19; see checks/source-check.md. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  4. s2 Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, et al. (2020). Scaling Laws for Neural Language Models. Paper. Checked 2026-11-19 by source-checker role, separate Claude session (Claude Opus 5.5), round-2 re-check 2026-11-19; see checks/source-check.md. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  5. s5 Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi (2023). Language Model Tokenizers Introduce Unfairness Between Languages. Paper. Checked 2026-11-19 by source-checker role, separate Claude session (Claude Opus 5.5), round-3 re-check 2026-11-19; see checks/source-check.md. ↩ ↩ ↩ ↩ ↩ ↩ ↩
  6. s4 Rico Sennrich, Barry Haddow, and Alexandra Birch (2016). Neural Machine Translation of Rare Words with Subword Units. Paper. Checked 2026-11-19 by source-checker role, separate Claude session (Claude Opus 5.5), round-2 re-check 2026-11-19; see checks/source-check.md. ↩ ↩ ↩ ↩ ↩

Next: the quiz