Try each exercise first, then open its model answer.
How training scales: loss, compute and tokens · Workbook
One exercise per objective. Each works in any gear: you need no code and no equations. Big Widget Company and its numbers are made up for teaching.
Exercise 1 (400.1-a): Two numbers, two claims
Big Widget's quality team sends you this note about two checkpoints of its in-house model:
Checkpoint A: validation loss on held-out general text is lower than B's. Checkpoint B: on a test set of the plant's own defect reports, engineers rated its summaries higher than A's. Proposal: "A has the lower loss, so A is the better model. Ship A for defect-report summaries."
- In one sentence each, write the claim the loss number supports and the claim the task score supports.
- Say what the loss measures at each position in the text, in plain words.
- Give two reasons from the lesson why a lower loss does not settle which checkpoint to ship.
- Rewrite the proposal so it rests on the right evidence.
Model answer
- The lower loss supports: "A predicts the next piece of held-out text of that kind better." The higher task score supports: "B writes better defect-report summaries, on this test set, as these engineers judged them."
- At each position the model gives a chance to every piece of text that could come next. The loss looks only at the chance it gave the piece that really came next: a high chance gives a small loss, a low chance a large one. It is averaged over a very large number of positions.
- Any two of these:
- Every position gets the same weight in the average. The GPT-3 authors wrote that their objective "weights every token equally and lacks a notion of what is most important to predict". Same weight is not same penalty: a torque value the model gave a low chance still adds a large penalty. But it gets no extra weight over "the", and averaged over many thousands of positions a few important tokens can be drowned out.
- Lower loss has gone with gains on many tasks, but the GPT-3 authors said "many (though not all)".
- Kaplan's team left open whether a better loss turns into better results on the tasks people care about.
- The loss belongs to a particular kind of text. On a different kind of text a model does worse by a roughly steady amount.
- A loss can be flattered if test text leaked into the training text; the Chinchilla authors warned that leakage "may artificially enhance the results".
- For example: "B scores higher on our defect-report test set, as our engineers judged it, so B is the candidate for defect-report summaries. A's lower loss tells us it predicts general text better; that is a different question." A strong answer also asks "out of how many?": the note gives no count of defect reports or engineers, and a real report must. It notes the task score covers only this test set and these judges, that module 400.6 covers making a task score hold up, and that the quality lead, not either number, makes the call.
Exercise 2 (400.1-b): Reallocate the budget, then test the assumptions
Big Widget's research team has a fixed training budget for a model trained on the company's service and engineering text. Their first plan:
A large model that reads roughly one training token for each parameter, like Gopher's 300 billion tokens for 280 billion parameters. Reason given: "The biggest model we can afford is always best."
Two more facts about Big Widget:
- The model will draft answers for every service desk, in every region, all day.
- The company has only so much engineering text. Reaching a much higher token count would mean reading the same text several times.
- Under the Chinchilla study's assumptions, should the team move towards more parameters or more training tokens for the same budget? Say what the study found, and the test it ran at full scale.
- Name the study's conditions you would check before following that answer.
- For each of the two Big Widget facts, say whether it falls inside the study's conditions, and what the team should do about it.
- Write one sentence to answer the "biggest is always best" reason.
Model answer
- Towards more tokens: a smaller model trained on more tokens, same budget. Training cost grows with model size times tokens read, so on a fixed budget a smaller model can read more. The Chinchilla study (Hoffmann and colleagues, 2022) trained over 400 models and found that model size and training tokens "should be scaled equally": double the model, double the tokens. It found the large models of the day "significantly undertrained". At full scale, Chinchilla (70 billion parameters, 1.4 trillion tokens) used the same compute as Gopher (280 billion parameters, 300 billion tokens) and outperformed it, and several other large models, on a wide range of tasks. The team's plan looks like Gopher's, so it looks undertrained.
- The study's conditions:
- It minimised loss for a fixed training budget; serving cost was not its question.
- Its scaling runs were all "trained on less than an epoch of data": each used fewer tokens than the whole dataset held. That does not mean no text was repeated (Chinchilla's own run went over Wikipedia about 3.40 times), but many passes over a small set of text was not tested.
- It was tested at certain scales: fitted runs up to just over 16 billion parameters, and only two comparable large runs. The authors warned of "significant uncertainty extrapolating out many orders of magnitude".
- It measured loss on its own dataset, not your task score.
- All-day service desks: outside the study's question. A model served this much makes running cost part of the decision. The study notes that Chinchilla's smaller size cuts inference cost, but it did not set out to weigh serving cost. The team should work out the running cost of each option with its own numbers before choosing.
- Limited text, reread several times: outside what the scaling fits tested. They covered runs that used less than one pass of their data, and the authors left "the multiple epoch regime" to future work. The team cannot take the study's answer as tested for its case. It does not know yet how much usable text it has. "I don't know yet" is an honest answer: it writes that down, and a named data engineer counts the text before the plan goes further.
- All-day service desks: outside the study's question. A model served this much makes running cost part of the decision. The study notes that Chinchilla's smaller size cuts inference cost, but it did not set out to weigh serving cost. The team should work out the running cost of each option with its own numbers before choosing.
- For example: "On a fixed training budget the Chinchilla study found the opposite of 'biggest is best', and even its answer holds only under conditions we have to check." A strong answer adds that two scaling studies already disagree: Kaplan's 2020 preprint put most extra compute into model size, so a scaling law is evidence with conditions, not a universal rule. The study is evidence for the decision; the research lead owns the choice of plan.
Exercise 3 (400.1-c): Count the cost of the chopping
Big Widget's service team writes one service instruction in English and in the languages of its plants abroad. A colleague says: "It's the same instruction, so it costs the same and the model reads it the same way in every language. And our part numbers, like BWX-40417-QR, are just words to the model."
- Explain in three or four steps how a byte-pair encoding tokenizer learns its vocabulary.
- Use those steps to explain why the same instruction can take very different numbers of tokens in different languages. Give one measured example from the lesson.
- Name three effects a higher token count can have on the service team, and say what each one depends on.
- Say what may happen to the part number, what you would need to do to find out, and what problem that could cause.
- Write the checks the team should run before it sets prices, limits or context budgets. Say who decides, and what happens to drafts if a part number comes back changed.
Model answer
- Start with single characters as the vocabulary. Count every pair of neighbouring symbols in a training corpus and merge the most frequent pair into a new symbol. Repeat; the number of merges is the algorithm's only setting. Frequent strings, even whole words, end up as one symbol; rare words stay split into subword pieces.
- Merges follow what was frequent in the tokenizer's training text, so its vocabulary reflects that text. Text in a language rare there is split into more pieces. The gap starts at the tokenizer, before the model sees anything. Examples (any one): for the tokenizer used by ChatGPT and GPT-4 in 2023, the same sentences took about 1.6 times as many tokens in Italian as in English, 3 times in Arabic and up to 15 times in Shan (Petrov and colleagues, 2023); the Bulgarian word for "why" took 6 tokens for 4 letters, while the GPT-2 tokenizer had one token for "BuyableInstoreAndOnline".
Cost depends on how you are billed. Services charge "per token or Unicode character". With the same price per token for every language, more tokens for the same text means a higher charge; the authors found users of some languages "paying at least 2.5 times more for the same task as users of English". With a price per character, token count alone does not set the price, but that has gaps of its own: the same text takes more characters in some languages than in others.
- Context is fixed in tokens, so in a token-hungry language less of a manual fits.
- Latency has to be measured. More tokens can take longer to process, but how much longer depends on the model and how it is served; it is not a ratio you can read off the token count.
A strong answer may add that quality can suffer: GPT-3's authors suggested, as a possible cause and not a proven one, that its weak English-to-Romanian translation came from a tokenizer built for an almost entirely English training dataset.
- A part number is not a common string on the public web, so a tokenizer built from public text may well split it into several tokens, as BPE does with rare words. We have not run it through any tokenizer, so we do not know the split: count its tokens with the tokenizer the team will deploy. The problem to test for: one changed character makes it a different part, so the team must check that the model gives back the exact string.
Count tokens for the same instruction in every supported language, with the tokenizer the model actually uses.
- Check how the provider bills (per token or per character) before setting prices or limits.
- Measure latency on the team's own setup.
- Test that part numbers come back out of the model exactly as they went in.
The service lead owns the per-language prices and limits, and decides from these results. The answer can be "only this far": for example, English only until the other languages are measured. If a part number comes back changed, drafts that quote part numbers wait for a person to check them against the parts list. Asking the same model whether it copied the number right is not a check.