
How it learned · Technical gear
You will be able to: describe training as optimising a model's parameters to predict its training data better, at very large scale, followed by tuning with human feedback; and explain why a model's output reflects its training distribution, including its gaps, its biases and its knowledge cutoff.
Facts about particular models and studies are dated in the text. This module is as of November 2026. The primary papers are listed in sources.yml; read them for the full methods.
1. Pretraining: next-token prediction
Figure data as text
From prediction to feedback
The two stages in order. Steps 2 to 4 are the tuning stage, shown as the InstructGPT example (s3).
- Pretrain — adjust the model to predict training text better. [s2, s3]
- Show — people show it good answers; train on the examples. [s3]
- Rank — people rank its answers; train a reward model on those preferences. [s3]
- Practise — the model practises; adjust it toward the answers scored higher. [s3]
Objective: 100.2-a. Text equivalent of figures/prediction-to-feedback.svg.
A large language model is a parameterised probability distribution over text. GPT-3 is described as "an autoregressive language model with 175 billion parameters".1 Autoregressive means the probability of a token sequence $x_1, \dots, x_T$ is factorised left to right,
p_\theta(x_1,\dots,x_T) = \prod_{t=1}^{T} p_\theta(x_t \mid x_{<t}),
and pretraining adjusts the parameters $\theta$ by gradient-based optimisation to minimise the cross-entropy loss on the training corpus,
\mathcal{L}(\theta) = -\frac{1}{T}\sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t}).
Kaplan et al. study exactly this quantity, the cross-entropy loss of autoregressive Transformer language models.2 Ouyang et al. summarise the objective as "predicting the next token on a webpage from the internet".3 No answer is written into $\theta$; whatever the model "knows" is whatever lowered $\mathcal{L}$ on its training data.
Scale
- GPT-3 (2020): 175 billion parameters; all model sizes trained for a total of 300 billion tokens. Its filtered Common Crawl component was 570 GB, roughly 400 billion byte-pair-encoded tokens.1
- PaLM (2022): 540 billion parameters, a dense autoregressive Transformer trained on 780 billion tokens, using 6,144 TPU v4 chips.4
Scaling laws
Kaplan et al. (2020, preprint) found that test loss "scales as a power-law with model size, dataset size, and the amount of compute used for training", with some trends spanning more than seven orders of magnitude.2 In the model-size case the fitted form is
\mathcal{L}(N) \approx \left(\frac{N_c}{N}\right)^{\alpha_N},
with $N$ the number of non-embedding parameters and $N_c$, $\alpha_N$ fitted constants; analogous forms hold for dataset size $D$ and compute $C$.2 Hoffmann et al. (2022, peer-reviewed) revisited the allocation question and found that for compute-optimal training "the model size and the number of training tokens should be scaled equally". Chinchilla, with 70B parameters and 4× more data, uses the same compute budget as the 280B-parameter Gopher and outperforms it.5 Prediction quality is a measured, smooth function of scale; it is not a list of answers someone added.
2. Tuning: supervised fine-tuning and RLHF
Pretraining optimises next-token likelihood, which Ouyang et al. call different from the objective "follow the user's instructions helpfully and safely".3 InstructGPT closes that gap with three stages:3
- Supervised fine-tuning (SFT). About 40 contractors, selected by a screening test, wrote demonstrations of the desired behaviour on (mostly English) prompts; GPT-3 is fine-tuned on them.
Reward modelling (RM). Labellers rank several model outputs per prompt. A 6B-parameter reward model $r_\theta(x, y)$ is trained on the pairwise comparisons with
\text{loss}(\theta) = -\frac{1}{\binom{K}{2}}\,\mathbb{E}_{(x,y_w,y_l)\sim D}\big[\log \sigma\big(r_\theta(x,y_w) - r_\theta(x,y_l)\big)\big],where $y_w$ is the preferred completion of the pair and $K$ the number of ranked outputs.
- Reinforcement learning (PPO). The SFT policy is optimised against the reward model with a per-token KL penalty from the SFT model; the "PPO-ptx" variant also mixes in pretraining gradients:
\text{objective}(\phi) = \mathbb{E}_{(x,y)\sim D_{\pi^{RL}_\phi}}\Big[r_\theta(x,y) - \beta \log\frac{\pi^{RL}_\phi(y\mid x)}{\pi^{SFT}(y\mid x)}\Big] + \gamma\, \mathbb{E}_{x\sim D_{\text{pretrain}}}\big[\log \pi^{RL}_\phi(x)\big].
Result: labellers preferred outputs from the 1.3B InstructGPT model to those of the 175B GPT-3, "despite having 100x fewer parameters". The authors also report that InstructGPT "still makes simple mistakes": it can fail to follow instructions, make up facts, give long hedging answers, or miss false premises.3 RLHF changes behaviour; it does not add a store of verified answers.
3. Beyond language: GraphCast
Learning from examples is not specific to text. GraphCast (Lam et al.) is a graph-neural-network weather model. The figures below are from the arXiv preprint, version 2 (4 August 2023); the version published in Science in 2023 reports a different evaluation with different figures. The model developed in the preprint was trained on 39 years (1979–2017) of historical data from ECMWF's ERA5 reanalysis archive.6 It predicts hundreds of weather variables over 10 days at 0.25° resolution globally in under one minute on a single Google Cloud TPU v4 device, and "significantly outperforms the most accurate operational deterministic systems on 90% of 1380 verification targets", with HRES as the comparator.6
GraphCast is trained directly on reanalysis data, not by next-token prediction on text, and involves no human-feedback tuning. It illustrates objective 100.2-a's core idea: parameters fitted to examples rather than hand-written rules. It illustrates 100.2-b with a distinction. The parameters are fitted to the training period, while each forecast is conditioned on new input, "the two most recent states of Earth's weather". The authors trained variants on data ending in 2017, 2018, 2019 and 2020 and report that performance can be improved "by re-training on more recent data". Its skill depends on its training data, including how recent that data is.6
4. The output reflects the training distribution
A model trained to minimise $\mathcal{L}$ on a corpus approximates that corpus's distribution. Four consequences follow.
Curation is a design choice. Corpora are selected and reweighted, not sampled from "all text". The Pile (Gao et al., 2020, preprint) is an 825 GiB English corpus of 22 sub-datasets; following Brown et al., its builders "increase the weights of higher quality components", with some datasets seen up to three times (epochs).7 GPT-3's datasets were likewise "not sampled in proportion to their size".1 Mixture weights shape what the model is good at.
Gaps and performance disparities. Sparse regions of the training distribution are modelled poorly. NIST AI 600-1 lists "Harmful Bias or Homogenization", including "performance disparities between sub-groups or languages, possibly due to non-representative training data".8
Bias. Brown et al. state that GPT-3 "retains the biases of the data it has been trained on" and that such biases "may lead models to generate stereotyped or prejudiced content"; Section 6.2 analyses gender, race and religion.1 PaLM's co-occurrence analysis "demonstrates the potential for the model to falsely affirm stereotypes, for instance, associating Muslims with terrorism, extremism, and violence", consistent across model scales.4 RLHF did not fix this: "InstructGPT does not significantly improve over GPT-3 on the Winogender and CrowS-Pairs datasets".3 The tuning stage also imports its own distribution: the authors say the model is aligned to "the stated preferences of a specific group of people (mostly our labelers and researchers)", on mostly English prompts.3
Knowledge cutoff. Parameters encode only the data seen in training. GPT-3's Common Crawl component came from "41 shards of monthly CommonCrawl covering 2016 to 2019"; GPT-3 also trained on other corpora, so this dates one component, not a whole-model cutoff.1 A whole-model cutoff for GPT-3 is not checked in this module and is not stated. The GraphCast model developed in the preprint was trained on data ending in 2017; its authors also trained variants through 2020.6 Events after the collection date are absent from the learned distribution. That is distinct from inference: a fixed model can condition on information supplied in context, and Brown et al. evaluate GPT-3 on tasks given as text, "without any gradient updates or fine-tuning".1 If newer facts are not supplied, the model can still produce confident output that is wrong. NIST names confidently stated but false output "confabulation".8 NIST describes this risk in general, not specifically for events after a training cutoff. Retrieval, covered in module 100.6, supplies newer context at inference time; it does not change what pretraining learned.
The misconceptions this corrects: "Someone programmed in every answer" (100.2-a) and "The model is neutral and up to date because it is a machine" (100.2-b). A model is an optimised fit to a chosen, dated, reweighted corpus, then shifted toward a particular group's preferences.
5. At Big Widget Company
Big Widget Company is the course's made-up setting. Its forecasting and planning team uses a model trained on ten years of the company's widget sales.
- Good AI: they forecast established products, which are well inside the training distribution. They document the data range and its end date. The planning lead checks each forecast and makes the decision; the model's output is a draft input, not an approval.
- Bad AI: they forecast a new product with no comparable history, which is out of distribution, and treat the point estimate as fact.
The three yeses: the data questions. These are this module's data-focused questions. Module 100.9 covers the full gate: safe sharing, a named owner and a written action rule. Passing these questions does not by itself authorise an action.
- Get to Yes (before use): is the task inside the training distribution? Record the data range and end date. If that is unknown, record "unknown" and hold the decision for the planning lead.
- Stay at Yes (in operation): watch for drift, whether the task, the data or the world has moved since the cutoff. An initial yes does not cover later changes.
- Recover to Yes (after a failure): stop; the planning lead owns the error and fixes the effects, including for anyone affected; the team changes the rule so out-of-distribution cases are held for a person. Treat recovery as a claim that needs evidence from later forecasts, not as done because the rule changed.
Records and receipts. The data range, model version and checker belong in the team's controlled work record. Sales figures and customer details stay out of anything shared beyond it. Module 100.P covers keeping a receipt of the decision that holds no private content.
Further learning (optional, not sources)
Free university materials; they are pointers only, and nothing here is copied from them.
- Stanford University, CS324: Large Language Models
- MIT OpenCourseWare, 6.034 Artificial Intelligence (Fall 2010)
- University of Cambridge, David MacKay, Information Theory, Pattern Recognition, and Neural Networks (lecture 15: data modelling with neural networks)