Level 100: How AI works: AI at work and in life · Module 100.2

Three coworkers sort stacks of cream sample cards on a wooden café table.
Learn from examples.

How it learned · Technical gear

You will be able to: describe training as optimising a model's parameters to predict its training data better, at very large scale, followed by tuning with human feedback; and explain why a model's output reflects its training distribution, including its gaps, its biases and its knowledge cutoff.

Facts about particular models and studies are dated in the text. This module is as of November 2026. The primary papers are listed in sources.yml; read them for the full methods.

1. Pretraining: next-token prediction

Pretraining improves prediction; InstructGPT tuning then uses human examples and rankings to adjust the assistant toward preferred answers.
Adjust the model to predict training text better (s2, s3). InstructGPT then trained on human examples, learned rankings with a reward model, and adjusted the assistant against those preferences (s3).
Figure data as text

From prediction to feedback

The two stages in order. Steps 2 to 4 are the tuning stage, shown as the InstructGPT example (s3).

  1. Pretrain — adjust the model to predict training text better. [s2, s3]
  2. Show — people show it good answers; train on the examples. [s3]
  3. Rank — people rank its answers; train a reward model on those preferences. [s3]
  4. Practise — the model practises; adjust it toward the answers scored higher. [s3]

Objective: 100.2-a. Text equivalent of figures/prediction-to-feedback.svg.

A large language model is a parameterised probability distribution over text. GPT-3 is described as "an autoregressive language model with 175 billion parameters".1 Autoregressive means the probability of a token sequence $x_1, \dots, x_T$ is factorised left to right,

p_\theta(x_1,\dots,x_T) = \prod_{t=1}^{T} p_\theta(x_t \mid x_{<t}),

and pretraining adjusts the parameters $\theta$ by gradient-based optimisation to minimise the cross-entropy loss on the training corpus,

\mathcal{L}(\theta) = -\frac{1}{T}\sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t}).

Kaplan et al. study exactly this quantity, the cross-entropy loss of autoregressive Transformer language models.2 Ouyang et al. summarise the objective as "predicting the next token on a webpage from the internet".3 No answer is written into $\theta$; whatever the model "knows" is whatever lowered $\mathcal{L}$ on its training data.

Scale

Scaling laws

Kaplan et al. (2020, preprint) found that test loss "scales as a power-law with model size, dataset size, and the amount of compute used for training", with some trends spanning more than seven orders of magnitude.2 In the model-size case the fitted form is

\mathcal{L}(N) \approx \left(\frac{N_c}{N}\right)^{\alpha_N},

with $N$ the number of non-embedding parameters and $N_c$, $\alpha_N$ fitted constants; analogous forms hold for dataset size $D$ and compute $C$.2 Hoffmann et al. (2022, peer-reviewed) revisited the allocation question and found that for compute-optimal training "the model size and the number of training tokens should be scaled equally". Chinchilla, with 70B parameters and 4× more data, uses the same compute budget as the 280B-parameter Gopher and outperforms it.5 Prediction quality is a measured, smooth function of scale; it is not a list of answers someone added.

2. Tuning: supervised fine-tuning and RLHF

Pretraining optimises next-token likelihood, which Ouyang et al. call different from the objective "follow the user's instructions helpfully and safely".3 InstructGPT closes that gap with three stages:3

  1. Supervised fine-tuning (SFT). About 40 contractors, selected by a screening test, wrote demonstrations of the desired behaviour on (mostly English) prompts; GPT-3 is fine-tuned on them.
  2. Reward modelling (RM). Labellers rank several model outputs per prompt. A 6B-parameter reward model $r_\theta(x, y)$ is trained on the pairwise comparisons with

    \text{loss}(\theta) = -\frac{1}{\binom{K}{2}}\,\mathbb{E}_{(x,y_w,y_l)\sim D}\big[\log \sigma\big(r_\theta(x,y_w) - r_\theta(x,y_l)\big)\big],

    where $y_w$ is the preferred completion of the pair and $K$ the number of ranked outputs.

  3. Reinforcement learning (PPO). The SFT policy is optimised against the reward model with a per-token KL penalty from the SFT model; the "PPO-ptx" variant also mixes in pretraining gradients:
    \text{objective}(\phi) = \mathbb{E}_{(x,y)\sim D_{\pi^{RL}_\phi}}\Big[r_\theta(x,y) - \beta \log\frac{\pi^{RL}_\phi(y\mid x)}{\pi^{SFT}(y\mid x)}\Big] + \gamma\, \mathbb{E}_{x\sim D_{\text{pretrain}}}\big[\log \pi^{RL}_\phi(x)\big].

Result: labellers preferred outputs from the 1.3B InstructGPT model to those of the 175B GPT-3, "despite having 100x fewer parameters". The authors also report that InstructGPT "still makes simple mistakes": it can fail to follow instructions, make up facts, give long hedging answers, or miss false premises.3 RLHF changes behaviour; it does not add a store of verified answers.

3. Beyond language: GraphCast

Learning from examples is not specific to text. GraphCast (Lam et al.) is a graph-neural-network weather model. The figures below are from the arXiv preprint, version 2 (4 August 2023); the version published in Science in 2023 reports a different evaluation with different figures. The model developed in the preprint was trained on 39 years (1979–2017) of historical data from ECMWF's ERA5 reanalysis archive.6 It predicts hundreds of weather variables over 10 days at 0.25° resolution globally in under one minute on a single Google Cloud TPU v4 device, and "significantly outperforms the most accurate operational deterministic systems on 90% of 1380 verification targets", with HRES as the comparator.6

GraphCast is trained directly on reanalysis data, not by next-token prediction on text, and involves no human-feedback tuning. It illustrates objective 100.2-a's core idea: parameters fitted to examples rather than hand-written rules. It illustrates 100.2-b with a distinction. The parameters are fitted to the training period, while each forecast is conditioned on new input, "the two most recent states of Earth's weather". The authors trained variants on data ending in 2017, 2018, 2019 and 2020 and report that performance can be improved "by re-training on more recent data". Its skill depends on its training data, including how recent that data is.6

4. The output reflects the training distribution

A model trained to minimise $\mathcal{L}$ on a corpus approximates that corpus's distribution. Four consequences follow.

Curation is a design choice. Corpora are selected and reweighted, not sampled from "all text". The Pile (Gao et al., 2020, preprint) is an 825 GiB English corpus of 22 sub-datasets; following Brown et al., its builders "increase the weights of higher quality components", with some datasets seen up to three times (epochs).7 GPT-3's datasets were likewise "not sampled in proportion to their size".1 Mixture weights shape what the model is good at.

Gaps and performance disparities. Sparse regions of the training distribution are modelled poorly. NIST AI 600-1 lists "Harmful Bias or Homogenization", including "performance disparities between sub-groups or languages, possibly due to non-representative training data".8

Bias. Brown et al. state that GPT-3 "retains the biases of the data it has been trained on" and that such biases "may lead models to generate stereotyped or prejudiced content"; Section 6.2 analyses gender, race and religion.1 PaLM's co-occurrence analysis "demonstrates the potential for the model to falsely affirm stereotypes, for instance, associating Muslims with terrorism, extremism, and violence", consistent across model scales.4 RLHF did not fix this: "InstructGPT does not significantly improve over GPT-3 on the Winogender and CrowS-Pairs datasets".3 The tuning stage also imports its own distribution: the authors say the model is aligned to "the stated preferences of a specific group of people (mostly our labelers and researchers)", on mostly English prompts.3

Knowledge cutoff. Parameters encode only the data seen in training. GPT-3's Common Crawl component came from "41 shards of monthly CommonCrawl covering 2016 to 2019"; GPT-3 also trained on other corpora, so this dates one component, not a whole-model cutoff.1 A whole-model cutoff for GPT-3 is not checked in this module and is not stated. The GraphCast model developed in the preprint was trained on data ending in 2017; its authors also trained variants through 2020.6 Events after the collection date are absent from the learned distribution. That is distinct from inference: a fixed model can condition on information supplied in context, and Brown et al. evaluate GPT-3 on tasks given as text, "without any gradient updates or fine-tuning".1 If newer facts are not supplied, the model can still produce confident output that is wrong. NIST names confidently stated but false output "confabulation".8 NIST describes this risk in general, not specifically for events after a training cutoff. Retrieval, covered in module 100.6, supplies newer context at inference time; it does not change what pretraining learned.

The misconceptions this corrects: "Someone programmed in every answer" (100.2-a) and "The model is neutral and up to date because it is a machine" (100.2-b). A model is an optimised fit to a chosen, dated, reweighted corpus, then shifted toward a particular group's preferences.

5. At Big Widget Company

Big Widget Company is the course's made-up setting. Its forecasting and planning team uses a model trained on ten years of the company's widget sales.

The three yeses: the data questions. These are this module's data-focused questions. Module 100.9 covers the full gate: safe sharing, a named owner and a written action rule. Passing these questions does not by itself authorise an action.

Records and receipts. The data range, model version and checker belong in the team's controlled work record. Sales figures and customer details stay out of anything shared beyond it. Module 100.P covers keeping a receipt of the decision that holds no private content.

Further learning (optional, not sources)

Free university materials; they are pointers only, and nothing here is copied from them.

Sources

  1. s1 Tom B. Brown et al. (2020). Language Models are Few-Shot Learners. Paper. Checked 2026-11-17 by source-checker role, separate Claude session (Claude Opus 5.5), round 3 re-check 2026-11-17; see checks/source-check.md. ↩ ↩ ↩ ↩ ↩ ↩
  2. s2 Jared Kaplan et al. (2020). Scaling Laws for Neural Language Models. Paper. Checked 2026-11-16 by source-checker role, separate Claude session (Claude Opus 5.5), 2026-11-16; see checks/source-check.md. ↩ ↩ ↩
  3. s3 Long Ouyang et al. (2022). Training language models to follow instructions with human feedback. Paper. Checked 2026-11-16 by source-checker role, separate Claude session (Claude Opus 5.5), 2026-11-16; see checks/source-check.md. ↩ ↩ ↩ ↩ ↩ ↩
  4. s7 Aakanksha Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways. Paper. Checked 2026-11-16 by source-checker role, separate Claude session (Claude Opus 5.5), 2026-11-16; see checks/source-check.md. ↩ ↩
  5. s6 Jordan Hoffmann et al. (2022). Training Compute-Optimal Large Language Models. Paper. Checked 2026-11-17 by source-checker role, separate Claude session (Claude Opus 5.5), round 2 re-check 2026-11-17; see checks/source-check.md. ↩
  6. s8 Remi Lam et al. (2022). GraphCast: Learning skillful medium-range global weather forecasting. Paper. Checked 2026-11-17 by source-checker role, separate Claude session (Claude Opus 5.5), round 2 re-check 2026-11-17; see checks/source-check.md. ↩ ↩ ↩ ↩
  7. s4 Leo Gao et al. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. Paper. Checked 2026-11-16 by source-checker role, separate Claude session (Claude Opus 5.5), 2026-11-16; see checks/source-check.md. ↩
  8. s5 Chloe Autio et al. (National Institute of Standards and Technology) (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). Standard. Checked 2026-11-17 by source-checker role, separate Claude session (Claude Opus 5.5), round 2 re-check 2026-11-17; see checks/source-check.md. ↩ ↩

Next: the quiz