Level 100: How AI works: AI at work and in life · Module 100.2

Three coworkers sort stacks of cream sample cards on a wooden café table.
Learn from examples.

How it learned · Standard gear

You will be able to: describe training as adjusting a model so it predicts its training data better, at very large scale, followed by tuning with human feedback; and explain why a model's output reflects its training data, including its gaps, its unfair patterns and its knowledge cutoff.

Facts about particular models and studies are dated in the text. This module is as of November 2026.

1. Nobody typed in the answers

Pretraining improves prediction; InstructGPT tuning then uses human examples and rankings to adjust the assistant toward preferred answers.
Adjust the model to predict training text better (s2, s3). InstructGPT then trained on human examples, learned rankings with a reward model, and adjusted the assistant against those preferences (s3).
Figure data as text

From prediction to feedback

The two stages in order. Steps 2 to 4 are the tuning stage, shown as the InstructGPT example (s3).

  1. Pretrain — adjust the model to predict training text better. [s2, s3]
  2. Show — people show it good answers; train on the examples. [s3]
  3. Rank — people rank its answers; train a reward model on those preferences. [s3]
  4. Practise — the model practises; adjust it toward the answers scored higher. [s3]

Objective: 100.2-a. Text equivalent of figures/prediction-to-feedback.svg.

A common belief is that someone programmed every answer into an AI model. Nobody did. An assistant-style language model, like the ones in this section, is built in two big steps. First comes pretraining. Then comes tuning.

Pretraining. The model is shown a huge amount of text and given one job: predict what comes next. The researchers who built InstructGPT describe this standard job as "predicting the next token on a webpage from the internet".1 A token is a small piece of text, often a word or part of a word. GPT-3, described by its builders in 2020, is "an autoregressive language model": it is trained to predict text one piece at a time, each piece following the ones before.2

Training means adjusting the model, a tiny amount at a time, so that its predictions match the training text better. Researchers measure how far off the predictions are with a number called the loss: the lower the loss, the better the model predicts.3 Nobody writes the answers into the model. What it ends up "knowing" is whatever helped it predict its training text.

At very large scale. The scale is hard to picture:

Scale matters because prediction gets better in a measurable, predictable way as models grow. A 2020 study found that loss falls as a smooth "power law" as you add model size, data and computing power, with some trends holding over more than seven orders of magnitude (a range of ten million times).3 A 2022 study found that, for a fixed computing budget, the size of the model and the amount of training text "should be scaled equally". Its model Chinchilla, with 70 billion parameters and four times more data, beat the 280-billion-parameter Gopher on the same budget. Bigger is not automatically better: the data has to grow with the model.5

2. Then people tune it

A model that only predicts the next piece of internet text is not the same as a helpful assistant. The InstructGPT researchers put it this way: predicting the next token on a webpage is a different goal from "follow the user's instructions helpfully and safely".1 So the model is tuned with human feedback, in three steps:1

  1. People show it good answers. A team of about 40 hired contractors wrote example answers to requests, and the model was trained on those examples (supervised fine-tuning).
  2. People rank its answers. The labellers compared several of the model's answers and ranked them. A second model, the reward model, was trained to predict which answer the labellers would prefer.
  3. The model practises against those preferences. The model was then adjusted, using a method called reinforcement learning, to produce answers the reward model scores highly.

This is called reinforcement learning from human feedback (RLHF). It made a large difference: people preferred answers from a 1.3-billion-parameter tuned model to answers from the 175-billion-parameter GPT-3, "despite having 100x fewer parameters".1 It did not make the model perfect. The authors say InstructGPT "still makes simple mistakes", including failing to follow instructions and making up facts.1

The misconception this corrects: "Someone programmed in every answer." Nobody did. The model was adjusted in many small steps to predict text better, and then adjusted again to match what a group of people preferred.

3. Learning from examples is not only for chatbots

The same idea, learning from examples instead of being told the rules, works far beyond text. GraphCast, a weather model from Google DeepMind, is described in its builders' preprint (version 2, August 2023). The model developed in that paper was trained on 39 years (1979 to 2017) of historical weather data from the European Centre for Medium-Range Weather Forecasts (ECMWF).6 It predicts hundreds of weather variables for the next 10 days across the whole globe in under a minute on a single specialised chip. In the preprint's tests it significantly outperformed HRES, the most accurate operational deterministic forecasting system, on 90% of 1,380 test targets. These figures are from the preprint. The version published in Science in 2023 reports a different evaluation with different figures.6

GraphCast is not a chatbot. It did not learn by predicting text and was not tuned with human feedback, so it shares only this module's core idea: a model that learns from a large set of examples can become very good at a task. It also shows the second half: what it learned during training came from those 39 years of records. To make each forecast, it is given the two most recent states of the weather as its starting point.6

4. A model reflects the data it learned from

Because a model learns from examples, its answers reflect those examples: what was in them, what was missing from them, and when they stopped.

Someone chose the data. Training data is not "all of human knowledge". People pick what goes in and how much each part counts. The Pile, an 825 GiB collection of English text built in 2020 for training language models, was assembled from 22 chosen parts, and its builders deliberately increased the weight of the parts they judged higher quality.7 GPT-3's builders also mixed their sources on purpose: during training, their datasets were "not sampled in proportion" to their size.2

Gaps. What was common in the data, the model tends to handle well. What was rare or missing, it tends to handle badly. The US National Institute of Standards and Technology (NIST) lists "harmful bias or homogenization" as a risk of generative AI. It includes performance differences between groups of people or languages, "possibly due to non-representative training data".8

Unfair patterns. GPT-3's builders wrote that the model "retains the biases of the data it has been trained on", and that biases in training data "may lead models to generate stereotyped or prejudiced content". They tested it for bias related to gender, race and religion.2 PaLM's builders found "the potential for the model to falsely affirm stereotypes", for instance associating Muslims with terrorism, extremism and violence.4 Tuning with human feedback did not remove this: InstructGPT "does not significantly improve over GPT-3" on two standard bias tests.1

Whose preferences. Tuning adds the views of the people who give the feedback. The InstructGPT authors say their model is aligned to "the stated preferences of a specific group of people (mostly our labelers and researchers)", not to any broader idea of human values, and that the requests it learned from were mostly in English.1

Knowledge cutoff. A model only learned from data up to the date its data was collected. GPT-3's Common Crawl text, one of several sources it trained on, came from "41 shards of monthly CommonCrawl covering 2016 to 2019".2 That dates one part of GPT-3's training data. A cutoff date for the whole model is not checked in this module, so we do not state one. The GraphCast model in the 2023 preprint learned from records ending in 2017. Its builders also trained versions on data ending in 2018, 2019 and 2020, and found that re-training on more recent data improved its forecasts.6 Anything that happened after the data was gathered is not in what the model learned. That does not make a model helpless with new information: GPT-3 was tested on tasks given to it only as text in the request, "without any gradient updates or fine-tuning".2 But if nobody supplies the newer facts, a model can still sound sure when it is wrong. NIST calls confidently stated but false content "confabulation".8 Module 100.6 covers tools that hand a model newer documents before it answers.

The misconception this corrects: "The model is neutral and up to date because it is a machine." A model is a mirror of chosen data, collected up to a date, tuned to the preferences of particular people.

5. At Big Widget Company

Big Widget Company is the made-up company used across this course. Its forecasting and planning team uses a model trained on ten years of the company's widget sales.

The three yeses: the data questions. This module asks only the data questions. Module 100.9 teaches the full gate, including safe sharing, a named owner and a written rule for what may run. Good answers here are not permission to act on their own.

What to keep, and what to share. Keep the data range, its end date and who checked each forecast in the team's own work record. Do not paste sales figures or customer details into anything shared beyond that record. Module 100.P shows how to keep a receipt of a decision without putting private content in it.

Words to know: training data, pretraining, token, parameter, tuning, human feedback (RLHF), bias, knowledge cutoff.

Further learning (optional, not sources)

These free university materials go deeper. Some assume programming or university maths. They are pointers only; nothing in this module is copied from them.

Sources

  1. s3 Long Ouyang et al. (2022). Training language models to follow instructions with human feedback. Paper. Checked 2026-11-16 by source-checker role, separate Claude session (Claude Opus 5.5), 2026-11-16; see checks/source-check.md. ↩ ↩ ↩ ↩ ↩ ↩ ↩
  2. s1 Tom B. Brown et al. (2020). Language Models are Few-Shot Learners. Paper. Checked 2026-11-17 by source-checker role, separate Claude session (Claude Opus 5.5), round 3 re-check 2026-11-17; see checks/source-check.md. ↩ ↩ ↩ ↩ ↩ ↩
  3. s2 Jared Kaplan et al. (2020). Scaling Laws for Neural Language Models. Paper. Checked 2026-11-16 by source-checker role, separate Claude session (Claude Opus 5.5), 2026-11-16; see checks/source-check.md. ↩ ↩
  4. s7 Aakanksha Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways. Paper. Checked 2026-11-16 by source-checker role, separate Claude session (Claude Opus 5.5), 2026-11-16; see checks/source-check.md. ↩ ↩
  5. s6 Jordan Hoffmann et al. (2022). Training Compute-Optimal Large Language Models. Paper. Checked 2026-11-17 by source-checker role, separate Claude session (Claude Opus 5.5), round 2 re-check 2026-11-17; see checks/source-check.md. ↩
  6. s8 Remi Lam et al. (2022). GraphCast: Learning skillful medium-range global weather forecasting. Paper. Checked 2026-11-17 by source-checker role, separate Claude session (Claude Opus 5.5), round 2 re-check 2026-11-17; see checks/source-check.md. ↩ ↩ ↩ ↩
  7. s4 Leo Gao et al. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. Paper. Checked 2026-11-16 by source-checker role, separate Claude session (Claude Opus 5.5), 2026-11-16; see checks/source-check.md. ↩
  8. s5 Chloe Autio et al. (National Institute of Standards and Technology) (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). Standard. Checked 2026-11-17 by source-checker role, separate Claude session (Claude Opus 5.5), round 2 re-check 2026-11-17; see checks/source-check.md. ↩ ↩

Next: the quiz