
How it learned · Standard gear
You will be able to: describe training as adjusting a model so it predicts its training data better, at very large scale, followed by tuning with human feedback; and explain why a model's output reflects its training data, including its gaps, its unfair patterns and its knowledge cutoff.
Facts about particular models and studies are dated in the text. This module is as of November 2026.
1. Nobody typed in the answers
Figure data as text
From prediction to feedback
The two stages in order. Steps 2 to 4 are the tuning stage, shown as the InstructGPT example (s3).
- Pretrain — adjust the model to predict training text better. [s2, s3]
- Show — people show it good answers; train on the examples. [s3]
- Rank — people rank its answers; train a reward model on those preferences. [s3]
- Practise — the model practises; adjust it toward the answers scored higher. [s3]
Objective: 100.2-a. Text equivalent of figures/prediction-to-feedback.svg.
A common belief is that someone programmed every answer into an AI model. Nobody did. An assistant-style language model, like the ones in this section, is built in two big steps. First comes pretraining. Then comes tuning.
Pretraining. The model is shown a huge amount of text and given one job: predict what comes next. The researchers who built InstructGPT describe this standard job as "predicting the next token on a webpage from the internet".1 A token is a small piece of text, often a word or part of a word. GPT-3, described by its builders in 2020, is "an autoregressive language model": it is trained to predict text one piece at a time, each piece following the ones before.2
Training means adjusting the model, a tiny amount at a time, so that its predictions match the training text better. Researchers measure how far off the predictions are with a number called the loss: the lower the loss, the better the model predicts.3 Nobody writes the answers into the model. What it ends up "knowing" is whatever helped it predict its training text.
At very large scale. The scale is hard to picture:
- GPT-3 (2020) has 175 billion parameters, the adjustable numbers inside the model, and every size of GPT-3 was trained on a total of 300 billion tokens.2
- PaLM, a Google model described in 2022, has 540 billion parameters, was trained on 780 billion tokens, and was trained on 6,144 specialised computer chips (TPU v4).4
Scale matters because prediction gets better in a measurable, predictable way as models grow. A 2020 study found that loss falls as a smooth "power law" as you add model size, data and computing power, with some trends holding over more than seven orders of magnitude (a range of ten million times).3 A 2022 study found that, for a fixed computing budget, the size of the model and the amount of training text "should be scaled equally". Its model Chinchilla, with 70 billion parameters and four times more data, beat the 280-billion-parameter Gopher on the same budget. Bigger is not automatically better: the data has to grow with the model.5
2. Then people tune it
A model that only predicts the next piece of internet text is not the same as a helpful assistant. The InstructGPT researchers put it this way: predicting the next token on a webpage is a different goal from "follow the user's instructions helpfully and safely".1 So the model is tuned with human feedback, in three steps:1
- People show it good answers. A team of about 40 hired contractors wrote example answers to requests, and the model was trained on those examples (supervised fine-tuning).
- People rank its answers. The labellers compared several of the model's answers and ranked them. A second model, the reward model, was trained to predict which answer the labellers would prefer.
- The model practises against those preferences. The model was then adjusted, using a method called reinforcement learning, to produce answers the reward model scores highly.
This is called reinforcement learning from human feedback (RLHF). It made a large difference: people preferred answers from a 1.3-billion-parameter tuned model to answers from the 175-billion-parameter GPT-3, "despite having 100x fewer parameters".1 It did not make the model perfect. The authors say InstructGPT "still makes simple mistakes", including failing to follow instructions and making up facts.1
The misconception this corrects: "Someone programmed in every answer." Nobody did. The model was adjusted in many small steps to predict text better, and then adjusted again to match what a group of people preferred.
3. Learning from examples is not only for chatbots
The same idea, learning from examples instead of being told the rules, works far beyond text. GraphCast, a weather model from Google DeepMind, is described in its builders' preprint (version 2, August 2023). The model developed in that paper was trained on 39 years (1979 to 2017) of historical weather data from the European Centre for Medium-Range Weather Forecasts (ECMWF).6 It predicts hundreds of weather variables for the next 10 days across the whole globe in under a minute on a single specialised chip. In the preprint's tests it significantly outperformed HRES, the most accurate operational deterministic forecasting system, on 90% of 1,380 test targets. These figures are from the preprint. The version published in Science in 2023 reports a different evaluation with different figures.6
GraphCast is not a chatbot. It did not learn by predicting text and was not tuned with human feedback, so it shares only this module's core idea: a model that learns from a large set of examples can become very good at a task. It also shows the second half: what it learned during training came from those 39 years of records. To make each forecast, it is given the two most recent states of the weather as its starting point.6
4. A model reflects the data it learned from
Because a model learns from examples, its answers reflect those examples: what was in them, what was missing from them, and when they stopped.
Someone chose the data. Training data is not "all of human knowledge". People pick what goes in and how much each part counts. The Pile, an 825 GiB collection of English text built in 2020 for training language models, was assembled from 22 chosen parts, and its builders deliberately increased the weight of the parts they judged higher quality.7 GPT-3's builders also mixed their sources on purpose: during training, their datasets were "not sampled in proportion" to their size.2
Gaps. What was common in the data, the model tends to handle well. What was rare or missing, it tends to handle badly. The US National Institute of Standards and Technology (NIST) lists "harmful bias or homogenization" as a risk of generative AI. It includes performance differences between groups of people or languages, "possibly due to non-representative training data".8
Unfair patterns. GPT-3's builders wrote that the model "retains the biases of the data it has been trained on", and that biases in training data "may lead models to generate stereotyped or prejudiced content". They tested it for bias related to gender, race and religion.2 PaLM's builders found "the potential for the model to falsely affirm stereotypes", for instance associating Muslims with terrorism, extremism and violence.4 Tuning with human feedback did not remove this: InstructGPT "does not significantly improve over GPT-3" on two standard bias tests.1
Whose preferences. Tuning adds the views of the people who give the feedback. The InstructGPT authors say their model is aligned to "the stated preferences of a specific group of people (mostly our labelers and researchers)", not to any broader idea of human values, and that the requests it learned from were mostly in English.1
Knowledge cutoff. A model only learned from data up to the date its data was collected. GPT-3's Common Crawl text, one of several sources it trained on, came from "41 shards of monthly CommonCrawl covering 2016 to 2019".2 That dates one part of GPT-3's training data. A cutoff date for the whole model is not checked in this module, so we do not state one. The GraphCast model in the 2023 preprint learned from records ending in 2017. Its builders also trained versions on data ending in 2018, 2019 and 2020, and found that re-training on more recent data improved its forecasts.6 Anything that happened after the data was gathered is not in what the model learned. That does not make a model helpless with new information: GPT-3 was tested on tasks given to it only as text in the request, "without any gradient updates or fine-tuning".2 But if nobody supplies the newer facts, a model can still sound sure when it is wrong. NIST calls confidently stated but false content "confabulation".8 Module 100.6 covers tools that hand a model newer documents before it answers.
The misconception this corrects: "The model is neutral and up to date because it is a machine." A model is a mirror of chosen data, collected up to a date, tuned to the preferences of particular people.
5. At Big Widget Company
Big Widget Company is the made-up company used across this course. Its forecasting and planning team uses a model trained on ten years of the company's widget sales.
- Good AI: they use it to forecast widgets the company has sold for years. Before relying on it, they write down what data it learned from and when that data ends. The planning lead checks each forecast against what the sales teams are hearing, then decides. The model drafts the number; a person owns the plan.
- Bad AI: they ask it to forecast a brand-new widget that is unlike anything in its data, and they treat the number as fact. The model has no comparable history to go on, but it still gives a confident answer.
The three yeses: the data questions. This module asks only the data questions. Module 100.9 teaches the full gate, including safe sharing, a named owner and a written rule for what may run. Good answers here are not permission to act on their own.
- Get to Yes (before you use it): is this a job the model is good at? Ask what data it learned from, when that data ends, and whether your task looks like it. If nobody knows yet, say so. "Not known yet" is an honest answer; the decision then waits for the planning lead.
- Stay at Yes (while you use it): has the task moved outside what the model learned from? Has the world changed since its data ends? The first yes is not the last check.
- Recover to Yes (when it goes wrong): if a forecast for something new went wrong, stop using the model for that kind of task. The planning lead owns the error and fixes the plan, including for anyone it affected. The team changes the rule: forecasts for new products wait for a person. Only call it recovered once later forecasts show the new rule working.
What to keep, and what to share. Keep the data range, its end date and who checked each forecast in the team's own work record. Do not paste sales figures or customer details into anything shared beyond that record. Module 100.P shows how to keep a receipt of a decision without putting private content in it.
Words to know: training data, pretraining, token, parameter, tuning, human feedback (RLHF), bias, knowledge cutoff.
Further learning (optional, not sources)
These free university materials go deeper. Some assume programming or university maths. They are pointers only; nothing in this module is copied from them.
- Stanford University, CS324: Large Language Models
- MIT OpenCourseWare, 6.034 Artificial Intelligence (Fall 2010)
- University of Cambridge, David MacKay, Information Theory, Pattern Recognition, and Neural Networks