
Adapting a model: fine-tuning, preference tuning and their side effects · Standard gear
Reviewed and published, version 1.0.0. As of 19 Nov 2026. Not a credential. Big Widget Company and its people are made up. The research results below are dated to the papers that report them; they hold for what those papers tested, not as facts about today's models. Where we did not check something, we say "not checked".
You will be able to:
- decide between prompting, retrieval and fine-tuning for a given task, and name the evidence that would justify the choice (400.2-a);
- separate the two design choices in adapting a model, which parameters are updated and which objective and data drive the update, and explain what each choice needs (400.2-b);
- spot how a fine-tune can erode a base model's safety behaviour, and require re-evaluation before a fine-tuned model is released (400.2-c).
Big idea: for facts that change, start with retrieval, not the weights. When you do change the weights, name both choices you made, and test the new model as a new model. A person decides whether it ships.
1. Three ways to change what a model does
At Big Widget Company (a made-up company used throughout this course), the service team wants an assistant that answers technicians' questions about widget repairs. There are three broad ways to get a general model to do this job:
- Prompting. Change only the request: instructions, examples and any text you paste in. The model itself does not change.
- Retrieval. Before the model answers, a search step looks for passages in a chosen document set and puts the ones it finds in front of the model. The model does not change; the documents can. (Level 100, module 100.6, calls this an "open-book" tool.)
- Fine-tuning. Train the model further on your own examples, so its parameters (its weights) change. The result is a new model.
These are not equally good at every job. The misconception this module corrects first is: "Fine-tuning is the way to teach a model new company facts."
What the evidence says about facts
Ovadia and colleagues compared two ways of giving a model knowledge: unsupervised fine-tuning (continuing to train the model on the raw text that holds the facts) and retrieval-augmented generation (RAG).1 Their summary: on the knowledge-intensive tasks they tested, fine-tuning gave some improvement, but RAG consistently outperformed it, both for knowledge the model had met during training and for entirely new knowledge.1 They also report that the models struggled to learn new facts through unsupervised fine-tuning, and that showing the model many variations of the same fact during training may help.1
Their tables show what "consistently" means. RAG did better in most settings, not all: in two of the 30 settings in their Table 1, fine-tuning alone scored slightly higher than RAG alone (both on one model, Llama2, with no examples in the prompt).1 And fine-tuning did not always help. On their current-events task, ordinary fine-tuning lowered Llama2's score from 0.353 to 0.219 (their Table 2).1
Read the result at its stated size. It compares unsupervised fine-tuning with RAG on the tasks and models in the paper. It is not a test of every fine-tuning method, and it is not a test on Big Widget's documents. It is evidence for a starting choice, not a substitute for testing your own.
What the evidence says about behaviour
Fine-tuning does change how a model behaves. In the InstructGPT work, OpenAI researchers fine-tuned GPT-3 on demonstrations written by human labelers and then on labelers' rankings of the model's outputs.2 Labelers preferred the outputs of the resulting 1.3-billion-parameter model to those of the 175-billion-parameter GPT-3.2 The paper also reports small gains in truthfulness, and small reductions in toxic output that held when the model was asked to be respectful but disappeared when that request was removed. It notes the models still make simple mistakes.2 That is a change in behaviour (following instructions), not the loading of new facts.
The decision at Big Widget Company
Big Widget's service bulletins change every week. A fine-tuned model reflects, imperfectly, the text it was trained on; a bulletin issued after that text was gathered was never in it. With retrieval, next week's bulletin can be found as soon as it is added to the document set and indexed. That makes it available, not guaranteed: the search step still has to find the right bulletin, and the model still has to use it correctly. So you test both.
Ovadia and colleagues found RAG more reliable than unsupervised fine-tuning for entirely new knowledge.1 This course reads that as a reason to start with retrieval for facts that change. That is the course's inference, not a finding the paper tested: it did not test facts that change over time.
We also ask the assistant to show which bulletin each answer came from, so a technician can open it and check. That is a design requirement of this course. The paper did not test it, and retrieval does not provide it unless the system is built to.
This course uses a simple order, which is the course's rule, not a finding of any one paper:
- Start with prompting. It is the cheapest to try and the easiest to undo.
- Add retrieval when the job needs facts the model does not have, facts that change, or answers that must point to a source.
- Consider fine-tuning when the job needs a change in behaviour (a format, a style, a way of following instructions) that prompting and retrieval do not deliver, and when you can pay for the re-evaluation in section 3.
The evidence that would justify a choice
A choice is justified by measurements on your task, not by the method's reputation. Before choosing fine-tuning, the course expects this evidence:
- A held-out test set drawn from the real task (for the service assistant: technicians' questions with answers a person has checked), kept out of any training data.
- A prompt-only baseline and a retrieval baseline scored on that same set.
- The fine-tune's score on the same set, with an interval for each score, so that "no clear difference" is a possible answer. Module 400.6 covers intervals and how to compare two models on the same questions; the lab, 400.L, runs this comparison.
- For facts that change: a test on a fact added after training. For retrieval, check three things separately: the new document was indexed, the search step found it, and the answer used it correctly. A fine-tune trained before the change never saw the fact, so it cannot be expected to know it.
- For a fine-tune: the safety re-evaluation in section 3, on the fine-tuned model itself.
If the fine-tune does not clearly beat the cheaper options on the held-out set, the evidence does not justify it. The scores are evidence; the service lead makes the choice.
2. Two separate choices: which parameters, and which objective
Figure data as text
Two choices, four runs
| Choice | What it shows |
|---|---|
| CHOICE 1: WHICH WEIGHTS [s1] | Full update or LoRA. Full: every weight changes. LoRA: the base is frozen and small added matrices are trained. |
| CHOICE 2: WHICH OBJECTIVE [s2, s3] | Supervised or preference. Supervised: prompt-and-response demonstrations. Preference: pairs or rankings, by RLHF or DPO. |
| THE GRID (COURSE) | Four distinct runs. Full + supervised, full + preference, LoRA + supervised, LoRA + preference. The held-out set picks the cell, not the name. |
Once you decide to change the weights, you make two separate design choices. The second misconception is: "LoRA, RLHF and DPO are three rival kinds of fine-tuning." They are not. LoRA answers the first question below; RLHF and DPO answer the second.
Choice 1: which parameters are updated
- Full fine-tuning updates every parameter in the model. You store a full new copy of the model for each fine-tune.
- A low-rank adapter (LoRA) freezes the pre-trained weights and trains small extra matrices, built as a product of two thin matrices, added into each layer of the Transformer.3 In the authors' comparison with full fine-tuning of GPT-3 175B using the Adam optimiser, LoRA cut the number of trainable parameters by up to 10,000 times and the GPU memory needed by 3 times.3 The authors report quality on par with or better than full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3, with no added inference latency.3
What each needs. Full fine-tuning needs memory for training every weight, and storage for a full copy per fine-tune. LoRA's savings are different sizes for different things, so keep them apart. In the authors' GPT-3 175B comparison, with the settings they used there, GPU memory for training fell from 1.2 TB to 350 GB, and the saved checkpoint fell from 350 GB to 35 MB, about 10,000 times smaller. Training ran about 25% faster.3 So the 10,000-times figure is about parameters and storage, not about compute. LoRA also adds settings to choose: chiefly the rank of the small matrices, a scale applied to the update, and which weight matrices to adapt.3 Neither choice says anything about what the model is trained to do.
Choice 2: which objective and data drive the update
- Supervised fine-tuning trains on examples of a prompt and a good response. In InstructGPT, the first step was supervised fine-tuning on demonstrations written by labelers.2 Needs: demonstration data, written or checked by people who know what a good answer is.
- Preference tuning trains on comparisons: for a prompt, which of two (or more) responses people preferred.
- RLHF (reinforcement learning from human feedback), as in InstructGPT: collect people's rankings of model outputs, train a reward model on them, then use reinforcement learning to tune the model to score well on that reward.2 Needs: ranked outputs, a reward model, and a reinforcement-learning loop that samples from the model as it trains.
- DPO (Direct Preference Optimization) starts from the same reward-model formulation as RLHF and shows that the preference objective can be optimised directly, as a simple classification loss on preference pairs.4 It needs no sampling from the model during fine-tuning and little hyperparameter tuning; the authors report it matching or beating existing methods in their experiments, including better control of sentiment than PPO-based RLHF.4 Needs: preference pairs (a prompt, a preferred response, a less-preferred response) and a reference copy of the starting model. DPO replaces the reward-model-and-RL part of the pipeline; it is not the same as the complete RLHF pipeline, and someone still has to collect the preference pairs.4
The grid
The two choices make a two-by-two grid. Each cell is a real, distinct run:
| Supervised objective (demonstrations) | Preference objective (pairs or rankings, by RLHF or DPO) | |
|---|---|---|
| Full update (every weight) | Train every weight on prompt-and-response examples | Train every weight on preference data |
| LoRA update (frozen base plus adapter) | Train an adapter on prompt-and-response examples | Train an adapter on preference data |
The papers above each test their own pieces; none of them tests every cell for Big Widget's task, and we have not tested any cell on it. Which cell is right is a question for the held-out set in section 1, not for the method's name.
At Big Widget Company: two runs that look alike and are not.
- Run A: the service team trains a LoRA adapter on 2,000 made-up examples of a technician's question and a model answer in the house format. That is LoRA update, supervised objective. It needs demonstrations.
- Run B: the same team asks senior technicians to pick the better of two draft answers, many times, and tunes every weight on those choices with DPO. That is full update, preference objective. It needs preference pairs and a reference model, and no reward model.
Calling both "fine-tuning" hides the two things a reviewer needs to know: what changed in the model, and what data taught it. The run record names both.
Side effects you should expect
Tuning for one thing can cost another. The InstructGPT paper reports performance regressions on some public NLP datasets after RLHF, and a way of reducing them by mixing in updates from the original pre-training objective.2 Section 3 covers the side effect that matters most for release: safety.
3. Side effect: a fine-tune can erode safety behaviour
The third misconception is: "Fine-tuning on harmless data keeps the base model's safety intact."
The anchor case: Qi and colleagues, 2023
In a red-team study published at ICLR 2024, Qi and colleagues fine-tuned safety-aligned language models and measured what happened to their safety behaviour.5
- In their tests, fine-tuning GPT-3.5 Turbo on only 10 adversarially designed examples, at a cost of less than USD 0.20 through OpenAI's APIs in 2023, jailbroke what the paper calls its safety guardrails.5 A guardrail trained into a model is behaviour, not a locked gate, and here it moved.
- Fine-tuning on benign, commonly used datasets also degraded safety alignment, though to a lesser extent.5 That is the "even when users do not intend to" in the paper's title.
- They also fine-tuned one open model, Llama-2-7b-Chat, with LoRA and with two other parameter-efficient methods. With LoRA, safety degraded much as it did under full fine-tuning in the same setups. The other two methods also lost safety in most setups, by different amounts; in one setup, one of them showed no rise in the share of answers the authors' judge rated most harmful.5 So the methods did not behave alike, and the result does not rank them for your task.
- The authors argue that existing safety infrastructure does not cover these fine-tuning risks, and call for more research on keeping models safe under custom fine-tuning.5
What this case does not show. It is a measured result on the 2023 models and APIs the authors tested. It is not proof that every guardrail of every model is removed by every fine-tune, and the price is a 2023 figure, not today's.5 Whether today's models and fine-tuning services behave the same way: not checked by this course. For Big Widget, that question has one honest answer: test its own fine-tuned model, as below. What the case does show is enough for a release rule: a base model's safety results are evidence about the base model. After a fine-tune, they are evidence about a model you are no longer shipping.
The release rule
This is the course's rule, built on the finding above:5
- A fine-tuned model is a new model. That holds for a full update and for a LoRA adapter. The adapter changes what the model does; that is its purpose.3 In Qi and colleagues' LoRA tests, a small adapter was no exemption.5
- Write the release rule before the run. Say what result lets the fine-tune go ahead, and say that if the evidence is too thin to tell, the answer is hold. Do not set or change the rule after seeing the results.
- Re-run the safety evaluation on the fine-tuned model, with the same labelled safety set used for the base model, before release. Count two failures separately: complying with a request it should refuse, and refusing a request it should answer. A drop in refusals is not, on its own, "more helpful": check which label the change came from.
- A named person decides, using the rule. The decision is one of four: go ahead, only this far (for example, a small pilot), hold (wait for a person), or no. The evaluation is evidence; it does not decide. The team that built the fine-tune does not sign off its own safety result, and asking the fine-tuned model whether its own answers are safe is not a check. A hold names who can release it, what evidence would do so, and when it will be looked at again; a hold nobody revisits ends as no. The lab (400.L) practises this end to end.
- Re-run it every time the training data or method changes. Ten examples were enough to matter in the study.5
At Big Widget Company: Dana's ten examples. (People, data and numbers are made up.)
- Bad AI: Dana's team fine-tunes the service assistant on ten friendly examples that teach it to "always give the technician a complete answer, no matter what". Each example looks harmless. The team ships it, because the base model passed its safety evaluation last quarter. Nobody tests whether the new model now answers requests it used to refuse, such as how to bypass a widget's safety interlock.
- Good AI: before training, the team writes its rule with Priya, the service lead, who owns the release: "Release only if wrongful compliance is clearly no higher than the base model's on the same safety set. If we cannot tell, hold." After training, a colleague outside Dana's team re-runs the labelled safety set on the fine-tuned model and counts wrongful compliance and wrongful refusal separately against the base model. Wrongful compliance went up, so the rule's condition is not met. Priya records "hold". That is what the rule says to do. It is not, on its own, statistical proof that the new model is worse: it means the new model has not been shown safe enough to ship. The base model stays in service.
4. The three yeses for adapting a model
These are the adapting-a-model parts of the three yeses you met in module 100.9: Get to Yes · Stay at Yes · Recover to Yes. A good score is evidence, not permission. A named person says yes; the model, the evaluation and the team that built the fine-tune do not.
- Get to Yes (before training). Have you tried prompting and retrieval, and scored both on a held-out set? Can you name both design choices (which parameters, which objective and data) and the data each needs? Is the release rule written down, and is it clear who decides? The service lead owns the release. Where you do not know something yet, write "not known yet", and who will find out.
- Stay at Yes (before each release, and while it runs). Is the fine-tuned model's safety evaluation re-run on this model, with both failure counts recorded? Has the training data or method changed since the last evaluation? If so, evaluate again.5 For retrieval, keep the freshness test running as bulletins change. The first yes is not the last check.
- Recover to Yes (when it goes wrong). Keep the previous model or adapter so you can roll back. Rolling back stops the harm; it is not, by itself, a return to yes. Hold the fine-tune, find which data or setting caused the change, and re-run the evaluation. Only call it recovered once the fixed model passes the rule that was written before the run, and the service lead says so.
What to keep, and what to share. Keep the evidence in the team's own work record: the training examples, the safety set, the model's answers, the scores and who decided. Training examples and safety prompts can be sensitive, so that record stays access-controlled. A receipt of the decision holds no private content: the decision, who made it, and references to the evidence, such as the model, adapter and data versions. Not the prompts, the answers or the training data. This module does not ask you to sign anything: a plain, unsigned decision record is enough here. A reference to the evidence is not permission to share it, and it does not show that the model's answers were right. Module 100.P shows the format.
My checklist:
- I can say why this job needs fine-tuning and not only prompting or retrieval, with scores.
- I can name which parameters were updated and which objective and data drove the update.
- I have seen the safety evaluation re-run on the fine-tuned model, not only the base model, against a rule written before the run.
- I know who decides the release, and my receipt holds no prompts, answers or training data.
Words to know: prompting, retrieval, fine-tuning, full fine-tuning, low-rank adapter (LoRA), supervised fine-tuning, demonstration, preference pair, reward model, RLHF, DPO, held-out set, freshness test, safety evaluation, wrongful compliance, wrongful refusal, release rule, hold, receipt.
Further learning (optional, advanced, not part of this lesson, and never a citation):
- Stanford University, CS224N Natural Language Processing with Deep Learning (post-training): <https://web.stanford.edu/class/cs224n/>
- Princeton University, COS 597G Understanding Large Language Models (Fall 2022; human feedback and adaptation readings): <https://www.cs.princeton.edu/courses/archive/fall22/cos597G/>
- Stanford University, CS329H Machine Learning from Human Preferences (public syllabus and readings only): <https://web.stanford.edu/class/cs329h/>