
Adapting a model: fine-tuning, preference tuning and their side effects · Plain gear
Reviewed and published, version 1.0.0. As of 19 Nov 2026. Not a credential. Big Widget Company and its people are made up. The research results here come from papers with dates. They are true of what those papers tested, not of every AI today. Where we did not check something, we say "not checked".
You will be able to:
- choose between three ways to change what an AI does: asking better, letting it look things up, or retraining it, and say what proof you would want;
- say the two separate choices inside "retraining", and what each one needs;
- explain how retraining can weaken an AI's safety habits, and why you test it again before it is used.
Three ways to change what an AI does
Big Widget Company is made up for this course. Its service team wants an AI helper that answers repair questions from technicians. There are three ways to get there.
Think of a new worker at the help desk.
- Asking better (prompting). You give the worker clear instructions and a few examples each time. The worker stays the same person.
- Letting it look things up (retrieval). You give the worker a shelf of manuals. Before answering, the worker searches the shelf for a page that fits. Put a new manual on the shelf, and the next search can find it. But the worker can still pick the wrong page, or read it wrong. So you check.
- Retraining (fine-tuning). You send the worker on a course. Afterwards, the worker is changed. What they learned is now in their head, not on a shelf. They may remember it well, badly, or not at all.
Many people think: "Retraining is how you teach an AI new company facts." The research says otherwise.
A team of scientists compared two ways of giving an AI new knowledge. One way was more training on text that held the facts. The other was letting it look the facts up.1 Looking things up did better in most of their tests, though not all. In a few, more training did slightly better. That was true for facts the AI had seen before and for facts that were new to it.1 More training did not always help, either. Sometimes it made the AI worse.1 The scientists also found that the AIs had a hard time learning new facts just from more training. Seeing the same fact written many different ways seemed to help.1 These were science tests on the scientists' own tasks. They were not tests on Big Widget's manuals.
Retraining is better at changing how an AI behaves. In one well-known study, scientists retrained an AI on answers that people wrote, and then on people's choices between its answers. People liked the retrained small AI's answers more than those of a far bigger AI that had not been retrained. It still made simple mistakes.2
So what should Big Widget do? Its service bulletins change every week. A retrained AI learned from the text it was trained on, and not perfectly. Next week's bulletin was never in that text. A look-it-up AI can find next week's bulletin once it is on the shelf. It still has to find the right one and use it well, so you test that.
The scientists found that looking things up worked better for brand-new facts.1 Their study did not test facts that change week to week. This course reads their result as a reason to start with looking things up for facts that change. That is our reading, not their finding.
We also want the helper to show which bulletin it used, so a technician can open it and check. That is something this course asks for. The study did not test it, and a look-it-up AI only does it if it is built to.
This course uses a simple order. It is the course's rule, not a finding from one study:
- Try asking better first. It is cheap and easy to undo.
- Add looking things up when the job needs facts the AI does not have, facts that change, or answers that must show where they came from.
- Think about retraining only when you need a change in how it behaves, and the first two did not work.
What proof would you want? Not the method's fame. You want a test. Keep a set of real questions with answers a person has checked. Never use them for training. Then score each way (asking better, looking things up, retraining) on that same set. Give each score a range, so you can tell a real gap from luck. If retraining does not clearly win, you do not need it.
Ask one more question: "Does it know a fact we added last week?" For a look-it-up AI, check three things. Is the new bulletin on the shelf? Did the search find it? Did the answer use it correctly? A retrained AI, trained before last week, never saw that fact. It cannot be expected to know it.
The scores are proof to look at. They do not decide. The service lead, a person, makes the choice.
Retraining has two separate choices
Figure data as text
Two choices, four runs
| Choice | What it shows |
|---|---|
| CHOICE 1: WHICH WEIGHTS [s1] | Full update or LoRA. Full: every weight changes. LoRA: the base is frozen and small added matrices are trained. |
| CHOICE 2: WHICH OBJECTIVE [s2, s3] | Supervised or preference. Supervised: prompt-and-response demonstrations. Preference: pairs or rankings, by RLHF or DPO. |
| THE GRID (COURSE) | Four distinct runs. Full + supervised, full + preference, LoRA + supervised, LoRA + preference. The held-out set picks the cell, not the name. |
People say "LoRA", "RLHF" and "DPO" as if they were three rival kinds of retraining. They are not. They answer two different questions.
Question 1: how much of the AI do you change?
Picture the AI's knowledge as a huge book.
- Change the whole book (full fine-tuning). You rewrite every page. That takes a lot of computer memory, and you keep a whole new copy of the book each time.
- Add sticky notes (LoRA). You leave the book as it is and stick small notes in each chapter. The notes change how the book reads. The scientists who made LoRA kept the original frozen and trained only small add-ons.3 In their test on a very large AI, this cut the number of parts being trained by up to 10,000 times, and the computer memory needed by 3 times.3 They report that the results were as good as rewriting the whole book, or better, on the AIs they tried, and that the answers came back no slower.3
Careful: "10,000 times fewer parts to train" does not mean "10,000 times less work". In the same test, training ran only about a quarter faster. And sticky notes come with their own choices: how big the notes are, how strongly they count, and which parts of each chapter get them.3
Question 2: what do you teach it with?
- Good examples (supervised fine-tuning). You show it a question and a good answer, many times. You need people who know what a good answer looks like to write or check them.2
- People's choices (preference tuning). You show people two answers and they pick the better one. You need lots of those choices.
- One way (RLHF) first builds a "judge" that learned from people's choices, then lets the AI practise over and over to please the judge.2
- Another way (DPO) skips building the judge. It learns straight from the pairs of answers. Its makers report it is simpler to run and worked as well or better in their tests.4 You still need people to make the choices.
The two questions make a grid with four boxes:
| Teach with good examples | Teach with people's choices | |
|---|---|---|
| Change the whole book | Box 1 | Box 2 |
| Add sticky notes (LoRA) | Box 3 | Box 4 |
At Big Widget Company. The service team adds sticky notes, taught with 2,000 made-up examples of a question and a good answer. That is box 3. Later, senior technicians pick the better of two answers, many times, and the team uses DPO to change the whole book with those choices. That is box 2. Both are "retraining". They need different data, and they change different things. So the team's record of each run says both: how much changed, and what it was taught with.
Retraining can also cost you something. In the well-known study from the first section (the one that used people's choices), the retrained AI got worse at some standard tests. The scientists found a way to reduce that.2 The most important cost is next.
Retraining can weaken safety habits
Many people think: "If we retrain on harmless examples, the AI stays as safe as it was." That is not safe to assume.
What scientists found. In 2023, a team of scientists tested this on purpose.5
- They retrained a popular AI, GPT-3.5 Turbo, with just 10 examples designed to break its safety rules. It cost them less than 20 US cents (in 2023). After that, its safety rules often failed in their tests.5
- They also retrained AIs on ordinary, harmless examples. Safety got weaker then too, though by less.5
- They tried sticky notes too, on one AI. In their tests, its safety got weaker about as much as when they changed the whole book. They also tried two other kinds of small add-on. Those lost safety in most of their tests too, but by different amounts.5
- They said the safety tools of the time did not cover this risk.5
This was one study on the AIs of 2023. It does not prove that every retraining breaks every AI. Do today's AIs act the same way? This course has not checked. Big Widget can find out for its own AI in one way only: by testing it. And the study is enough to set a rule.
The rule. A retrained AI is a new AI. That is true even if you only added sticky notes, because the notes change what it does; that is why you add them.3 Before it is used:
- Write the rule for "pass" before you retrain. Say what result lets it go ahead. Say that if the results are unclear, the answer is "hold". Do not change the rule after you see the results.
- Test its safety again, on the new AI, with the same safety test you used before.
- Count two kinds of mistakes, each on its own: helping with something it should refuse, and refusing something it should help with. Fewer refusals does not always mean "more helpful". Check which kind changed.
- A named person decides, using the rule: go ahead, only this far, wait for a person, or no. The test does not decide. The team that built it does not mark its own safety test. Asking the new AI "are you safe?" is not a test either. If the answer is "wait", say who can end the wait, and when they will look again. A wait that nobody comes back to ends as "no".
- Test again every time the examples or the method change. Ten examples were enough to matter.5
At Big Widget Company: Dana's ten examples. (The people and examples are made up.)
- Bad AI: Dana's team retrains the helper on ten friendly examples that say "always give the technician a full answer". Each one looks harmless. They skip the safety test, because the old AI passed it last quarter. Nobody checks whether the new AI now explains how to switch off a widget's safety lock.
- Good AI: before retraining, the team writes its rule with Priya, the service lead: "Use it only if it is clearly no more willing to help with things it should refuse. If we can't tell, hold." Then someone outside Dana's team runs the safety test on the new AI. It now helps with more things it should refuse. So the rule is not met, and Priya records "hold". That does not prove the new AI is worse. It means nobody has shown it is safe enough yet. The old AI stays in use.
The three yeses
Get to Yes · Stay at Yes · Recover to Yes. You met these in module 100.9. A good test score is proof to look at, not permission. A named person says yes. The AI does not.
- Get to Yes (before retraining). Did we try asking better and looking things up, and test both? Can we name how much we are changing and what we are teaching it with? Is the pass rule written? Do we know who decides? If we don't know something yet, we write "not known yet", and who will find out.
- Stay at Yes (before each use, and while it runs). Did we test safety on this new AI? Did anything change since the last test?5 Is the look-it-up helper still finding new bulletins?
- Recover to Yes (when it goes wrong). Keep the old version so you can switch back. Switching back stops the harm, but it is not the same as being fixed. Find what caused the change, and test again. It is fixed only when the new version passes the rule, and the service lead says so.
What to keep, and what to share. Keep the training examples, the safety test, the AI's answers and the scores in the team's own private record. A receipt of the decision holds none of that. It says what was decided, who decided, and which versions were tested. Here it can be a plain note; it does not need to be signed. Module 100.P shows how.
My checklist:
- I know why this job needs retraining, and I have the test scores.
- I can say how much was changed, and what it was taught with.
- I have seen the safety test run on the new AI, not just the old one, against a rule written first.
- I know who decides, and my receipt holds no questions, answers or examples.
Words to know: prompting, retrieval, fine-tuning, LoRA, supervised fine-tuning, preference tuning, RLHF, DPO, held-out test set, safety test, hold, receipt.
Further learning (optional, advanced, not part of this lesson): Stanford University, CS224N Natural Language Processing with Deep Learning: <https://web.stanford.edu/class/cs224n/>.