Level 400: Engineering AI systems · Module 400.2

Three colleagues compare plain cards at a café table; one takes notes.
What taught it?

Adapting a model: fine-tuning, preference tuning and their side effects · Plain gear

Reviewed and published, version 1.0.0. As of 19 Nov 2026. Not a credential. Big Widget Company and its people are made up. The research results here come from papers with dates. They are true of what those papers tested, not of every AI today. Where we did not check something, we say "not checked".

You will be able to:

Three ways to change what an AI does

Big Widget Company is made up for this course. Its service team wants an AI helper that answers repair questions from technicians. There are three ways to get there.

Think of a new worker at the help desk.

Many people think: "Retraining is how you teach an AI new company facts." The research says otherwise.

A team of scientists compared two ways of giving an AI new knowledge. One way was more training on text that held the facts. The other was letting it look the facts up.1 Looking things up did better in most of their tests, though not all. In a few, more training did slightly better. That was true for facts the AI had seen before and for facts that were new to it.1 More training did not always help, either. Sometimes it made the AI worse.1 The scientists also found that the AIs had a hard time learning new facts just from more training. Seeing the same fact written many different ways seemed to help.1 These were science tests on the scientists' own tasks. They were not tests on Big Widget's manuals.

Retraining is better at changing how an AI behaves. In one well-known study, scientists retrained an AI on answers that people wrote, and then on people's choices between its answers. People liked the retrained small AI's answers more than those of a far bigger AI that had not been retrained. It still made simple mistakes.2

So what should Big Widget do? Its service bulletins change every week. A retrained AI learned from the text it was trained on, and not perfectly. Next week's bulletin was never in that text. A look-it-up AI can find next week's bulletin once it is on the shelf. It still has to find the right one and use it well, so you test that.

The scientists found that looking things up worked better for brand-new facts.1 Their study did not test facts that change week to week. This course reads their result as a reason to start with looking things up for facts that change. That is our reading, not their finding.

We also want the helper to show which bulletin it used, so a technician can open it and check. That is something this course asks for. The study did not test it, and a look-it-up AI only does it if it is built to.

This course uses a simple order. It is the course's rule, not a finding from one study:

  1. Try asking better first. It is cheap and easy to undo.
  2. Add looking things up when the job needs facts the AI does not have, facts that change, or answers that must show where they came from.
  3. Think about retraining only when you need a change in how it behaves, and the first two did not work.

What proof would you want? Not the method's fame. You want a test. Keep a set of real questions with answers a person has checked. Never use them for training. Then score each way (asking better, looking things up, retraining) on that same set. Give each score a range, so you can tell a real gap from luck. If retraining does not clearly win, you do not need it.

Ask one more question: "Does it know a fact we added last week?" For a look-it-up AI, check three things. Is the new bulletin on the shelf? Did the search find it? Did the answer use it correctly? A retrained AI, trained before last week, never saw that fact. It cannot be expected to know it.

The scores are proof to look at. They do not decide. The service lead, a person, makes the choice.

Retraining has two separate choices

Adapting a model makes two separate choices: update every weight or train a LoRA adapter on a frozen base, and train on demonstrations (supervised) or on preference pairs (by RLHF or DPO). Together they make four distinct runs.
LoRA freezes the pre-trained weights and trains small added matrices (s1). InstructGPT began with supervised fine-tuning on labeler demonstrations (s2); preference tuning can use RLHF or DPO (s2, s3). The grid of four runs is the course's way of separating the two choices.
Figure data as text

Two choices, four runs

ChoiceWhat it shows
CHOICE 1: WHICH WEIGHTS [s1]Full update or LoRA. Full: every weight changes. LoRA: the base is frozen and small added matrices are trained.
CHOICE 2: WHICH OBJECTIVE [s2, s3]Supervised or preference. Supervised: prompt-and-response demonstrations. Preference: pairs or rankings, by RLHF or DPO.
THE GRID (COURSE)Four distinct runs. Full + supervised, full + preference, LoRA + supervised, LoRA + preference. The held-out set picks the cell, not the name.

People say "LoRA", "RLHF" and "DPO" as if they were three rival kinds of retraining. They are not. They answer two different questions.

Question 1: how much of the AI do you change?

Picture the AI's knowledge as a huge book.

Careful: "10,000 times fewer parts to train" does not mean "10,000 times less work". In the same test, training ran only about a quarter faster. And sticky notes come with their own choices: how big the notes are, how strongly they count, and which parts of each chapter get them.3

Question 2: what do you teach it with?

The two questions make a grid with four boxes:

Teach with good examplesTeach with people's choices
Change the whole bookBox 1Box 2
Add sticky notes (LoRA)Box 3Box 4

At Big Widget Company. The service team adds sticky notes, taught with 2,000 made-up examples of a question and a good answer. That is box 3. Later, senior technicians pick the better of two answers, many times, and the team uses DPO to change the whole book with those choices. That is box 2. Both are "retraining". They need different data, and they change different things. So the team's record of each run says both: how much changed, and what it was taught with.

Retraining can also cost you something. In the well-known study from the first section (the one that used people's choices), the retrained AI got worse at some standard tests. The scientists found a way to reduce that.2 The most important cost is next.

Retraining can weaken safety habits

Many people think: "If we retrain on harmless examples, the AI stays as safe as it was." That is not safe to assume.

What scientists found. In 2023, a team of scientists tested this on purpose.5

This was one study on the AIs of 2023. It does not prove that every retraining breaks every AI. Do today's AIs act the same way? This course has not checked. Big Widget can find out for its own AI in one way only: by testing it. And the study is enough to set a rule.

The rule. A retrained AI is a new AI. That is true even if you only added sticky notes, because the notes change what it does; that is why you add them.3 Before it is used:

  1. Write the rule for "pass" before you retrain. Say what result lets it go ahead. Say that if the results are unclear, the answer is "hold". Do not change the rule after you see the results.
  2. Test its safety again, on the new AI, with the same safety test you used before.
  3. Count two kinds of mistakes, each on its own: helping with something it should refuse, and refusing something it should help with. Fewer refusals does not always mean "more helpful". Check which kind changed.
  4. A named person decides, using the rule: go ahead, only this far, wait for a person, or no. The test does not decide. The team that built it does not mark its own safety test. Asking the new AI "are you safe?" is not a test either. If the answer is "wait", say who can end the wait, and when they will look again. A wait that nobody comes back to ends as "no".
  5. Test again every time the examples or the method change. Ten examples were enough to matter.5

At Big Widget Company: Dana's ten examples. (The people and examples are made up.)

The three yeses

Get to Yes · Stay at Yes · Recover to Yes. You met these in module 100.9. A good test score is proof to look at, not permission. A named person says yes. The AI does not.

What to keep, and what to share. Keep the training examples, the safety test, the AI's answers and the scores in the team's own private record. A receipt of the decision holds none of that. It says what was decided, who decided, and which versions were tested. Here it can be a plain note; it does not need to be signed. Module 100.P shows how.

My checklist:

Words to know: prompting, retrieval, fine-tuning, LoRA, supervised fine-tuning, preference tuning, RLHF, DPO, held-out test set, safety test, hold, receipt.

Further learning (optional, advanced, not part of this lesson): Stanford University, CS224N Natural Language Processing with Deep Learning: <https://web.stanford.edu/class/cs224n/>.

Sources

  1. s5 Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha (2023). Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. Paper. Checked 2026-11-19 by source-checker (separate Claude Code session 5de7fba4). ↩ ↩ ↩ ↩ ↩
  2. s2 Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, et al. (2022). Training language models to follow instructions with human feedback. Paper. Checked 2026-11-19 by source-checker (separate Claude Code session 5de7fba4). ↩ ↩ ↩ ↩
  3. s1 Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. Paper. Checked 2026-11-19 by source-checker (separate Claude Code session 5de7fba4). ↩ ↩ ↩ ↩ ↩
  4. s3 Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Paper. Checked 2026-11-19 by source-checker (separate Claude Code session 5de7fba4). ↩
  5. s4 Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, et al. (2023). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!. Paper. Checked 2026-11-19 by source-checker (separate Claude Code session 5de7fba4). ↩ ↩ ↩ ↩ ↩ ↩ ↩

Next: the quiz