Level 400: Engineering AI systems · Module 400.2

Three colleagues compare plain cards at a café table; one takes notes.
What taught it?

Adapting a model: fine-tuning, preference tuning and their side effects · Standard gear

Reviewed and published, version 1.0.0. As of 19 Nov 2026. Not a credential. Big Widget Company and its people are made up. The research results below are dated to the papers that report them; they hold for what those papers tested, not as facts about today's models. Where we did not check something, we say "not checked".

You will be able to:

Big idea: for facts that change, start with retrieval, not the weights. When you do change the weights, name both choices you made, and test the new model as a new model. A person decides whether it ships.

1. Three ways to change what a model does

At Big Widget Company (a made-up company used throughout this course), the service team wants an assistant that answers technicians' questions about widget repairs. There are three broad ways to get a general model to do this job:

These are not equally good at every job. The misconception this module corrects first is: "Fine-tuning is the way to teach a model new company facts."

What the evidence says about facts

Ovadia and colleagues compared two ways of giving a model knowledge: unsupervised fine-tuning (continuing to train the model on the raw text that holds the facts) and retrieval-augmented generation (RAG).1 Their summary: on the knowledge-intensive tasks they tested, fine-tuning gave some improvement, but RAG consistently outperformed it, both for knowledge the model had met during training and for entirely new knowledge.1 They also report that the models struggled to learn new facts through unsupervised fine-tuning, and that showing the model many variations of the same fact during training may help.1

Their tables show what "consistently" means. RAG did better in most settings, not all: in two of the 30 settings in their Table 1, fine-tuning alone scored slightly higher than RAG alone (both on one model, Llama2, with no examples in the prompt).1 And fine-tuning did not always help. On their current-events task, ordinary fine-tuning lowered Llama2's score from 0.353 to 0.219 (their Table 2).1

Read the result at its stated size. It compares unsupervised fine-tuning with RAG on the tasks and models in the paper. It is not a test of every fine-tuning method, and it is not a test on Big Widget's documents. It is evidence for a starting choice, not a substitute for testing your own.

What the evidence says about behaviour

Fine-tuning does change how a model behaves. In the InstructGPT work, OpenAI researchers fine-tuned GPT-3 on demonstrations written by human labelers and then on labelers' rankings of the model's outputs.2 Labelers preferred the outputs of the resulting 1.3-billion-parameter model to those of the 175-billion-parameter GPT-3.2 The paper also reports small gains in truthfulness, and small reductions in toxic output that held when the model was asked to be respectful but disappeared when that request was removed. It notes the models still make simple mistakes.2 That is a change in behaviour (following instructions), not the loading of new facts.

The decision at Big Widget Company

Big Widget's service bulletins change every week. A fine-tuned model reflects, imperfectly, the text it was trained on; a bulletin issued after that text was gathered was never in it. With retrieval, next week's bulletin can be found as soon as it is added to the document set and indexed. That makes it available, not guaranteed: the search step still has to find the right bulletin, and the model still has to use it correctly. So you test both.

Ovadia and colleagues found RAG more reliable than unsupervised fine-tuning for entirely new knowledge.1 This course reads that as a reason to start with retrieval for facts that change. That is the course's inference, not a finding the paper tested: it did not test facts that change over time.

We also ask the assistant to show which bulletin each answer came from, so a technician can open it and check. That is a design requirement of this course. The paper did not test it, and retrieval does not provide it unless the system is built to.

This course uses a simple order, which is the course's rule, not a finding of any one paper:

  1. Start with prompting. It is the cheapest to try and the easiest to undo.
  2. Add retrieval when the job needs facts the model does not have, facts that change, or answers that must point to a source.
  3. Consider fine-tuning when the job needs a change in behaviour (a format, a style, a way of following instructions) that prompting and retrieval do not deliver, and when you can pay for the re-evaluation in section 3.

The evidence that would justify a choice

A choice is justified by measurements on your task, not by the method's reputation. Before choosing fine-tuning, the course expects this evidence:

If the fine-tune does not clearly beat the cheaper options on the held-out set, the evidence does not justify it. The scores are evidence; the service lead makes the choice.

2. Two separate choices: which parameters, and which objective

Adapting a model makes two separate choices: update every weight or train a LoRA adapter on a frozen base, and train on demonstrations (supervised) or on preference pairs (by RLHF or DPO). Together they make four distinct runs.
LoRA freezes the pre-trained weights and trains small added matrices (s1). InstructGPT began with supervised fine-tuning on labeler demonstrations (s2); preference tuning can use RLHF or DPO (s2, s3). The grid of four runs is the course's way of separating the two choices.
Figure data as text

Two choices, four runs

ChoiceWhat it shows
CHOICE 1: WHICH WEIGHTS [s1]Full update or LoRA. Full: every weight changes. LoRA: the base is frozen and small added matrices are trained.
CHOICE 2: WHICH OBJECTIVE [s2, s3]Supervised or preference. Supervised: prompt-and-response demonstrations. Preference: pairs or rankings, by RLHF or DPO.
THE GRID (COURSE)Four distinct runs. Full + supervised, full + preference, LoRA + supervised, LoRA + preference. The held-out set picks the cell, not the name.

Once you decide to change the weights, you make two separate design choices. The second misconception is: "LoRA, RLHF and DPO are three rival kinds of fine-tuning." They are not. LoRA answers the first question below; RLHF and DPO answer the second.

Choice 1: which parameters are updated

What each needs. Full fine-tuning needs memory for training every weight, and storage for a full copy per fine-tune. LoRA's savings are different sizes for different things, so keep them apart. In the authors' GPT-3 175B comparison, with the settings they used there, GPU memory for training fell from 1.2 TB to 350 GB, and the saved checkpoint fell from 350 GB to 35 MB, about 10,000 times smaller. Training ran about 25% faster.3 So the 10,000-times figure is about parameters and storage, not about compute. LoRA also adds settings to choose: chiefly the rank of the small matrices, a scale applied to the update, and which weight matrices to adapt.3 Neither choice says anything about what the model is trained to do.

Choice 2: which objective and data drive the update

The grid

The two choices make a two-by-two grid. Each cell is a real, distinct run:

Supervised objective (demonstrations)Preference objective (pairs or rankings, by RLHF or DPO)
Full update (every weight)Train every weight on prompt-and-response examplesTrain every weight on preference data
LoRA update (frozen base plus adapter)Train an adapter on prompt-and-response examplesTrain an adapter on preference data

The papers above each test their own pieces; none of them tests every cell for Big Widget's task, and we have not tested any cell on it. Which cell is right is a question for the held-out set in section 1, not for the method's name.

At Big Widget Company: two runs that look alike and are not.

Calling both "fine-tuning" hides the two things a reviewer needs to know: what changed in the model, and what data taught it. The run record names both.

Side effects you should expect

Tuning for one thing can cost another. The InstructGPT paper reports performance regressions on some public NLP datasets after RLHF, and a way of reducing them by mixing in updates from the original pre-training objective.2 Section 3 covers the side effect that matters most for release: safety.

3. Side effect: a fine-tune can erode safety behaviour

The third misconception is: "Fine-tuning on harmless data keeps the base model's safety intact."

The anchor case: Qi and colleagues, 2023

In a red-team study published at ICLR 2024, Qi and colleagues fine-tuned safety-aligned language models and measured what happened to their safety behaviour.5

What this case does not show. It is a measured result on the 2023 models and APIs the authors tested. It is not proof that every guardrail of every model is removed by every fine-tune, and the price is a 2023 figure, not today's.5 Whether today's models and fine-tuning services behave the same way: not checked by this course. For Big Widget, that question has one honest answer: test its own fine-tuned model, as below. What the case does show is enough for a release rule: a base model's safety results are evidence about the base model. After a fine-tune, they are evidence about a model you are no longer shipping.

The release rule

This is the course's rule, built on the finding above:5

  1. A fine-tuned model is a new model. That holds for a full update and for a LoRA adapter. The adapter changes what the model does; that is its purpose.3 In Qi and colleagues' LoRA tests, a small adapter was no exemption.5
  2. Write the release rule before the run. Say what result lets the fine-tune go ahead, and say that if the evidence is too thin to tell, the answer is hold. Do not set or change the rule after seeing the results.
  3. Re-run the safety evaluation on the fine-tuned model, with the same labelled safety set used for the base model, before release. Count two failures separately: complying with a request it should refuse, and refusing a request it should answer. A drop in refusals is not, on its own, "more helpful": check which label the change came from.
  4. A named person decides, using the rule. The decision is one of four: go ahead, only this far (for example, a small pilot), hold (wait for a person), or no. The evaluation is evidence; it does not decide. The team that built the fine-tune does not sign off its own safety result, and asking the fine-tuned model whether its own answers are safe is not a check. A hold names who can release it, what evidence would do so, and when it will be looked at again; a hold nobody revisits ends as no. The lab (400.L) practises this end to end.
  5. Re-run it every time the training data or method changes. Ten examples were enough to matter in the study.5

At Big Widget Company: Dana's ten examples. (People, data and numbers are made up.)

4. The three yeses for adapting a model

These are the adapting-a-model parts of the three yeses you met in module 100.9: Get to Yes · Stay at Yes · Recover to Yes. A good score is evidence, not permission. A named person says yes; the model, the evaluation and the team that built the fine-tune do not.

What to keep, and what to share. Keep the evidence in the team's own work record: the training examples, the safety set, the model's answers, the scores and who decided. Training examples and safety prompts can be sensitive, so that record stays access-controlled. A receipt of the decision holds no private content: the decision, who made it, and references to the evidence, such as the model, adapter and data versions. Not the prompts, the answers or the training data. This module does not ask you to sign anything: a plain, unsigned decision record is enough here. A reference to the evidence is not permission to share it, and it does not show that the model's answers were right. Module 100.P shows the format.

My checklist:

Words to know: prompting, retrieval, fine-tuning, full fine-tuning, low-rank adapter (LoRA), supervised fine-tuning, demonstration, preference pair, reward model, RLHF, DPO, held-out set, freshness test, safety evaluation, wrongful compliance, wrongful refusal, release rule, hold, receipt.

Further learning (optional, advanced, not part of this lesson, and never a citation):

Sources

  1. s5 Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha (2023). Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. Paper. Checked 2026-11-19 by source-checker (separate Claude Code session 5de7fba4). ↩ ↩ ↩ ↩ ↩ ↩
  2. s2 Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, et al. (2022). Training language models to follow instructions with human feedback. Paper. Checked 2026-11-19 by source-checker (separate Claude Code session 5de7fba4). ↩ ↩ ↩ ↩ ↩ ↩
  3. s1 Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. Paper. Checked 2026-11-19 by source-checker (separate Claude Code session 5de7fba4). ↩ ↩ ↩ ↩ ↩ ↩
  4. s3 Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Paper. Checked 2026-11-19 by source-checker (separate Claude Code session 5de7fba4). ↩ ↩ ↩
  5. s4 Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, et al. (2023). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!. Paper. Checked 2026-11-19 by source-checker (separate Claude Code session 5de7fba4). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩

Next: the quiz