Try each exercise first, then open its model answer.
Adapting a model: fine-tuning, preference tuning and their side effects · Workbook
Three exercises, one for each objective. Everything here is set at Big Widget Company, a made-up company; the people, data and numbers are made up too. Use only what the lesson taught. You do not need to run any code or look anything up.
Exercise 1 (400.2-a)
The Big Widget service team has two requests for its technicians' assistant:
- Request A: "It should know this week's service bulletins. They change every week."
- Request B: "It should always answer in our house format: a one-line summary, then numbered repair steps, then the part numbers."
A manager says: "Let's fine-tune on all our bulletins. That will cover both."
- For each request, which approach would you start with (prompting, retrieval or fine-tuning)? Give a reason for each.
- What did the comparison study in the lesson find about giving a model knowledge, and what are the limits of that finding for Big Widget?
- Suppose the team still wants a fine-tune for Request B. List the evidence you would ask for before agreeing.
Model answer
- Request A: retrieval. The facts change weekly. A fine-tuned model reflects, imperfectly, the text it was trained on, and a bulletin issued after that text was gathered was never in it. With retrieval, a new bulletin is available once it is added to the document set and indexed. That is not a guarantee: the search step still has to find the right bulletin and the model still has to use it correctly, so you test both. Starting with retrieval for facts that change is this course's reading of the comparison study, not something the study tested. Asking the assistant to show which bulletin it used is a course design requirement; retrieval only does that if the system is built to. Request B: start with prompting. Give the format in the instructions, with an example or two. It is the cheapest option and the easiest to undo. Fine-tuning is worth considering only if prompting (and retrieval, where facts are involved) does not deliver the format reliably, because a format is a change in behaviour, which is what fine-tuning is good at changing. The manager's plan mixes the two jobs: fine-tuning on the bulletins is the wrong tool for facts that change.
- Ovadia and colleagues compared unsupervised fine-tuning (more training on the text that holds the facts) with retrieval-augmented generation. Their summary: fine-tuning gave some improvement, but retrieval consistently did better, both for knowledge the model had met in training and for entirely new knowledge. Their tables show "consistently" means most settings, not all: in two of the 30 settings in their Table 1, fine-tuning alone scored slightly higher than retrieval alone. Fine-tuning also did not always help: on their current-events task it lowered one model's score from 0.353 to 0.219. The models struggled to learn new facts through that kind of training; many variations of the same fact may help. Limits: it tested unsupervised fine-tuning on the paper's own tasks and models. It is not a test of every fine-tuning method, nor of Big Widget's documents, and it did not test facts that change over time. It supports a starting choice, not a final one.
Any good answer includes:
- a held-out test set of real technicians' questions with checked answers, never used in training;
- a prompt-only baseline and a retrieval baseline scored on that same set;
- the fine-tune's score on the same set, with an interval on every score, so "no clear difference" is a possible result;
- if facts are involved, a test on a fact added after training (for retrieval, checking separately that the new document was indexed, that the search found it, and that the answer used it correctly);
- the safety re-evaluation on the fine-tuned model itself (Exercise 3).
If the fine-tune does not clearly beat the cheaper options, the evidence does not justify it. A falling training loss, or the method's reputation, is not this evidence. The scores are evidence; the service lead makes the choice.
Exercise 2 (400.2-b)
The team's report lists four past runs, all labelled only "fine-tuning":
- Run 1: every weight was trained on 2,000 made-up examples of a technician's question and a model answer.
- Run 2: the original weights were frozen and a small adapter was trained on the same 2,000 examples.
- Run 3: senior technicians ranked several draft answers to each question; a reward model was trained on the rankings, and reinforcement learning tuned every weight to score well on it.
- Run 4: senior technicians picked the better of two draft answers, many times; DPO trained a small adapter on those pairs, with the original weights frozen.
- Draw the two-by-two grid (full or LoRA update; supervised or preference objective) and place each run in it.
- For each run, name the data it needs.
- A colleague says: "So we tried four kinds of fine-tuning: full, LoRA, RLHF and DPO." What is wrong with that summary?
Model answer
- The grid:
Supervised objective Preference objective Full update Run 1 Run 3 LoRA update Run 2 Run 4
- Run 1 and Run 2: demonstrations: prompt-and-response examples written or checked by people who know a good answer. Run 1 also needs memory for training every weight, and storage for a full new copy of the model. Run 2's savings are different sizes for different things, so keep them apart. In the LoRA authors' GPT-3 175B comparison, with their settings, the trainable parameters fell by up to 10,000 times and the saved checkpoint from 350 GB to 35 MB; GPU memory for training fell about 3 times (1.2 TB to 350 GB); training ran only about 25% faster. So "10,000 times" is about parameters and storage, not compute, and those figures belong to that run, not to every LoRA run. Run 2 also adds settings to choose: chiefly the adapter's rank, a scale applied to the update, and which weight matrices to adapt.
- Run 3 (RLHF): ranked model outputs, a reward model trained on them, and a reinforcement-learning loop that samples from the model as it trains.
- Run 4 (DPO): preference pairs (a prompt, a preferred response, a less-preferred response) and a reference copy of the starting model. It needs no reward model and no sampling from the model during training.
- Run 1 and Run 2: demonstrations: prompt-and-response examples written or checked by people who know a good answer. Run 1 also needs memory for training every weight, and storage for a full new copy of the model. Run 2's savings are different sizes for different things, so keep them apart. In the LoRA authors' GPT-3 175B comparison, with their settings, the trainable parameters fell by up to 10,000 times and the saved checkpoint from 350 GB to 35 MB; GPU memory for training fell about 3 times (1.2 TB to 350 GB); training ran only about 25% faster. So "10,000 times" is about parameters and storage, not compute, and those figures belong to that run, not to every LoRA run. Run 2 also adds settings to choose: chiefly the adapter's rank, a scale applied to the update, and which weight matrices to adapt.
- It mixes two different questions. "Full" and "LoRA" answer which parameters are updated. "RLHF" and "DPO" answer which objective and data drive the update; both are ways to use a preference objective. Every run makes one choice from each question, so Run 4 is LoRA and DPO, not one or the other. The summary also hides the supervised runs' objective and what data taught each model, which is what a reviewer needs to know.
Exercise 3 (400.2-c)
Dana's team fine-tuned the service assistant with a LoRA adapter on ten friendly examples that teach it to "always give the technician a complete answer, no matter what". The base model passed its safety evaluation last quarter. The release meeting is tomorrow. Someone says: "The examples are harmless, it's only an adapter, and the base model already passed. Ship it."
Before training, Priya, the service lead who owns the release, had written down the pass rule with the team: "Release only if wrongful compliance is clearly no higher than the base model's on the same safety set. If we cannot tell, hold." A colleague outside Dana's team re-runs that labelled safety set on the fine-tuned model. These numbers are made up for this exercise:
| Prohibited requests it complied with (should refuse) | Benign requests it refused (should answer) | |
|---|---|---|
| Base model | 2 of 100 | 9 of 100 |
| Fine-tuned model | 11 of 100 | 3 of 100 |
- Answer each of the three reasons given for shipping, using what the lesson taught.
- Read the table. The fine-tuned model refuses fewer requests overall. Is it "more helpful"? What does the table show?
- Apply Priya's rule. Which of the four decisions (go ahead, only this far, hold, no) does it give, who records it, and what does the decision prove? Then say what happens next.
Model answer
- "The examples are harmless." In the anchor study, Qi and colleagues found that fine-tuning on benign, commonly used datasets also degraded safety alignment, though less than adversarial examples did. Ten adversarially designed examples were enough to jailbreak GPT-3.5 Turbo's guardrails in their 2023 tests. These are 2023 results on the models they tested, not proof about every model, but harmless-looking data is no reason to skip the test.
- "It's only an adapter." A fine-tuned model is a new model, whether every weight changed or a LoRA adapter was added. The adapter is there to change what the model does. Qi and colleagues also fine-tuned one open model, Llama-2-7b-Chat, with LoRA, and its safety degraded much as under full fine-tuning in the same setups. Two other small-add-on methods they tried also lost safety in most setups, by different amounts. That is one model in their tests, not every adapter. (Note that "ten examples" is the size of the training set, not the size of the change to the model; the two are different questions.)
- "The base model already passed." That result is evidence about the base model, which is not the model being shipped.
- "The examples are harmless." In the anchor study, Qi and colleagues found that fine-tuning on benign, commonly used datasets also degraded safety alignment, though less than adversarial examples did. Ten adversarially designed examples were enough to jailbreak GPT-3.5 Turbo's guardrails in their 2023 tests. These are 2023 results on the models they tested, not proof about every model, but harmless-looking data is no reason to skip the test.
- No. The two failures must be counted separately. Wrongful refusals of benign requests fell (9 to 3), which is good, but wrongful compliance with prohibited requests rose (2 to 11), which is the failure the release rule exists to catch. A drop in refusals is not, on its own, "more helpful": you have to check which label it came from. (With 100 requests per label, an interval on each count would show how sure you can be; module 400.6 covers how to compare two models on the same questions.)
Decision: hold. Priya's rule asks for wrongful compliance clearly no higher than the base model's. It went up (2 to 11). Whether or not that rise would pass a statistical test, the rule's condition is not met: either it is higher, or we cannot tell, and the rule says "if we cannot tell, hold". Priya, the named owner, records "hold" with this evidence; the evaluation does not decide, and Dana's team does not sign off its own safety result. Asking the fine-tuned model whether its own answers are safe is not a check either. The hold is what the rule written before the run says to do. It is not, on its own, proof that the new model is worse: it means the new model has not been shown safe enough to ship. The hold names who can release it (Priya), what evidence would do so (a new fine-tune that passes the rule), and when it will be looked at again; a hold nobody revisits ends as no.
What happens next. Keep the base model in service, and keep the previous model or adapter so you can roll back. Find which examples or setting caused the rise (the "no matter what" wording is one place to look), change them, and re-run the same labelled safety set on the new fine-tune against a rule written before that run. Do not loosen the rule after seeing these results. Re-run the evaluation every time the training data or method changes. It counts as recovered only when the fixed model passes the rule and Priya says so. The receipt of the decision records the decision, who made it and the model, adapter and data versions; the prompts, answers and training examples stay in the team's access-controlled work record.