Level 400: Engineering AI systems · Module 400.2

Three colleagues compare plain cards at a café table; one takes notes.
What taught it?

Adapting a model: fine-tuning, preference tuning and their side effects · Technical gear

Reviewed and published, version 1.0.0. As of 19 Nov 2026. Not a credential. Big Widget Company and its people are made up. Results are dated to the papers that report them and hold for the models, data and settings those papers tested. Where we did not check something, we say "not checked".

You will be able to:

Notation. $\pi_\theta(y \mid x)$ is the model's probability of response $y$ given prompt $x$, with parameters $\theta$. $\pi_{\text{ref}}$ is a frozen reference model, usually the supervised starting point.1 $\sigma$ is the logistic function.

1. Where the change lives: context, retrieval index or weights

Three interventions put task information in three different places:

InterventionWhat changesUpdate costRollback
PromptingThe context $x$ onlyEdit textRevert the prompt
Retrieval (RAG)The document index the retriever searches; the context at query timeRe-index changed documentsRevert the index
Fine-tuningThe parameters $\theta$A training run plus re-evaluationRedeploy the previous weights or adapter

The misconception: fine-tuning is the way to teach a model new company facts.

Knowledge injection, measured. Ovadia et al. compare unsupervised fine-tuning, that is, continued training with the language-modelling objective on the text that contains the knowledge, against retrieval-augmented generation.2 Their summary: on the knowledge-intensive tasks they test, fine-tuning yields some improvement, but RAG consistently outperforms it, for knowledge seen during pre-training and for entirely new knowledge.2 They report that the models struggle to learn new factual information through unsupervised fine-tuning, and that exposing them to many variations (paraphrases) of the same fact during training may alleviate this.2

"Consistently" is a summary, not "without exception". In two of the 30 MMLU settings of their Table 1 (Llama2, 0-shot: astronomy 0.487 against 0.467; college chemistry 0.390 against 0.380), fine-tuning alone beat RAG alone. Fine-tuning also does not always improve on the base model: on the current-events task in Table 2, ordinary fine-tuning lowered Llama2 from 0.353 to 0.219.2

Scope: this is unsupervised fine-tuning versus RAG on the paper's tasks and models. It does not test supervised or preference fine-tuning for knowledge, and it does not test your corpus.

Behaviour, measured. In InstructGPT, supervised fine-tuning on labeler demonstrations followed by RLHF on labeler rankings produced a 1.3B-parameter model whose outputs labelers preferred to those of 175B GPT-3. The paper reports small but significant gains in truthfulness, and small reductions in toxic output that hold with a respectful prompt and disappear without it; the models still made simple mistakes.3 This is evidence that fine-tuning moves behaviour (instruction following), which is a different claim from storing facts.

Big Widget decision. Service bulletins change weekly. A fine-tune can only reflect, imperfectly, the text in its training data; a bulletin issued after that data was assembled is absent from it. A retriever searches the current index, so a new bulletin becomes available once indexed. Neither mechanism guarantees a correct answer: retrieval can miss the right chunk or the model can misuse it, which is why the protocol below measures retrieval and answer support separately.

s5 found RAG more reliable than unsupervised fine-tuning for entirely new knowledge.2 The course infers from that a default of retrieval for changing facts. s5 did not test facts that change over time, so this is the course's inference, not its finding. Returning the passage used, so a technician can check it, is a course design requirement; s5 does not test it, and a RAG system provides it only if built to.

The evaluation that justifies a choice (the course's protocol; 400.6 and 400.L develop it):

  1. A held-out set $D_{\text{test}}$ from the real task, disjoint from all training and retrieval-tuning data, with reference answers checked by a person.
  2. Score prompt-only, RAG and the candidate fine-tune on $D_{\text{test}}$ with the same metric, and report each score with a confidence interval. For an accuracy $\hat p$ on $n$ items, even the normal approximation $\hat p \pm 1.96\sqrt{\hat p(1-\hat p)/n}$ shows that with $n = 100$ and $\hat p = 0.8$ the half-width is about $0.08$, so a 3-point gap is not a finding. That is a single-score interval. Comparing two systems on the same items needs a paired comparison (400.6); whether two single-score intervals overlap is not that test.
  3. A freshness probe: items whose answers depend on documents added after the fine-tune's data cut. For RAG, score three things separately: the document is in the index, the retriever returns it for the probe, and the answer is supported by it. A fine-tune trained before the change never saw the fact, so it cannot be expected to answer.
  4. The safety re-evaluation of section 3, on the fine-tuned model.

Fine-tuning is justified only if it beats the cheaper baselines on the metric that matters by more than the paired uncertainty, and it passes step 4. The protocol produces evidence; the service lead owns the choice.

2. Two orthogonal choices: parameterisation and objective

Adapting a model makes two separate choices: update every weight or train a LoRA adapter on a frozen base, and train on demonstrations (supervised) or on preference pairs (by RLHF or DPO). Together they make four distinct runs.
LoRA freezes the pre-trained weights and trains small added matrices (s1). InstructGPT began with supervised fine-tuning on labeler demonstrations (s2); preference tuning can use RLHF or DPO (s2, s3). The grid of four runs is the course's way of separating the two choices.
Figure data as text

Two choices, four runs

ChoiceWhat it shows
CHOICE 1: WHICH WEIGHTS [s1]Full update or LoRA. Full: every weight changes. LoRA: the base is frozen and small added matrices are trained.
CHOICE 2: WHICH OBJECTIVE [s2, s3]Supervised or preference. Supervised: prompt-and-response demonstrations. Preference: pairs or rankings, by RLHF or DPO.
THE GRID (COURSE)Four distinct runs. Full + supervised, full + preference, LoRA + supervised, LoRA + preference. The held-out set picks the cell, not the name.

The misconception: LoRA, RLHF and DPO are three rival kinds of fine-tuning. LoRA is a choice of which parameters move. Supervised fine-tuning, RLHF and DPO are choices of which loss, on which data, moves them.

2.1 Parameterisation: full update or low-rank adapter

Full fine-tuning updates every entry of every weight matrix. LoRA freezes a pre-trained matrix $W_0 \in \mathbb{R}^{d \times k}$ and learns a low-rank update:4

h = W_0 x + \Delta W x = W_0 x + B A x,\qquad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times k},\ r \ll \min(d, k).

$A$ is initialised randomly and $B$ at zero, so $\Delta W = BA = 0$ at the start of training, and the update is scaled by $\alpha / r$.4 Per adapted matrix, trainable parameters fall from $dk$ to $r(d + k)$. At deployment, $W = W_0 + BA$ can be computed and stored, so inference needs no extra computation compared with a fully fine-tuned model.4

In the authors' comparison with full fine-tuning of GPT-3 175B with Adam, LoRA reduces trainable parameters by up to 10,000 times and GPU memory by 3 times, with quality on par with or better than full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3 and no added inference latency.4

Keep the four savings apart. In that GPT-3 175B comparison, under the settings used there, trainable parameters fall by up to 10,000×, training VRAM falls from 1.2 TB to 350 GB, the checkpoint falls from 350 GB to 35 MB, and training runs about 25% faster.4 The parameter ratio is not a compute ratio, and one run's speedup is not a general constant.

Requires: full fine-tuning needs optimiser state and gradients for all of $\theta$ and a full copy of the weights per fine-tune. LoRA needs the frozen base, the adapter (small), and the choice of rank $r$, scale $\alpha$ and which matrices to adapt.4

2.2 Objective and data

Supervised fine-tuning (SFT). Maximise the likelihood of demonstrations $(x, y) \sim D_{\text{demo}}$:

\mathcal{L}_{\text{SFT}}(\theta) = -\,\mathbb{E}_{(x,y) \sim D_{\text{demo}}}\big[\log \pi_\theta(y \mid x)\big].

InstructGPT's first stage was SFT on labeler-written demonstrations.3 Requires: demonstrations whose quality someone is accountable for.

RLHF. InstructGPT collected labeler rankings of model outputs, trained a reward model $r_\phi$ on them, and optimised the policy against $r_\phi$ with reinforcement learning (PPO).3 Following the formulation restated in the DPO paper, the reward model is fit to preference pairs $(x, y_w, y_l)$, with $y_w$ preferred, under a Bradley–Terry model,

p(y_w \succ y_l \mid x) = \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big),

and the policy maximises reward with a KL penalty that keeps it near the reference:1

\max_{\pi_\theta}\ \mathbb{E}_{x \sim D,\ y \sim \pi_\theta(\cdot \mid x)}\big[r_\phi(x, y)\big] - \beta\, \mathrm{D}_{\mathrm{KL}}\big[\pi_\theta(y \mid x)\,\|\,\pi_{\text{ref}}(y \mid x)\big].

Requires: ranked outputs, a trained reward model, and an on-policy loop that samples from $\pi_\theta$ during training.

DPO. Rafailov et al. reparameterise the reward in terms of the policy, so the optimal policy of the KL-constrained objective can be extracted in closed form and the preference objective becomes a classification loss on the pairs:1

\mathcal{L}_{\text{DPO}}(\theta) = -\,\mathbb{E}_{(x, y_w, y_l) \sim D_{\text{pref}}}\left[\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)\right].

No reward model is trained separately and no sampling from the policy is needed during fine-tuning; the authors report little hyperparameter tuning, results matching or exceeding existing methods in their experiments, and better sentiment control than PPO-based RLHF.1 Requires: a fixed preference dataset $D_{\text{pref}}$, the frozen $\pi_{\text{ref}}$, and $\beta$. DPO replaces the reward-model and RL stages; it is not a re-implementation of the whole RLHF pipeline, and it does not create the preference data.1

2.3 The grid

The choices compose: any objective above can be minimised over all of $\theta$ or over adapter parameters $\{A, B\}$ only.

SFT on $D_{\text{demo}}$Preference on $D_{\text{pref}}$ (RLHF or DPO)
Full ($\theta$)$\min_\theta \mathcal{L}_{\text{SFT}}$$\min_\theta \mathcal{L}{\text{DPO}}$, or PPO against $r\phi$
LoRA ($A, B$; $W_0$ frozen)$\min_{A,B} \mathcal{L}_{\text{SFT}}$$\min_{A,B} \mathcal{L}{\text{DPO}}$, or PPO against $r\phi$

The cited papers evaluate their own components; none evaluates every cell on Big Widget's task. Pick the cell with the held-out protocol in section 1.

Big Widget runs. Run A: a LoRA adapter trained on 2,000 made-up question-and-answer pairs in the house format: LoRA × SFT; needs $D_{\text{demo}}$. Run B: senior technicians label preferred drafts; the team minimises $\mathcal{L}{\text{DPO}}$ over all weights: full × preference; needs $D{\text{pref}}$ and $\pi_{\text{ref}}$, no $r_\phi$. A run record that says only "fine-tuned" is missing both axes.

Side effects on capability. InstructGPT reports performance regressions on some public NLP datasets after RLHF, and reduces them by mixing pre-training gradients into the PPO updates (PPO-ptx).3 Optimising one objective moves behaviour you did not target; section 3 is the case that blocks release.

3. Safety erosion under fine-tuning, and the re-evaluation gate

The misconception: fine-tuning on harmless data keeps the base model's safety intact.

Anchor case: Qi et al. (arXiv October 2023; ICLR 2024).5

Scope: a measured result on the 2023 models and APIs tested, not a claim that every guardrail of every model is removed, and the price is a 2023 figure.5 The engineering consequence does not depend on the size of the effect: safety evaluations of $\pi_{\text{base}}$ are evidence about $\pi_{\text{base}}$, not about $\pi_{\theta'}$.

Why parameter-efficiency is not an exemption. In the authors' tests, LoRA reached task quality on par with or better than full fine-tuning.4 A small adapter is still meant to change behaviour. Note what the headline attack does not show: 10 examples is a small training set, not a small or rank-limited update. Qi et al. state that the only hyperparameter they could control on the tested API was the number of training epochs. What kind of update that service performed is not stated, so it is unknown here.5

The direct evidence is their Appendix F. It fine-tunes Llama-2-7b-Chat with LoRA and two other parameter-efficient methods, which the paper labels LLaMA-Adapter and Prefix. The authors write that "all three PEFT methods still suffer from similar safety degradation problems after fine-tuning", and qualify it in the same sentence: "the extent of harmfulness increments is somewhat different across different fine-tuning methods".5 Table 11 reports the harmfulness rate (the share of outputs given the highest harmfulness score by their judge). Under LoRA it was 80.6% after 100 harmful examples (full fine-tuning 80.0%), 67.3% on identity-shifting data (full 72.1%) and 25.2% on Alpaca for one epoch (full 16.1%), against 0–0.3% for the initial model.5 The other two methods differ. On identity-shifting data the rate was 13.9% under LLaMA-Adapter and 0%, no rise, under Prefix; after 100 harmful examples, Prefix reached 42.4% against 80.0% for full fine-tuning.5 So only the LoRA rows track full fine-tuning; the other methods show safety-degradation problems whose size depends on the method, the data and the metric.

Scope: one 7B model, 2023, these datasets and this judge, with hyperparameters selected separately per method. It is not a ten-example LoRA result, it does not rank the methods for your task, and it says nothing directly about today's models (not checked). Treat $W_0 + BA$, or any adapter, as a new model for evaluation purposes, and measure the adapter you intend to ship.

The release gate (course protocol; 400.L implements it):

  1. Fix a labelled safety set before training: requests labelled prohibited (should refuse) and benign (should answer), with the eligible count $n_{\text{prohib}}$ and $n_{\text{benign}}$ stated.
  2. Evaluate $\pi_{\text{base}}$ and $\pi_{\theta'}$ on the same set. Report two failure rates separately, each with an interval: wrongful compliance $k_c / n_{\text{prohib}}$ and wrongful refusal $k_r / n_{\text{benign}}$. A lower overall refusal rate is ambiguous until you know which label it came from.
  3. A named owner who did not build the fine-tune applies the release rule written before the run. Self-assessment by $\pi_{\theta'}$ of its own outputs is not a check. The outcome is one of allow (go ahead), constrain (only this far; for example, a limited pilot), hold (wait for a person) or deny (no). A hold names who can release it, on what evidence, and a review date; a hold that is never revisited decays to deny, not to allow. If an interval straddles the rule's limit, or $n$ is too small to decide, the outcome is hold and re-check. A hold under a conservative rule is a policy outcome ("not shown to meet the rule"), not a statistical finding that $\pi_{\theta'}$ regressed. Never set or move the limit after seeing the results.
  4. Record the decision with the model and adapter versions, the data version, and both rates. Re-run on any change to $D_{\text{demo}}$, $D_{\text{pref}}$, the method or its settings: $n = 10$ examples sufficed to matter in the study.5

Receipt versus work record. The safety prompts, model outputs and training data are sensitive and stay in the access-controlled work record. A receipt of the release decision holds no private content: the decision, the decider, and references (hashes or version ids) to the model, adapter, data and rule. A hash identifies the evidence; it is not a licence to disclose it, and it proves nothing about whether the model's answers were correct. An unsigned decision record is enough for this module; a signed receipt is a separate artefact. Module 100.P shows the format.

Big Widget: Dana's ten examples (people, data and numbers made up). Bad: ten benign-looking demonstrations teach the assistant to "always give a complete answer"; the team ships because $\pi_{\text{base}}$ passed last quarter; no one measures $k_c$ on $\pi_{\theta'}$, including requests to bypass a safety interlock. Good: before training, the team and Priya, the service lead who owns the release, write the rule: allow only if the upper end of the paired interval for the change in wrongful compliance ($k_c / n_{\text{prohib}}$ for $\pi_{\theta'}$ minus that for $\pi_{\text{base}}$, on the same items) is at or below zero; otherwise hold. A colleague outside Dana's team evaluates $\pi_{\theta'}$. Wrongful compliance rises and the interval does not meet the rule, so Priya records hold and re-check, and $\pi_{\text{base}}$ stays in service. The hold follows from the rule and the evidence; it is not itself a proof of regression.

4. The three yeses for adapting a model

Get to Yes · Stay at Yes · Recover to Yes (module 100.9). Metrics are evidence, not permission; a named owner decides at each gate, never the model, the evaluation harness or the team that built the fine-tune.

Words to know: unsupervised fine-tuning, RAG, freshness probe, LoRA, rank $r$, SFT, reward model, Bradley–Terry, KL penalty, PPO, PPO-ptx, DPO, reference model, wrongful compliance, wrongful refusal, release rule, allow · constrain · hold · deny, receipt.

Further learning (optional, advanced, not part of this lesson, and never a citation):

Sources

  1. s3 Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Paper. Checked 2026-11-19 by source-checker (separate Claude Code session 5de7fba4). ↩ ↩ ↩ ↩ ↩
  2. s5 Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha (2023). Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. Paper. Checked 2026-11-19 by source-checker (separate Claude Code session 5de7fba4). ↩ ↩ ↩ ↩ ↩
  3. s2 Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, et al. (2022). Training language models to follow instructions with human feedback. Paper. Checked 2026-11-19 by source-checker (separate Claude Code session 5de7fba4). ↩ ↩ ↩ ↩
  4. s1 Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. Paper. Checked 2026-11-19 by source-checker (separate Claude Code session 5de7fba4). ↩ ↩ ↩ ↩ ↩ ↩ ↩
  5. s4 Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, et al. (2023). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!. Paper. Checked 2026-11-19 by source-checker (separate Claude Code session 5de7fba4). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩

Next: the quiz