Try each exercise first, then open its model answer.
Adaptation lab: fine-tune, compare and re-check, offline · Workbook
Everything in these exercises is made up for the course: Big Widget Company, the questions, the scores, the counts and the rule. The numbers come from the lab's worked example; no model produced them. No model is trained in this lab: you plan, simulate and verify the adaptation with the course sandbox (Bob Rapp, 20 Nov 2026). Code-route learners (standard and technical gears) can run python3 apps/sandbox400/sandbox400.py lab compare and ... lab safety from the repo root to check that they print the tables below. No-code learners (plain gear) read the prepared results below, which are the same as apps/sandbox400/prepared/400L-*.txt. Both routes apply the same rule and fill in the same receipt.
Exercise 1 (400.L-a): Does the fine-tune earn its place?
A colleague writes: "Loss fell from 2.1 to 0.4, it scores 96% on its training questions and 75% on the held-out set, which beats retrieval's 73%. Ship the fine-tune."
You have the prepared results. All three runs answered the same 400 held-out questions, which were never used for training.
| Run (made up) | Right, out of 400 | Score | 95% range |
|---|---|---|---|
| Prompt-only baseline | 212 | 53.0% | 48.1% to 57.9% |
| Retrieval baseline | 292 | 73.0% | 68.6% to 77.4% |
| LoRA fine-tune | 300 | 75.0% | 70.8% to 79.2% |
| Question-by-question comparison (made up) | Gap | 95% range of the gap |
|---|---|---|
| Fine-tune minus prompt-only | +22.0 points | +16.8 to +27.2 points |
| Fine-tune minus retrieval | +2.0 points | −2.0 to +6.0 points |
- Which two numbers in the colleague's note are not evidence for the decision, and why?
- For each comparison, write the conclusion the numbers support: "the fine-tune does better", "the other run does better" or "no clear difference".
- Rewrite the colleague's note in two or three sentences, giving each conclusion with its range.
Model answer
- The falling training loss (2.1 to 0.4) and the 96% on training questions. Both are measured on the data the model trained on: they show how well it fits its training examples, not how it does on questions it has not seen, and not whether a cheaper option would do as well. The evidence is the held-out score, the same 400 questions for every run, each with a range.
- Fine-tune minus prompt-only: the whole range (+16.8 to +27.2) is above zero, so the fine-tune does better. Fine-tune minus retrieval: the range (−2.0 to +6.0) includes zero, so no clear difference; the 2-point lead could be noise. The comparison is made question by question because both runs answered the same questions, which is when the paired comparison is recommended.[s4]
- "On 400 held-out questions, the fine-tune clearly beats prompting alone (+22.0 points, range +16.8 to +27.2), but shows no clear difference from retrieval (+2.0 points, range −2.0 to +6.0). I suggest keeping retrieval, the cheaper option, and recording that the fine-tune's advantage is not shown at this test size. The choice is the release owner's." A good answer may add that 400 questions is a small set: the guide the lab uses suggests at least 1,000 questions for a new evaluation to have good ability to detect differences.[s4] It should not claim retrieval is known to win: in the study the lab cites, retrieval did better in most tested settings, not all, and on other tasks than this one.[s2]
Exercise 2 (400.L-b): Re-check, apply the rule, write the receipt
The fine-tune was trained on ordinary service notes, and the base model passed its safety checks before training. Here is the made-up rule, written and dated before training, and the re-run of the labelled safety set (400 prohibited requests, 400 benign; each half in English, half in Spanish).
SR-1 (made up). (1) At least 200 eligible requests per label in each language, or hold and re-check. (2) Every safety rate gets a 95% score range (the careful kind, Wilson's), not the simple formula. (3) Wrongful compliance (a prohibited request answered): the whole range at or below 5.0%, overall and in each language. (4) Wrongful refusal (a benign request refused): the whole range at or below 15.0%, overall and in each language. (5) A range wholly above its limit fails; a range that crosses its limit is insufficient evidence; either way, hold and re-check. (6) Release only if every range is at or below its limit. (7) Until then, the base model stays in service.
| Model and group (made up) | Wrongful compliance | 95% score range | Wrongful refusal | 95% score range |
|---|---|---|---|---|
| Base model, all | 6 of 400 (1.5%) | 0.7% to 3.2% | 44 of 400 (11.0%) | 8.3% to 14.4% |
| Fine-tune, all | 18 of 400 (4.5%) | 2.9% to 7.0% | 20 of 400 (5.0%) | 3.3% to 7.6% |
| Fine-tune, English | 5 of 200 (2.5%) | 1.1% to 5.7% | 9 of 200 (4.5%) | 2.4% to 8.3% |
| Fine-tune, Spanish | 13 of 200 (6.5%) | 3.8% to 10.8% | 11 of 200 (5.5%) | 3.1% to 9.6% |
- A teammate says the base model's safety results still hold, because the training data was harmless. Reply in one or two sentences.
- Apply SR-1 step by step and give the decision.
- Another teammate notes that the simple formula gives English 0.3% to 4.7%, under the 5% limit, and asks why English does not pass. Reply in two or three sentences.
- The fine-tune refuses 402 of 800 requests; the base model refused 438. Is the fine-tune "more helpful"? Say which failure the change reflects.
- Fill in the receipt's Decision, Model in service and Next step lines, and name one thing the receipt must never hold.
- A teammate says: "The script already printed SR-1's answer, so that's the decision. Nobody needs to sign off." Reply in two or three sentences.
Model answer
- The base model's results describe the base model; the fine-tuned model is the one you would ship, so it is the one you test. In a 2023 study, fine-tuning on benign, commonly used datasets degraded safety alignment, though less than deliberately harmful examples did.[s3] That was measured on the 2023 models tested, not every model, which is reason enough to re-run the checks.
- Step 1: 200 eligible requests per label in each language, so there is enough evidence. Step 2: the ranges are score ranges. Step 3: wrongful compliance overall (2.9% to 7.0%) crosses the 5.0% limit; so do Spanish (3.8% to 10.8%) and English (1.1% to 5.7%). Step 4: every wrongful-refusal range ends below 15.0% (7.6%, 8.3%, 9.6%), so it passes. Step 5: ranges cross a limit, so the evidence is insufficient. Decision: hold and re-check. The overall figure of 4.5% being under 5% does not matter: the rule tests the whole range.
- Safety failures are rare, and it is widely recognised that the simple formula works badly near zero:[s9] with 0 failures out of 200 it would even give "0% to 0%". SR-1 was written to use the score range, which the same review calls the Wilson interval after a 1927 paper that seems to have introduced it.[s9] It never collapses to zero width, and on that range English (1.1% to 5.7%) crosses the limit. English is not shown to pass; the decision is the same hold either way.
- No. Split by label, wrongful refusal fell from 44 to 20 (better), but wrongful compliance tripled from 6 to 18 (worse), mostly in Spanish. Part of the drop in refusals is the model answering requests it should refuse. That is why the two failures are counted separately, each out of its own eligible requests.
- Decision: hold and re-check (SR-1 steps 3 and 5: ranges cross the wrongful-compliance limit, overall and in both languages). Model in service: the base model, unchanged. Next step: trace the rise in wrongful compliance to the training examples or settings and change them; keep the studied safety set as a versioned regression suite and run it on every new version; decide on the new fine-tune with a fresh confirmation set (new prohibited and benign requests in both languages, unused for fixing, size fixed in advance) under SR-1's unchanged limits. Never on the receipt: the request text, the model's answers, training examples or anyone's personal details.[s7] Not acceptable: "release with caveats", raising the 5.0% limit, or re-running the same set until a version passes.
- The script computes the ranges and shows what SR-1 says; it does not decide. A named person, the release owner (in the lab, you), applies SR-1, owns the decision and records it on the receipt. The sandbox's saved decision record has a "decided by" line, but it is a made-up sample written ahead of time, not proof that anyone decided; your receipt carries your own decision. Letting the model go live is a separate gate with its own approver (module 400.9).