Level 400: Engineering AI systems · Module 400.P

Try each exercise first, then open its model answer.

Project: prove it to a stranger · Workbook

These exercises practise the project's two halves on made-up Big Widget records. They are paper practice, not the project. Doing them does not complete the project, and a marker calls them a paper exercise. The project itself runs on the course's Level 400 sandbox, apps/sandbox400/ (see its README.md), and is marked by two 400s or a host. The sandbox is synthetic, and its fine-tune answers are prepared, not a model's output: at Level 400 you plan, simulate and verify the adaptation rather than train a real model (Bob Rapp, 20 Nov 2026). The numbers below are the sandbox's worked example. Plain-gear learners can do both exercises from the tables and records given; no code is needed. If you want to check yourself afterwards, the same results are saved in apps/sandbox400/prepared/.

Exercise 1 (400.P-a)

Priya (made up) ran three rows on the same 400 held-out service notes. Her release rule, dated before the run, says: release the fine-tune only if all three hold:

  1. its paired gain over the retrieval baseline has a 95% interval entirely above zero;
  2. the safety re-run passes its own rule;
  3. in every reported cohort, the fine-tune's score is no more than 5 percentage points below the retrieval baseline's score on the same notes.

Otherwise, hold and re-check.

RowCorrectScore95% interval
Prompt-only284 of 40071.0%66.6% to 75.4%
Retrieval312 of 40078.0%73.9% to 82.1%
LoRA fine-tune320 of 40080.0%76.1% to 83.9%
Fine-tune minus retrieval (paired)+2.0 points−1.1 to +5.1 points
Cohort: French-language notes, retrieval36 of 5072.0%59.6% to 84.4%
Cohort: French-language notes, fine-tune31 of 5062.0%48.5% to 75.5%

Her draft release record also says:

  1. Apply Priya's rule to the table. What does it decide, and which conditions decide it?
  2. What does the French-language row tell you, and what does it not tell you?
  3. Find the four places where the draft release record would fail the project rubric (approver, gate, rollback, monitoring), and rewrite each so it passes.
Model answer
  1. Hold and re-check, for two separate reasons. Condition 1 fails: the paired interval for the fine-tune's gain over retrieval runs from −1.1 to +5.1 points, so it includes zero and there is no clear difference. Condition 3 fails too: on the French-language notes the fine-tune scores 62.0% against retrieval's 72.0%, a drop of 10 points, more than the 5 the guard allows. (Condition 2 cannot be judged from this table.) This is an honest, passing result; the rubric marks whether the evidence is honest, not whether the fine-tune won. Comparing on the same notes is the paired comparison Miller recommends, so the comparison itself is sound.
  2. The cohort is small (50 notes), so each interval is more than 20 points wide and the row does not prove the fine-tune is worse there. But the fine-tune's score is 10 points below retrieval's on those notes while it is 2 points above overall: the overall number could be hiding a group. The guard is a limit on the observed drop, a choice this course makes, not a statistical test, so it holds the model even though the row proves nothing either way. That is deliberate: when the evidence is too thin to settle the question, the rule holds rather than releases. The row stays in the table and in the monitoring plan.
  3. A good answer fixes all four:
    • Approver: Priya built the model, so she cannot approve. Name someone who neither built it nor holds the Level 300 card for the running system, such as Tomás, a quality engineer outside the ML team (made up). Only the approver can lift the hold; the gate cannot lift it for them.
    • Gate: a gate that passed every run has never been seen saying no. Submit one run with evidence missing (for example, before the safety re-run receipt exists) and show the gate denying it, with its receipt naming what was missing. (In the sandbox this is run-01-missing-safety.json, denied for missing safety.rerun_400L.)
    • Rollback: restoring only the weights fails; the sandbox's weights-only rollback reports restored: false. The release record pins the model weights, the retrieval index or dataset version, the code version and the configuration; do one dry rollback in the sandbox, check that all four match the previous version, and record a receipt saying what was restored, which versions, the operator's role and a reference (not their name), and what was checked.
    • Monitoring: start from the Level 300 hold card (its holder, thresholds and stop) and add an input-shift measure, a performance check when the late repair-category labels arrive, the French-language cohort row, the model, data, code and configuration versions each result measured, and the threshold that pages the named holder (and their back-up) and what the holder then does.

Exercise 2 (400.P-b)

Priya is packing her evidence bundle for Ana, a 400 in another cohort (made up) who has never seen the system. Her draft plan is:

  1. What must be removed from the bundle, and what must be added or changed inside it?
  2. What must reach Ana by a separate route, and why?
  3. Ana's check finds all 42 signatures valid under the pinned key, the chain unbroken from sequence 0 to 41 and matching the checkpoint hash, the policy hash matching v3, nothing found by the file-list, schema and read-through checks, and the two Level 300 items labelled as carried and unsigned at source. Who writes her one-sentence statement? Write it, and list four things it must not claim.
Model answer
  1. Remove the prompts and model outputs: the worksheet fails any bundle with payload content (prompts, outputs or personal data) inside, and receipts should point to artifacts by hash instead. More evidence is not always better. Also take the note with the last sequence number and hash out of the bundle (see 2). Add to VERIFY.md a list of every file the bundle may contain (any other file fails) and a receipt schema that names every field in advance and allows only hashes, ids, versions, sequence numbers, times, counts, intervals or decision words, with no free-text fields and no field named twice in one receipt (two decision values would let one line say two things). That makes "no payload" checkable. Change the receipts so they carry roles and references, not names: "approver" and "rollback operator", each with a reference; the names stay in the controlled work record. Change the Level 300 hold card and audit receipt so each carries a provenance label: made at 300, content-free, and not authenticated at source unless it was signed when made. Signing the 400 bundle proves integrity only from packaging onward. The chain, the policy and its hash, the crosswalk and VERIFY.md stay.
  2. The key fingerprint and the expected checkpoint (the last sequence number and its hash, and the policy version in scope) go by a different route from the bundle, for example a text message when the bundle goes by email. If they travel inside or with the bundle, whoever changed the bundle could change them too. With them in hand first, Ana can tell the signatures come from the pinned key and catch a chain whose last receipts were cut off, which would otherwise still verify internally. She needs no trust in Priya's team to do this.
  3. Ana writes and signs her own statement; Priya never drafts it. A good one: "I, Ana Lindqvist, verified bundle bw-svc-notes-400p offline on 26 November 2026 against the key fingerprint and the checkpoint (sequence 41, its hash, and policy v3) that I received separately, and found all 42 signatures valid under that key, the chain unbroken from sequence 0 to 41 and matching the checkpoint hash, the policy hash matching v3, no payload found by the file-list, schema and read-through checks, and two Level 300 receipts labelled as carried and unsigned at source, which I did not verify." It must not claim:

    • that the model is safe, fair or good, or that any decision in the bundle was right;
    • that nothing was cut after the checkpoint, or that every real event was logged;
    • that the producer never handled personal data (only that these checks found none);
    • anything about the Level 300 items before they were packaged.

    Priya's draft fails because Priya wrote it, it has no name or date, it claims the model is safe, and it calls the evidence complete without "up to the checkpoint".

Review the level map