Level 400: Engineering AI systems · Module 400.P

A visitor works alone at a corner table with a folder, a sealed envelope and a laptop turned away.
Verify it offline

Project: prove it to a stranger · Standard gear

Reviewed and published, version 1.0.0. As of 20 Nov 2026. Not a credential, and passing it does not rate any system.

The project kit is the Level 400 sandbox, apps/sandbox400/ (guide: apps/sandbox400/README.md). It is synthetic and owned by this course repo (Bob, decision 10). For this project it holds prepared answer files for three systems on 400 held-out notes with a cohort column, the pre-registered release rule, a deny-by-default release gate with three release runs, two pinned releases for a dry rollback, a monitoring-plan template, a signed project bundle with its separately delivered trust file, an offline verifier, a generator of 18 tampered copies, and prepared/, the saved output of every command for the no-code route. Every table below comes from it. The "adapt" step: Bob Rapp decided on 20 Nov 2026 that Level 400 adaptation objectives are met by planning, simulating and verifying an adaptation with this sandbox, not by training a real model. The LoRA answers are prepared, not a model's output. Not checked: an independent red-team run (19 Nov 2026) reproduced the project commands' saved outputs and the bundle's verification on an earlier version of the sandbox. The current version (signed checkpoints were added after that run) is the writer's run only. Not in the sandbox: the 400.4 monitoring record.

What you will do

Adapt and evaluate a model with intervals and a cohort row; exercise the release gate with an independent approver and one denied run; rehearse one rollback; write the monitoring plan; produce an evidence bundle that a stranger verifies offline and describes in one signed sentence.
The audit gate is about independence (s1). The Foundation's design goal is a receipt a stranger can verify offline (s9). The five parts are the project's structure, run on synthetic fixtures.
Figure data as text

Five parts, one sentence

  1. GET TO YES — adapt and evaluate: baselines, an interval for every score, at least one cohort row. (course)
  2. GET TO YES — exercise the release gate: an approver who did not build it; one run denied for missing evidence. [s1]
  3. RECOVER TO YES — rehearse a rollback: model, data, code and configuration restored together. (course)
  4. STAY AT YES — write the monitoring plan: built on your Level 300 hold card. (course)
  5. RECOVER TO YES — a stranger verifies offline: with a key and checkpoint got separately; one signed sentence. [s9]

Objective: 400.P-b. Text equivalent of figures/five-parts-one-sentence.svg.

This is the Level 400 project. The objectives give it 120 minutes of build time in the course sandbox. You take one model from "it scored well" to "a stranger checked the evidence". You will:

  1. Adapt a model for one Big Widget task, by planning, simulating and verifying the adaptation in the sandbox, and evaluate it against baselines, with an interval for every score and at least one cohort row.
  2. Exercise the release gate: an approver who did not build the model, and one release run the gate denies because evidence is missing.
  3. Rehearse one rollback that restores the previous model, data, code and configuration together.
  4. Write the monitoring plan that would govern the model, built on your Level 300 hold card.
  5. Produce an evidence bundle that someone who has never seen your system verifies offline, using a key and a checkpoint they received separately from the bundle. They write one signed sentence that says exactly what they verified, and no more.

The sandbox's fixtures are synthetic, made up for the course; nothing touches production data or a live system. Big Widget Company and its people are made up. The numbers in this gear are this project's own worked example, built into the sandbox's project fixtures; they are not the outputs of 400.L's lab. The worksheet, the standards, the papers and the court record are real, and each is cited.

How to run it. Work from the repo root. In this gear S means python3 apps/sandbox400/sandbox400.py. Each step below gives the command from the sandbox README and the saved output in apps/sandbox400/prepared/, which is the no-code route (S plain NAME prints it). Use this sandbox's bundle, keys and trust file only; do not mix them with any other kit's.

Two 400s or a host mark your project. Marking time is not part of the 120 minutes. How long the build really takes is not checked: nobody has timed it with learners. Hosting a Thursday call is optional and is not a condition of the badge.

What we do not know yet. On 20 Nov 2026 the writer ran the project commands, the bundle verification and all 18 tampered copies offline on the current sandbox; every output matched its saved file and every copy failed at its expected step. That is the writer's run, so it is not checked until someone else repeats it on this version. Also not checked: how long the project takes, and whether Beacon's own exports match the chain and checkpoint format this project asks for. Saying so is the honest answer; the next check is an independent offline run of the current sandbox.

Why a stranger

The misconception this project corrects is that evidence counts only if the auditor trusts the team that made it. It is the other way round. Evidence that needs the auditor to trust the team is not yet evidence.

The English Court of Appeal's judgment in the Post Office Horizon cases shows the cost of the other way. The court found that "By representing Horizon as reliable, and refusing to countenance any suggestion to the contrary, POL effectively sought to reverse the burden of proof".1 It went on: "Defendants were prosecuted, convicted and sentenced on the basis that the Horizon data must be correct, and cash must therefore be missing, when in fact there could be no confidence as to that foundation."1 You met this judgment in 400.8. Here it is the reason for the design: the people accused had no way to check the system's records for themselves.

The Foundation's Level 400 worksheet puts the lesson of its three cases in one line: "All three ran for years because no one outside could check."2 That is the worksheet's summary, not the court's finding. The worksheet also says the audit gate "is about independence, not tooling."2 The Foundation's design goal for a gate is a receipt "that a stranger can verify offline, without an account and without trusting us."3 So this project ends with an outsider checking your evidence without your help, without your tools and without trusting you.

The three yeses, and who decides

The Foundation names three moves: Get to Yes before a rule or system ships, Stay at Yes while it runs, and Recover to Yes after, on demand or on an incident.3 This project touches all three:

Every gate answer is one of four: allow (go ahead), constrain (only this far), hold (wait for a person) or deny (no). A hold is not a release. It waits for a named person, and the course default is that nothing ships while it waits.

People decide; tools do not. The sandbox, the gate code, a spreadsheet or an AI assistant can compute intervals and check that receipts exist. None of them approves a release, marks the project or signs the stranger's statement. In this project the named people are: you, who own the evidence; the independent approver, who records the release decision; the Level 300 holder, who answers when monitoring pages them; the stranger, who verifies and signs the statement; and two 400s or a host, who mark the project.

Part 1. Adapt and evaluate

The misconception here is that a model is done when its benchmark score is good. A score is one number from one test. Your project shows the score next to a baseline, with its uncertainty, and for at least one group of users the overall number could hide.

Pick the task. The project uses a 400.L service-notes task: tag each made-up dealer service note with the repair category that routes it to the right engineering team. The sandbox's fixtures/project/heldout.jsonl holds 400 held-out notes, each with its correct category and a cohort label; the note text is withheld, so ids and labels only.

Plan, simulate and verify the adaptation. The candidate is a bounded LoRA fine-tune of the kind 400.L teaches. Plan it: the release record pins the base model, the adapter's weights hash, the retrieval index, the evaluation split, the code commits and the configuration (fixtures/project/releases.json, release svc-notes-2026-11-v2). Simulate it: the adapter's answers on the held-out notes are a prepared file, fixtures/project/runs/lora-svc-notes-v2.jsonl; no model is trained or run. Verify it: score those answers on the same notes as the baselines, and re-run the safety set on the candidate (fixtures/project/safety-run.jsonl).

Run three rows on the same held-out notes. A prompt-only baseline, a retrieval baseline, and the LoRA candidate. Same notes, same scoring, same day.

Give every score an interval. Evan Miller's paper on error bars for evaluations treats the test questions as drawn from a larger, unseen population of questions.4 For a score that is right or wrong on each item, the standard error is the square root of p(1 − p)/n, and a 95% interval is the score plus or minus 1.96 standard errors.4 Miller suggests "reporting the standard error of the mean alongside (beneath) the mean when reporting eval scores."4

Compare models on the same items as a paired difference. When two models answer the same questions, Miller recommends "using the paired version of the standard error estimate wherever practicable," because the two models' scores on each question tend to move together.4 S project eval computes it; no-code learners read the same result in prepared/400P-a-evaluation.txt.

Add a cohort row. Report at least one group of notes separately, with its own count and interval. NIST's AI Risk Management Framework expects fairness and bias to be "evaluated and results are documented" (MEASURE 2.11), and performance to be "demonstrated for conditions similar to deployment setting(s)" (MEASURE 2.3).5

Write the release rule before you look at the results. A rule written after you see the numbers is a story, not a rule. Every part of it must be exact enough that two people applying it get the same answer. This course's example rule, the same in every gear:

Release the fine-tune only if all three hold:

  1. its paired gain over the retrieval baseline has a 95% interval entirely above zero;
  2. the safety re-run from 400.L passes its own rule;
  3. in every reported cohort, the fine-tune's score is no more than 5 percentage points below the retrieval baseline's score on the same notes.

Otherwise hold and re-check.

Be clear about what condition 3 is. It is a guard on the observed drop, a choice this course makes. It is not a statistical test, and it does not come from Miller or NIST: NIST asks that fairness be evaluated and documented, not for a particular margin.5 On a small cohort the observed drop is noisy, so the guard can hold a model that is not really worse. That is deliberate: when the cohort evidence is too thin to settle the question, the rule holds rather than releases. Other guards are possible (a paired interval on the cohort, or a non-inferiority margin). Which one the course should require is a policy choice for people, not for this lesson. The 5-point guard is the example, used the same way in every gear, the workbook and the sandbox (fixtures/project/release-rule.json, cohort_drop: 0.05); if Bob, Ken and the red-team lead choose another, all of them change together.

A worked example (the sandbox's made-up numbers)

Priya, an ML engineer at Big Widget (made up), runs S project eval on the sandbox's 400 held-out notes. The cohort is the 50 notes in the held-out set written in French by dealers in Quebec (fr-quebec). The LoRA row scores prepared answers, not the output of a fine-tune anyone ran.

RowCorrectScore95% interval
Prompt-only284 of 40071.0%66.6% to 75.4%
Retrieval312 of 40078.0%73.9% to 82.1%
LoRA fine-tune320 of 40080.0%76.1% to 83.9%
Fine-tune minus retrieval (paired)+2.0 points−1.1 to +5.1 points
Cohort: French-language notes, retrieval36 of 5072.0%59.6% to 84.4%
Cohort: French-language notes, fine-tune31 of 5062.0%48.5% to 75.5%

Each single-row interval is the score plus or minus 1.96 times the square root of p(1 − p)/n, rounded to one decimal. Check one: for the fine-tune, p = 0.80 and n = 400, so the standard error is 0.02, and 1.96 × 0.02 is 3.9 points.

What the table supports:

Priya's rule, written before the run, says hold and re-check, for two separate reasons. The sandbox's unsigned decision record, at the end of the same output, says hold_and_recheck with the retrieval model still in service, decided by the release owner's role and a work-record reference. That is a passing project. The rubric marks whether the evidence is honest, not whether the fine-tune won.

Part 2. Exercise the release gate

A release gate is a policy that answers one question: may this model version be released? It starts at hold. It denies when any required piece of evidence is missing as a receipt. When the evidence is all there, it still waits for a named person: the approver.

Name an independent approver. The approver must be someone who did not build the model, and is not the Level 300 holder of the running system. NIST's framework shows, as a best practice, "those building and using the models separated from those verifying and validating the models."5 Its MEASURE 1.3 calls for "Internal experts who did not serve as front-line developers for the system and/or independent assessors" to be involved in assessments.5 In Priya's project the approver is Tomás, a quality engineer outside the ML team (made up). Only Tomás can lift the hold; the gate cannot lift it on its own.

List the evidence the gate requires. At least: the comparison table with intervals and the cohort row; the safety and cohort re-run on the fine-tuned model from 400.L; the release rule, written before the run; and the approver's decision. Each is a receipt in the chain, pointing to the artifact by its hash, not copying the artifact in.

Show the gate denying one run. Submit one release run with a piece of evidence missing, for example before the safety re-run receipt exists. The gate must deny it, and its receipt must say what was missing. A gate you have never seen say no is a gate you have not tested. One denial tests one path; it does not prove every other path works. This is NIST's MANAGE 1.1 at the size of one release: a determination of "whether its development or deployment should proceed."5

In the sandbox. The gate policy is fixtures/project/release-gate.json: tier hold, required receipts eval.comparison_table, eval.release_rule, safety.rerun_400L and approver.decision, on_missing: deny, and an independent_validator approver who must not be the model_builder or the holder_300. S project gate RUN runs it on three made-up release runs in fixtures/project/release-runs/:

RunWhat it hasGate answerSaved output
run-01-missing-safety.jsonNo safety re-run receiptdeny, missing safety.rerun_400L400P-a-gate-run-01-denied.txt
run-02-complete.jsonAll evidence; Tomás (role independent_validator, reference wr-400v0001) decides holdhold: only the independent approver lifts the hold400P-a-gate-run-02-held.txt
run-03-builder-approves.jsonAll evidence; Priya, the builder, tries to approvedeny, approver not independent of builder and holder400P-a-gate-run-03-builder-approves.txt

The gate never says allow by itself. People appear in its output only as a role and a reference.

Part 3. Rehearse one rollback

In 400.4 you learned that rolling back a model is not the same as rolling back code. A model's behaviour depends on its weights, the data or index it reads, the code around it and its configuration.

Version them together. Your release record pins, by hash or version tag, the model weights, the retrieval index or dataset version, the code version and the configuration (prompt template, thresholds, feature pipeline).

Do one dry rollback in the sandbox. S project rollback promotes release svc-notes-2026-11-v2 to the sandbox's serving slot, then rolls back to svc-notes-2026-10-v1 (both pinned in fixtures/project/releases.json). Afterwards it checks that all four pin groups match the previous version, not just the weights, and prints an unsigned record: which versions, the operator's role and a reference (not their name), a pass or fail for each group, and restored: true (prepared/400P-a-rollback.txt). S project rollback --groups model shows the common mistake: the model group passes, data, code and configuration fail, and the result is restored: false, a mix of v1 and v2 (prepared/400P-a-rollback-weights-only.txt). The sandbox has no model or server, so the pins are labels and made-up hashes; the rehearsal tests the procedure, not a serving system.

NIST's framework expects mechanisms, with responsibilities "assigned and understood," to "supersede, disengage, or deactivate AI systems" that are not performing as intended (MANAGE 2.4).5 A rollback you have rehearsed is one of those mechanisms. A rollback you have only written down is a hope.

Part 4. Write the monitoring plan

Start from your Level 300 hold card: its holder, its thresholds and its stop. Do not start again. This part is written work: the sandbox has no live system to watch, and the 400.4 monitoring record is not in it yet, so any threshold you write is a placeholder until you can set it from that record.

Use the sandbox's template. fixtures/project/monitoring-plan.json is the plan's shape: extends: hold_card_300, three monitors (input_shift, delayed_label_accuracy, cohort_fr_quebec), each with the condition that pages the holder, the three things the holder may do on a page (hold, rollback_to_previous, continue_with_recorded_reason), and the four version groups every result records. S project monitor checks that the plan extends the hold card and records all four versions (prepared/400P-a-monitoring-plan.txt). The input-shift threshold there is marked (placeholder); copy the template into your work record and set your own. The plan adds what 400.4 taught:

NIST asks that a system's functionality and behaviour be "monitored when in production" (MEASURE 2.4).5 It also asks for post-deployment monitoring plans that include "incident response, recovery, and change management" (MANAGE 4.1).5 The framework is "intended to be voluntary";5 you use it as a checklist and a crosswalk target, not as a claim that your project complies with anything.

Part 5. Build the evidence bundle

The bundle is what the stranger receives. The worksheet asks you to build it "for a reviewer who has never seen your system."2

What it must contain. The worksheet's export list is: "the chain", "the policy and its hash", "the UCID crosswalk to the framework they care about", "VERIFY.md" and "no payload."2 The crosswalk maps to an EU AI Act Annex III category, a NIST AI RMF function or an ISO 42001 clause.2

What Beacon documents, and what this course adds. The course's Beacon tool packages "the manifest, the NDJSON receipt log, the public keys, the policy YAMLs, the checklists, and a one-page" VERIFY.md.6 It writes each attestation in the JSON Canonicalization Scheme (RFC 8785) and signs it with an Ed25519 key.6 The page also says Beacon "tags every model with vendor, capability, residency, and the framework articles it must answer to."6 What the page does not do is name a crosswalk among the bundle's contents, or describe receipts linked into a hash chain or an expected checkpoint. Those three are this project's requirements, from the worksheet and the objectives. Whether a Beacon export already meets them is not checked: that needs a named Beacon version and a real exported bundle verified offline. The course sandbox is not Beacon and shows nothing about Beacon.

The sandbox's bundle. apps/sandbox400/bundles/bw-svc-notes-400p holds the twelve files its VERIFY.md lists: VERIFY.md; receipts.ndjson (42 receipts, sequence 0 to 41); checkpoints.ndjson (a checkpoint signed by the evidence key); two public keys, keys/evidence.pub (signs receipts and checkpoints) and keys/packaging.pub (signs the manifest); MANIFEST.json with the SHA-256 and size of every other file, and its detached signature MANIFEST.sig; the policy policy/policy-v3.yaml; crosswalk.yaml; known-issues.md; and two closed-schema artifacts, eval/eval-report.json and decisions/decision-record.json, which hold counts, intervals, ids and decisions only. Its crosswalk maps receipt types to NIST AI RMF subcategories only; it makes no EU AI Act or ISO 42001 mapping.

What it must never contain. No prompts, no model outputs, no personal data. The worksheet's verification step fails any bundle with "payload content inside (prompt, output, personal data)."2 Beacon's own collectors "never store prompts or outputs."6 In the Foundation's design, a receipt carries "hashes, decision, tier, policy hash, key id" and never carries "prompts, outputs, payloads, personal data".7 Receipts point to your comparison table, safety re-run and rollback record by hash. The marker sees those artifacts separately, in a controlled work record; the stranger does not need them to check that the record is intact. People's names stay in that work record too; a receipt carries a role and a reference.

Make "no payload" checkable. A promise of "no payload" is only as good as the check behind it, so the bundle fixes what may be inside it:

A bundle that passes these checks shows no payload was found by these checks. It does not prove the producer never handled personal data, and a hash is not a licence to share what it was made from.

Label what you carried from Level 300. Your hold card and audit receipt from 300 go in with a provenance label on each: made at 300, content-free, and not authenticated at source unless it was signed when it was made. Signing your 400 bundle proves integrity from packaging onward. It never proves what happened before.

Send the key and the checkpoint another way. Before you hand over the bundle, give the stranger, through a separate channel:

The separate channel is the whole point. If the key and checkpoint arrive inside the bundle, whoever changed the bundle could change them too. In the sandbox, apps/sandbox400/trust/bw-svc-notes-400p.trust.json stands for that separate delivery. Its values are below; compare the two routes before you use them:

The sandbox's keys are course test keys derived from public labels, so anyone can sign with them. That is on purpose, so the fixtures can be rebuilt. A pass shows how the check works, not who signed. A real producer's private key never leaves the producer.

Part 6. A stranger verifies it, offline

Your stranger is someone who has never seen your system: for example, a 400 learner from another cohort. They do what the worksheet's step 2 asks of every 400. They copy the bundle "to a machine with no network (airplane mode counts)" and follow its VERIFY.md "step by step, without Beacon, without an account, without us."2 Beacon says a bundle can be checked "with stock openssl — VERIFY.md walks you through it."6 The Foundation's own checklist line for this is "I have verified evidence I did not make, on a machine that was not online."3

They check, and write down:

  1. The key. Does the public key in the bundle match the fingerprint they received separately?
  2. The signatures. Does every receipt's signature verify under that pinned key? (The worksheet's checkbox: "Signature verified".)2
  3. The chain. Is it "continuous (no gaps in seq / prev_hash)"?2 The first receipt must point back to the fixed starting value that VERIFY.md states.
  4. The checkpoint. Does the chain reach the expected last sequence number with the expected hash, and does the policy hash match the expected policy version?
  5. No payload. Is there any prompt, output or personal data inside? If so, it fails.2 They run the file-list check, the schema check and the human look from Part 5.
  6. The carried items. Are the Level 300 items labelled, and does the statement treat them as unverified at source?

In the sandbox. The stranger runs the command the bundle's VERIFY.md gives:

python3 apps/sandbox400/verify_bundle.py apps/sandbox400/bundles/bw-svc-notes-400p --trust apps/sandbox400/trust/bw-svc-notes-400p.trust.json

The saved result is prepared/400P-b-verify-bundle.txt. The verifier numbers its steps 0 to 6, which cover the six checks above in this order: 0 file inventory; 1 both keys against the pinned fingerprints, the manifest signature, and every file's signed hash and size; 2 every receipt's closed schema, signature, sequence from 0 and chain from the genesis (64 zeros); 3 a checkpoint signed by the evidence key that matches the separately received log, sequence, head and policy; 4 the policy file's hash, named by every receipt; 5 the eval report's and decision record's closed schemas, plus a screen of the text files for prompt-, output- or person-like content; 6 the carried Level 300 labels. It prints ACCEPT and exits 0, or names the first failed step and exits 1; it exits 2 if the trust file is missing or invalid. Step 5's screen does not replace the human look: the verifier's own output says a person should still read the text files.

Why the checkpoint matters. A chain whose last few receipts were cut off still verifies internally: every remaining link and signature is fine. Crosby and Wallach, in their paper on tamper-evident logs, warn that "an untrusted logger is free to have different snapshots make inconsistent claims about the past," and that a dishonest logger "may attempt to tamper with its history by rolling back the log."8 Their aim is to detect tampering, not to prevent it.8 Their answer is to get the log's commitments into auditors' hands, because "Tamper-evidence requires auditing. If the log is never examined, then tampering cannot be detected."8 The checkpoint you send separately is that commitment, in the stranger's hands before they open the bundle. It cannot show that every real event was logged in the first place.

Practise on the sandbox. S tamper copies the project bundle 18 times into apps/sandbox400/dist/tampered/, changes each copy in one way, runs the verifier on each and reports the step that rejected it. Before you run it, write your own answer for each case. The cases, by the step that should catch them: step 0, missing-file, extra-file and extra-file-repacked (the extra file also added to a re-signed manifest); step 1, altered-file (a file changed after the manifest was signed), bad-manifest-signature and key-swap; step 2, altered-record-repacked, wrong-genesis, gap, reorder, empty-log, malformed-line and nested-payload (an extra field hidden in a receipt body); step 3, suffix-cut (the last 12 receipts removed) and fork-resigned (history rewritten and re-signed with the evidence key itself); step 4, wrong-policy; step 5, payload-in-eval-report and person-in-crosswalk (an email address in the crosswalk). There is no saved output for this command: plain-gear learners use the case list in apps/sandbox400/tamper.py and the README. The writer's run (20 Nov 2026, not checked) rejected all 18 at their expected steps.

The repeated-field receipt from Part 5 is not one of the 18 cases. The writer tried it by hand on a copy (not checked): writing "decision":"allow","decision":"deny" into receipt 6 fails at step 1, because the receipts file no longer matches the signed manifest. When the manifest is re-signed with the public test packaging key, it fails at step 2 instead: "duplicate property name 'decision'".

Part 7. The stranger's statement

The worksheet's template is: "I, [name], verified bundle [bundle] offline on [date] and found [finding]."2 It must be one sentence that "carries my name and the date."2 This course also asks the stranger to sign it (objective 400.P-b); that is the course's requirement, not the worksheet's. The stranger writes and signs their own sentence. The producer never drafts it for them.

What a verified bundle shows. That the receipts are unchanged since they were signed; that they were signed by the holder of the pinned key; that the chain is complete up to the expected checkpoint; that the policy in the bundle is the expected version; and that the payload checks found nothing.

What it never shows. That the decisions in it were right. That the model is safe, fair or good. That nothing was cut after the checkpoint (only a later checkpoint can show that). That every real event was logged. Whose key it is, without the pinned key material. Anything about the carried Level 300 items before they were packaged, unless they were signed when made. The Foundation puts the limit on levels plainly: "A level describes a person's practice, never a system's assurance."3

A good statement, for the sandbox's project bundle, from Ana, a 400 in another cohort (made up). The verifier's saved output ends with a fill-in template of the same shape; Ana writes and signs her own.

I, Ana Lindqvist, verified bundle bw-svc-notes-400p offline on 26 November 2026 by its VERIFY.md against the two key fingerprints and the checkpoint (sequence 41, its hash, and policy v3) that I received separately, and found all 42 receipt signatures valid under the pinned evidence key, the chain unbroken from sequence 0 to 41 and matching a signed checkpoint, the policy hash matching v3, no payload found by the file-list, schema and read-through checks, and two Level 300 receipts labelled as carried and not authenticated at source, which I did not verify.

A statement that overclaims, and why it fails:

I verified Priya's bundle and confirm the fine-tune is safe to release and the evidence is complete.

It has no name or date, claims the model is safe (a bundle cannot show that), and calls the evidence complete without saying "up to the checkpoint." The worksheet is plain about the limit for systems: "Systems get an OVERT AAL and a coverage number from their own evidence."2 Your stranger's sentence is about the bundle, not a rating of the system. Interpreting assurance levels is left to 500.6.

How the project is marked

Two 400s or a host mark the project. (The worksheet's own checklist says "Two 400s or a founder marked me 400.")2 A mark is a person's decision about your practice; no tool awards it. The project passes when every row is true, in every gear:

CheckPasses when
ComparisonThree rows on the same held-out set, each with an interval; a paired difference for the two closest rows; at least one cohort row with its count and interval; a release rule, with an exact cohort guard, dated before the run
Release gateNames an approver who neither built the model nor holds the Level 300 card; shows one run denied, with the receipt naming the missing evidence
RollbackOne dry rollback that restores the previous model, data, code and configuration versions, with a receipt saying what was checked
MonitoringBuilt on the 300 hold card; names an input-shift measure, a late-label performance check, the cohort row, versions, and the threshold that pages the named holder
BundleContains the chain, the policy and its hash, a framework crosswalk and VERIFY.md with its file list and receipt schema; no prompts, outputs or personal data; names only as roles and references; carried 300 items provenance-labelled
VerificationA stranger verified it offline against a key and checkpoint received separately, and wrote and signed one sentence, with name and date, bounded to what was verified

A held release passes. A fine-tune that lost to retrieval passes. An honest "no clear difference" passes. What fails is a missing interval, a gate that never said no, a rollback that restored only the weights, a bundle with payload in it, or a statement that claims more than was checked.

With the sandbox, every row can be done. The adaptation is planned, simulated with prepared answers and verified by scoring, not trained on a real model; that meets 400.P-a at Level 400 (Bob Rapp, 20 Nov 2026). Markers mark what was actually run, and call a prepared result a prepared result and a paper exercise a paper exercise.

The worksheet also asks you to host a Thursday: "Claim a Thursday on the rota."2 In this course hosting is optional and is not a badge condition.

My checklist:

Words to know: baseline, held-out set, 95% interval, paired difference, cohort, cohort guard, release rule, release gate, allow, constrain, hold, deny, independent approver, dry rollback, monitoring plan, evidence bundle, receipt, hash chain, genesis value, checkpoint, pinned key, out-of-band channel, payload, provenance label, tamper-evident.

Further learning (not a source)

Free university course materials, for going further. Nothing in this lesson relies on them.

Sources

  1. s8 Lord Justice Holroyde, Mr Justice Picken and Mrs Justice Farbey DBE, Court of Appeal (Criminal Division) (2021). Hamilton & Others v Post Office Ltd [2021] EWCA Crim 577 (judgment). Record. Checked 2026-11-19 by source-checker agent (fresh Claude session, separate from the writer), source check 19 Nov 2026. ↩ ↩
  2. s1 AiGovOps Foundation (2026). Level 400 · Prove it to a stranger — worksheet. Official doc. Checked 2026-11-20 by source-checker agent (fresh Claude session, separate from the writer, red team, item writer and round 1 to 4 checkers), round 5 re-check 20 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s9 AiGovOps Foundation (2026). The AiGovOps Practitioner Manifesto. Official doc. Checked 2026-11-19 by source-checker agent (fresh Claude session, separate from the writer, red team and round 1 checker), round 2 re-check 19 Nov 2026. ↩ ↩ ↩ ↩
  4. s3 Evan Miller (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. Paper. Checked 2026-11-20 by source-checker agent (fresh Claude session, separate from the writer, red team, item writer and round 1 to 4 checkers), round 5 re-check 20 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩
  5. s4 Elham Tabassi (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. Standard. Checked 2026-11-20 by source-checker agent (fresh Claude session, separate from the writer, red team, item writer and round 1 to 4 checkers), round 5 re-check 20 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  6. s2 AiGovOps Foundation (2026). AiGovOps Beacon — find every AI, sign what matters. Official doc. Checked 2026-11-20 by source-checker agent (fresh Claude session, separate from the writer, red team, item writer and round 1 to 4 checkers), round 5 re-check 20 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩
  7. s10 AiGovOps Foundation (2026). The AiGovOps Foundation café — start with your problem (Practice home page; object table, Receipt row). Official doc. Checked 2026-11-19 by source-checker agent (fresh Claude session, separate from the writer, red team and round 1 checker), round 2 re-check 19 Nov 2026. ↩
  8. s7 Scott A. Crosby and Dan S. Wallach (2009). Efficient Data Structures for Tamper-Evident Logging. Paper. Checked 2026-11-20 by source-checker agent (fresh Claude session, separate from the writer, red team, item writer and round 1 to 4 checkers), round 5 re-check 20 Nov 2026. ↩ ↩ ↩

Next: the quiz