Level 400: Engineering AI systems · Module 400.P

A visitor works alone at a corner table with a folder, a sealed envelope and a laptop turned away.
Verify it offline

Project: prove it to a stranger · Plain gear

Reviewed and published, version 1.0.0. As of 20 Nov 2026. Not a credential. Passing it does not rate any AI system.

The project kit is the Level 400 sandbox, in the course files at apps/sandbox400/. Its guide is apps/sandbox400/README.md. It has made-up results for the three ways of doing the job, a gate, a place to practise going back, a watching-plan form, a practice bundle with its two keys, and a checking tool. Everything in it is made up. The no-code route reads the ready-made results in apps/sandbox400/prepared/. The training step: Bob Rapp decided on 20 Nov 2026 that at Level 400 you plan the extra training, try it out in the sandbox and check its results. You do not train a real model. The sandbox's trained-model answers are ready-made, not made by a model. What we do not know yet: nobody has timed the project. An independent checker ran the sandbox's project steps on 19 Nov 2026, but the sandbox has changed since; the writer's run of today's version is not checked.

The big idea

Adapt and evaluate a model with intervals and a cohort row; exercise the release gate with an independent approver and one denied run; rehearse one rollback; write the monitoring plan; produce an evidence bundle that a stranger verifies offline and describes in one signed sentence.
The audit gate is about independence (s1). The Foundation's design goal is a receipt a stranger can verify offline (s9). The five parts are the project's structure, run on synthetic fixtures.
Figure data as text

Five parts, one sentence

  1. GET TO YES — adapt and evaluate: baselines, an interval for every score, at least one cohort row. (course)
  2. GET TO YES — exercise the release gate: an approver who did not build it; one run denied for missing evidence. [s1]
  3. RECOVER TO YES — rehearse a rollback: model, data, code and configuration restored together. (course)
  4. STAY AT YES — write the monitoring plan: built on your Level 300 hold card. (course)
  5. RECOVER TO YES — a stranger verifies offline: with a key and checkpoint got separately; one signed sentence. [s9]

Objective: 400.P-b. Text equivalent of figures/five-parts-one-sentence.svg.

Imagine you sell a used car. You say, "It runs great. Trust me." A careful buyer does not trust you. They want the service record, stamped by the garage. They want to check it themselves.

This project is about the same thing, for an AI model. You show whether your model is ready, and you pack the proof so that a stranger can check it. The stranger does not need to trust you. They do not need your computer, your account or an internet connection.

You do this with the course sandbox. You do not need to write code. You read the results the sandbox has ready, and you make the decisions. Everything in the sandbox is made up. Big Widget Company and its people are made up too. The worksheet, the standards, the papers and the court case are real, and each one is cited.

The course plans about two hours for the project. Nobody has timed it yet. Two people who have passed Level 400, or a host, mark it after. Hosting a Thursday call is your choice. You do not need to do it to pass.

Why "trust me" is not enough

In the UK, people who ran local post offices were taken to court when the Post Office's Horizon computer records showed cash missing. The Court of Appeal later said they were convicted "on the basis that the Horizon data must be correct, and cash must therefore be missing, when in fact there could be no confidence as to that foundation."1 The court said the Post Office had, in effect, turned the burden of proof around.1 The people accused had to prove the computer was wrong, and they had no way to check it.

The Foundation's Level 400 worksheet puts it in one line: "All three ran for years because no one outside could check."2 Horizon is one of its three cases. That line is the worksheet's summary, not the court's.

So the lesson is this. Evidence should not need the checker to trust the team that made it. Good evidence lets an outsider check it alone. The Foundation wants a record "that a stranger can verify offline, without an account and without trusting us."3

Three yeses, and who decides

The Foundation names three steps: Get to Yes before something ships, Stay at Yes while it runs, and Recover to Yes after, when someone asks or something goes wrong.3 In this project:

A gate gives one of four answers: go ahead, only this far, wait for a person, or no. The exact words are allow, constrain, hold and deny. "Wait for a person" is not a yes. In this project nothing ships while it waits.

People decide. Tools do not. The sandbox can work out the numbers and check that papers exist. It never approves the model, marks your project or signs the stranger's sentence. The people are: you; the person who approves the release; the person who looks after the running system; the stranger; and the two 400s or the host who mark your work.

How to use the sandbox

Every step below names a ready-made result file in apps/sandbox400/prepared/. Open it and read it. That is the whole no-code route. If you want to see the sandbox print it for you, type python3 apps/sandbox400/sandbox400.py plain and then the file's name without .txt. The sandbox guide, apps/sandbox400/README.md, lists the command for every step. Use only this sandbox's files for this project.

Part 1. Is the model actually better?

Many people think a model is done when it gets a good score. A score on its own does not tell you much. You need to know: better than what? Out of how many? How sure are we? And is it good for everyone, or only on average?

The job. Big Widget gets notes from dealers about repairs. Each note has to go to the right team. The model's job is to sort each note into the right repair group.

Three ways to do it. The sandbox has results for three ways, all tried on the same 400 practice notes the model had never seen:

Plan it, try it, check it. For the extra training you do three things. Plan it: write down which starting model, which notes and which settings you would use. The sandbox's release record does this with a label for each piece. Try it in the sandbox: its ready-made answers stand in for the trained model. No real model is trained. Check it: score those answers on the same 400 notes as the other two ways, and run the safety check again.

How sure are we? Every score comes with a range. Think of a weather forecast that says "between 15 and 20 degrees." The range shows how much the score could move if you tried a different set of notes. Evan Miller, who writes about AI tests, says a score should be reported together with how uncertain it is.4 He also says that when you compare two models, compare them on the very same questions, because that makes the comparison more precise.4

Here are the sandbox's results (made up). The file is 400P-a-evaluation.txt.

WayRightScoreLikely range
Just ask284 of 40071%about 67% to 75%
Look it up312 of 40078%about 74% to 82%
Extra training320 of 40080%about 76% to 84%
Extra training compared with look it up, on the same notes2 points betterfrom about 1 point worse to 5 points better

The extra-training model scored 2 points higher. But the range for that gap goes from a little worse to a little better. So we cannot say it is really better. "No clear difference" is an honest answer.

Is it fair to everyone? An average can hide a group that does badly. So you look at one group on its own. Here, it is the 50 notes written in French by dealers in Quebec.

French-language notesRightScoreLikely range
Look it up36 of 5072%about 60% to 84%
Extra training31 of 5062%about 49% to 76%

Fifty notes is a small group, so the ranges are wide. They do not prove extra training is worse for this group. But it scored 10 points lower here. That is a warning sign, not a pass. NIST, the US standards agency, says AI systems should be checked for fairness and bias, and the results written down.5

The same file also has the safety check, run again on the extra-training answers. In the sandbox it passes (made up).

Decide the rule first. Before you look at the results, write your rule. Make it exact, so two people would get the same answer. The course's example rule:

We use extra training only if all three are true:

  1. it is clearly better than look it up (its whole range for the gap is above zero);
  2. its safety check passes;
  3. in every group we report, it scores no more than 5 points lower than look it up.

Otherwise we wait and check again.

The 5-point limit is the course's own choice. It does not come from NIST or Miller. In a small group a score can jump about by chance, so the limit may make you wait when the model is really fine. That is on purpose: when you cannot tell, you wait. A person, not this lesson, chooses the limit. If Bob, Ken and the red-team lead pick a different one, every gear changes with it.

Here, rule 1 fails (the range for the gap goes below zero). Rule 2 passes. Rule 3 fails (10 points lower is more than 5). So the rule says: wait. The sandbox's decision record at the end of the file says the same: "hold and recheck", and the look-it-up model stays in use. That is a good result for this project. It is honest.

Part 2. The gate that can say no

A release gate is like a sign-out sheet with a guard. It starts at "wait for a person". It says no if any paper is missing. If every paper is there, it still waits for someone who did not build the model.

Who approves? Not the person who built the model. Not the person who already looks after the running system at Level 300. Someone else. NIST says it is good practice to keep the people who build a model separate from the people who check it.5 In our example, Priya built it and Tomás, a quality engineer, checks it (both made up). Only Tomás can turn "wait" into "go ahead". The gate cannot do that on its own.

Watch it say no. The sandbox has three tries at a release. Read each file:

The files never show Tomás's name. They show his job and a reference number.

If you have never seen your gate say no, you do not know that it works. One "no" tests one way it can fail, not all of them.

Part 3. Practise going back

If a new model goes wrong, you need to go back to the old one. But a model is not just one file. It also depends on the guides it looks things up in, the settings around it and the program that runs it. Going back means putting all of them back, together. Think of a recipe: going back to last week's cake means last week's flour and oven setting too, not just last week's recipe card.

The sandbox does one practice go-back. It moves the new version in, then goes back. Then it checks that all four pieces match the old version: the model, the guides it reads, the program and the settings. Read 400P-a-rollback.txt: all four pass. Then read 400P-a-rollback-weights-only.txt. It shows the common mistake: going back on the model only, which leaves three pieces on the new version. The record says what was checked, and the job (not the name) of who did it. NIST says there should be a way to stop or turn off an AI system that is not working as planned, and that people should know whose job that is.5

Part 4. Keep watching

A model that passed its tests can still drift. New dealers open. Notes change. You find out whether the model was right only weeks later, when engineers finish the repair.

You start from the hold card you made at Level 300. It already names who looks after the system, the limits, and how to stop it. You add:

The sandbox has a form for this plan. Read 400P-a-monitoring-plan.txt. It starts from the Level 300 hold card, lists the three things to watch and what the person in charge can do, and notes the version of all four pieces. Its first alarm number is marked as a placeholder: you set your own. This part is written, not run. The sandbox has no live system to watch. NIST says a working AI system should be watched while it runs.5

Part 5. Pack the proof

Now you pack the proof into a bundle for a stranger. The worksheet says to build it "for a reviewer who has never seen your system."2

What goes in. A chain of receipts. Each receipt is a short signed note: "this was checked," "this was approved," "this was denied." Each one is linked to the one before, like links in a chain. The first one links to a fixed starting mark that the guide names; that is this course's design. The bundle also has your rule and its fingerprint, a list that links your work to well-known rule books, and a one-page guide called VERIFY.md that explains how to check it.2 The sandbox's bundle is in apps/sandbox400/bundles/bw-svc-notes-400p. It has 42 receipts, numbered 0 to 41, and twelve files. One of them is a signed packing list of the other files.

What Beacon's page says. The course's Beacon tool builds a signed bundle with a list of receipts, your rules and a one-page VERIFY.md.6 Its page also says Beacon matches each model to the rule-book articles it must answer to.6 But its list of what goes in a bundle does not name a list of rule books, and the page does not say it links receipts into a chain or gives a last-receipt check. Whether a Beacon bundle does what this project asks is not checked. The course sandbox is not Beacon.

What never goes in. No questions you typed to the model. No answers the model gave. No personal information. If a bundle has any of these, it fails.2 Beacon is built the same way: it never stores prompts or outputs.6 The Foundation says a receipt holds things like fingerprints and the decision, and never "prompts, outputs, payloads, personal data".7 Even names stay out: a receipt says "the approver" and a reference number, not "Tomás". The real papers stay in your own work record.

Make "nothing private" checkable. The guide lists every file allowed in the bundle, and every box allowed on a receipt. Each box may hold only a fingerprint, a number, a date, a reference or a word from a short list. No free writing. Anything extra fails. Each box gets one answer, never two. Then the stranger reads what text is left, to be sure. Passing means they found nothing private. It does not prove there never was any.

Label old papers. Your Level 300 hold card and receipt go in too, each with a label: "made at Level 300, not signed when made." Signing your new bundle shows nothing was changed after you packed it. It does not prove what happened before.

Send things another way. Before you hand over the bundle, send the stranger some things by a different route, like a text message when the bundle goes by email:

Why another route? If someone changed the bundle, they could also change anything inside it. They cannot change the message the stranger already has. In the sandbox, these values sit in apps/sandbox400/trust/bw-svc-notes-400p.trust.json, which stands for the separate message. The standard and technical gears print them too, so you can compare.

Part 6. The stranger checks it

The stranger takes the bundle to a computer with no internet. The worksheet says "airplane mode counts."2 They follow the VERIFY.md guide "step by step, without Beacon, without an account, without us."2

The sandbox's checking tool runs seven checks, numbered 0 to 6:

  1. Are exactly the listed files there, with none missing and none extra?
  2. Do both keys match the fingerprints they got separately? Does the signed packing list match every file?
  3. Are all the signatures good?2 Is the chain whole, with no missing links?2 Does it start from the fixed starting mark? Does every receipt use only the allowed boxes?
  4. Does the last receipt match the number and fingerprint they got separately?
  5. Is the rule file the one they expected?
  6. Is there no private content inside?2 The tool checks the boxes and looks through the text, but the stranger still reads it.
  7. Are the old Level 300 papers labelled?

Read the result in 400P-b-verify-bundle.txt. Every check passes, and the result is "ACCEPT".

Check 3 matters most. If someone cut the last few receipts off the end, every remaining link would still look fine. Two computer scientists, Crosby and Wallach, studied tamper-evident records: records that let a checker see a change, not records that nobody can change.8 They warn that whoever keeps a record can change its story about the past.8 They also say: "If the log is never examined, then tampering cannot be detected."8 The number you sent separately is how the stranger catches a cut. It cannot show that everything that really happened was written down in the first place.

Try it. The sandbox can make 18 broken copies of the bundle. Each one should fail at one check. For example: one has a file changed after packing (check 1); one has a missing file and one an extra file (check 0); one has the wrong key (check 1); one has receipts swapped round, and one hides some writing in a receipt (check 2); one is cut short, and one has its end rewritten and signed again (check 3); one has the wrong rule file (check 4); and one has an email address in its text (check 5). For each, say which check should catch it. Then compare with the sandbox guide's list. Running it needs the code route (python3 apps/sandbox400/sandbox400.py tamper); there is no ready-made file for it. The tool also refuses a receipt that gives the same box two answers, such as "allow" and "deny". Check 5 is always partly yours: no tool can read for private content for you.

Part 7. One honest sentence

The stranger writes one sentence, with their name and the date.2 The worksheet gives the shape: "I, [name], verified bundle [bundle] offline on [date] and found [finding]."2 This course also asks them to sign it. They write it themselves. You never write it for them.

The sentence must say only what they checked. A good one, for the sandbox's bundle (made up):

I, Ana Lindqvist, checked bundle bw-svc-notes-400p offline on 26 November 2026 against the two key fingerprints and the last receipt number (41) and its fingerprint that I got separately, and found all 42 signatures good, the chain whole up to receipt 41, the rule file the one expected, no private content found by the file, box and read-through checks, and two Level 300 papers labelled as carried, which I did not check.

A bad one:

I checked it and the model is safe.

The bad one has no name or date. And a bundle can never show that a model is safe. It can only show that the records were not changed after they were signed, and that none were cut before the last receipt the stranger was told about. As the Foundation puts it, "A level describes a person's practice, never a system's assurance."3

How it is marked

Two people who have passed Level 400, or a host, mark your project. A person decides the mark, not a tool. The worksheet's own list says "Two 400s or a founder marked me 400."2 You pass when you have all of these:

A model that waits can pass. A model that did not win can pass. Being honest is what passes.

The extra training is planned, tried out in the sandbox and checked, not done on a real model. That is what Level 400 asks for (Bob Rapp, 20 Nov 2026). Markers mark what was really done, and say so when a result was ready-made.

My checklist:

Sources

  1. s8 Lord Justice Holroyde, Mr Justice Picken and Mrs Justice Farbey DBE, Court of Appeal (Criminal Division) (2021). Hamilton & Others v Post Office Ltd [2021] EWCA Crim 577 (judgment). Record. Checked 2026-11-19 by source-checker agent (fresh Claude session, separate from the writer), source check 19 Nov 2026. ↩ ↩
  2. s1 AiGovOps Foundation (2026). Level 400 · Prove it to a stranger — worksheet. Official doc. Checked 2026-11-20 by source-checker agent (fresh Claude session, separate from the writer, red team, item writer and round 1 to 4 checkers), round 5 re-check 20 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s9 AiGovOps Foundation (2026). The AiGovOps Practitioner Manifesto. Official doc. Checked 2026-11-19 by source-checker agent (fresh Claude session, separate from the writer, red team and round 1 checker), round 2 re-check 19 Nov 2026. ↩ ↩ ↩
  4. s3 Evan Miller (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. Paper. Checked 2026-11-20 by source-checker agent (fresh Claude session, separate from the writer, red team, item writer and round 1 to 4 checkers), round 5 re-check 20 Nov 2026. ↩ ↩
  5. s4 Elham Tabassi (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. Standard. Checked 2026-11-20 by source-checker agent (fresh Claude session, separate from the writer, red team, item writer and round 1 to 4 checkers), round 5 re-check 20 Nov 2026. ↩ ↩ ↩ ↩
  6. s2 AiGovOps Foundation (2026). AiGovOps Beacon — find every AI, sign what matters. Official doc. Checked 2026-11-20 by source-checker agent (fresh Claude session, separate from the writer, red team, item writer and round 1 to 4 checkers), round 5 re-check 20 Nov 2026. ↩ ↩ ↩
  7. s10 AiGovOps Foundation (2026). The AiGovOps Foundation café — start with your problem (Practice home page; object table, Receipt row). Official doc. Checked 2026-11-19 by source-checker agent (fresh Claude session, separate from the writer, red team and round 1 checker), round 2 re-check 19 Nov 2026. ↩
  8. s7 Scott A. Crosby and Dan S. Wallach (2009). Efficient Data Structures for Tamper-Evident Logging. Paper. Checked 2026-11-20 by source-checker agent (fresh Claude session, separate from the writer, red team, item writer and round 1 to 4 checkers), round 5 re-check 20 Nov 2026. ↩ ↩ ↩

Next: the quiz