
The audit gate: prove it to a stranger · Technical gear
As of 20 November 2026. Reviewed and published, version 1.0.0. People, not agents, signed every review. The source checker's 20 November re-check found the two Robodebt report (s1) wording fixes supported. The lab now uses the course sandbox in apps/sandbox400/. Big Widget Company, its formula, its policy files, its keys and its bundles are fictional. Court, inquiry and regulator records support their own case and jurisdiction only; nothing here is legal advice. The Foundation's worksheet, audit page and Beacon tool are course practice, not independent validation. Unknowns are marked as unknown or not checked, with who checks next.
Scope
Figure data as text
Three gates on the three yeses
| Where | What it shows |
|---|---|
| GET TO YES [s5] | Premise gate. An independent validator checks the legal basis, a statistical validation and an equity test by cohort. |
| GET TO YES, THEN STAY [s5] | Protected-attribute gate. Protected attributes and stand-ins barred; a person reviews each adverse decision. |
| RECOVER TO YES [s5] | Stranger's check. Someone outside the team verifies the evidence offline, with a key and checkpoint got separately. |
This gear treats an automated decision system as three separable artefacts, each with its own control and its own place in the three yeses (Get to Yes before running, Stay at Yes while running, Recover to Yes after, on demand or on incident):
| Artefact | Failure mode | Control | Yes | Primary anchor |
|---|---|---|---|---|
| The decision rule (formula, threshold, feature list, data) | The rule computes correctly but its premise is unlawful or unsound | Premise gate: binding independent validation before first issuance | Get to Yes | Robodebt Royal Commission report1; Bates (No 6) para 9362; Hamilton para 1213 |
| The feature list | A protected attribute, or a proxy for one, drives adverse decisions | Protected-attribute gate: deny on protected features and failed proxy tests; human review of adverse decisions | Get to Yes, Stay at Yes | AP fine decision, 25 Nov 20214; Ongekend onrecht5 |
| The evidence record | The record is unverifiable, signed by an unknown key, or silently truncated | Offline verification against pinned trust material and an independently received checkpoint | Recover to Yes | Hamilton para 1373; RFC 80326; RFC 91627 |
The course's Level 400 worksheet draws the same three gates from the same three cases.8 The rest of this framing is the course's own, not the worksheet's: it carries forward the Level 300 artefacts (a hold with a named holder; a content-free receipt), requires an independent validator distinct from that holder, and asks for a bundle a stranger verifies against trust material obtained outside it.
None of this approves, certifies or accredits a system. A level describes a person's practice, not a system's assurance.
The premise gate
Arithmetic correctness is not premise validity
Let a dealer $d$ have monthly warranty claims $c_{d,1}, \dots, c_{d,12}$, with mean $\bar c_d = \tfrac{1}{12}\sum_m c_{d,m}$. Big Widget's proposed rule (fictional; an analogy, not a reconstruction of Robodebt) flags month $m$ when
c_{d,m} > (1 + \tau)\,\bar c_d, \qquad \tau = 0.4,
and issues a clawback notice for $c_{d,m} - \bar c_d$. Unit tests can establish that the implementation computes this expression. They cannot establish the premise, which has three independent parts:
- Legal basis. The contract permits a clawback on this evidence, with the notice and appeal owed.
- Construct validity. A flag implies an overclaim: $P(\text{overclaim} \mid \text{flag})$ is high enough to justify a demand. Seasonality (claims cluster in some months) and small-count variance (a low-volume dealer's monthly counts fluctuate widely around its mean) both break the implication without any error in the code.
- Data fitness. The inputs mean what the rule assumes: claims are recorded in the month of repair, not the month of payment, and so on.
The record: Robodebt
The Royal Commission into the Robodebt Scheme reported on debts raised by income averaging.1 Commissioner Catherine Holmes AC SC presented the report to the Governor-General on 7 July 2023, when it was tabled; an updated edition with corrections was published on 11 July 2023.1 The report makes 57 recommendations.1 Volume 1: "Robodebt was a crude and cruel mechanism, neither fair nor legal, and it made many people feel like criminals."1
Volume 1's Overview of Robodebt also records that in late 2014, before the scheme began, the Department of Social Services had in-house legal advice raising concern that averaging might not be consistent with the legislative framework, and policy advice that averaging did not accord with the legislation, and that the scheme went ahead without the legislative change the department had said was needed.1 In mid-2018 the Department of Human Services referred to the Department of Social Services an Administrative Appeals Tribunal decision that income averaging was unlawful.1
The engineering reading separates three things a process can conflate:
| Stage | What it is | What it does on its own |
|---|---|---|
| Advice obtained | An opinion exists in a file | Nothing |
| Validation decision | An independent party records allow, hold or deny on the evidence | Nothing, unless enforced |
| Enforcement | The issuing path checks for a valid allow receipt and fails closed | Blocks issuance |
Robodebt had adverse advice before issuance, and the scheme went ahead; advice on its own blocks nothing. In-house advice is not evidence that an independent validation gate existed or passed. The design requirement is the third row: a hold or failed validation must make issuance impossible, not merely inadvisable.
The mechanics of the averaging, the volume of debts and the timeline of each later decision are not taught here. They are not checked against the report, and await the source checker.
The record: Horizon
The same failure appears with system output in place of a formula. In Hamilton v Post Office Ltd [2021] EWCA Crim 577, the Court of Appeal heard 42 appeals by former sub-postmasters and sub-postmistresses prosecuted between 2003 and 2013.9 Of the "Horizon cases" it held: "In each of those cases, there was no independent evidence of an actual shortfall, and it was essential to the prosecution case that the Horizon data was reliable."3 It accepted the High Court's finding of "significant problems with Horizon, which gave rise to a material risk that an apparent shortfall in the branch accounts did not in fact reflect missing cash or stock".3 In Bates v Post Office Ltd (No 6) (16 December 2019, about sixteen months earlier) the High Court found that Legacy Horizon "was not robust", and that Horizon Online in its HNG-X form had robustness that "was questionable, and did not justify the confidence placed in it by the Post Office in terms of its accuracy"; it treated the later HNG-A version as a different matter, far more robust.2 The findings are version-specific. They bound the Horizon cases the Court of Appeal identified, not every version of the system or every prosecution.
Evidence the validator requires
The worksheet's premise gate requires legal_basis, statistical_validation and equity_test_by_cohort, approved by an independent_validator, "not the team that wrote the rule".8 For the clawback rule, a defensible statistical validation looks like this (a course design, with acceptance thresholds fixed before the data is examined):
- Draw a sample of $n$ historical flagged months whose true status auditors established by hand. Let $k$ be the number that were not overclaims. Estimate the false discovery rate $\widehat{\text{FDR}} = k/n$ with a 95% confidence interval for a proportion.
- Pre-register a ceiling, for example: allow only if the upper bound of the interval is below the ceiling the legal and finance owners agree.
- Equity by cohort: compute $\widehat{\text{FDR}}_g$ for each cohort $g$ (volume band, region, climate zone) and pre-register a bound on the gap or ratio between cohorts. If a cohort's sample is too small to bound its error, that is insufficient evidence, and the answer is hold, not allow.
- Legal basis: an adverse opinion is a fail. It is recorded as deny, not filed as a caveat.
The output is a content-free validation receipt bound to the exact rule version (below). Who decides is a person: the validator records allow, hold or deny, and an agent that assembled the evidence does not. Validation of v0.2 does not license v0.3.
The two gates as policy
Receipt validity
Both gates depend on one predicate, defined once. First, what is signed. A receipt record is $r = (\text{seq}, \text{prev\hash}, \text{body}, \text{sig})$. The body holds every field the receipt claims: action, decision, per-check results, role, rule hash, policy version, key id and time. The signed part is the record without its signature, $u = (\text{seq}, \text{prev\_hash}, \text{body})$, so the position in the log is signed too. The signature is detached: it is never part of the signed message. Issuance computes $\text{sig} = \mathrm{Sign}(sk, \mathrm{canon}(u))$; verification recomputes $\mathrm{canon}(u)$ from the record and checks $\text{sig}$ against it. $\mathrm{canon}$ is one fixed byte encoding used on both sides. The course sandbox uses RFC 8785 JCS, as its bundles' VERIFY.md states. This is the construction the lab below verifies: record $i$ is {seq, prev_hash, body, sig} and sig signs JCS({seq, prev_hash, body}). Let $R$ be the set of receipts the gate can read, $K{\text{val}}$ the set of keys Big Widget registered for the independent-validator role, and $H(\text{rule}_{\text{current}})$ the hash of the current formula, threshold and feature list. For an action $a$ and a set of required checks $C$:
\mathrm{valid}(r, a, C) \iff
\begin{cases}
b.\text{action} = a \\
b.\text{rule\_sha256} = H(\text{rule}_{\text{current}}) \;\wedge\; b.\text{policy\_version} = v_{\text{current}} \\
b.\text{decision} = \text{allow} \\
\forall c \in C:\; b.\text{result}[c] = \text{pass} \\
\mathrm{Verify}(pk_{b.\text{key\_id}}, r.\text{sig}, \mathrm{canon}(u)) \;\wedge\; pk_{b.\text{key\_id}} \in K_{\text{val}} \\
b.\text{approver} \notin \text{Authors} \cup \{\text{Holder}\}
\end{cases}
where $b = r.\text{body}$, $u$ is $r$ without its signature, and $pk_{b.\text{key\_id}}$ is the registered public key the body names. The key id sits inside the signed body, so it cannot be swapped without breaking the signature.
A receipt whose decision is hold or deny, whose rule hash is stale, whose required check failed, or whose signer is not a registered validator key is not valid, and the gate treats it exactly as absence. The binding of approver identity to key is an operational control (who holds the key, and who registered it); this gear assumes it and does not test it.
The receipt carries hashes, the decision, per-check pass or fail, the role, the rule and policy versions, the key id and the time. It never carries the legal opinion, the validation sample, dealer records or any personal data. Those live in a separately controlled work record, referenced by hash. A hash reference is not a licence to disclose the underlying material, and a hash of low-entropy data is not automatically anonymous.
Gate 1: premise
The worksheet expresses the gate as a hold on publishing the rule and a deny on issuing notices unless: receipt_exists(decision.rule.publish) ("no validated premise, no notice — ever").8 Taken literally, receipt_exists is satisfied by a receipt recording hold or deny. The Big Widget version replaces it with the validity predicate above. That replacement, the rule-hash binding and the holder exclusion are the course's design, not the worksheet's.
# premise-gate.yml (fictional, Big Widget warranty clawback; course design)
- action: decision.rule.publish
decision: hold # wait for a person
approver_role: independent_validator
approver_not_in: [rule.authors, system.holder] # separation of duties
approver_keys: registry.independent_validator # K_val
evidence_required: [legal_basis, statistical_validation, equity_test_by_cohort]
receipt_binds: [rule_sha256, policy_version, evidence_refs, results, approver_role, key_id, decision, time]
- action: notice.issue
decision: deny
unless: valid(decision.rule.publish,
checks: [legal_basis, statistical_validation, equity_test_by_cohort])
Formally, notice.issue is permitted iff $\exists\, r \in R : \mathrm{valid}(r, \texttt{decision.rule.publish}, \{\text{legal\_basis}, \text{statistical\_validation}, \text{equity\_test\_by\_cohort}\})$. Absence of such a receipt is a deny, never an allow-with-warning: the gate fails closed. A pending hold is not a slow allow.
A gate has four answers: allow, constrain, hold, deny (in plain words: go ahead, only this far, wait for a person, no). This gate uses three. decision.rule.publish starts at hold and ends at allow or deny; notice.issue is deny until a valid allow exists. Constrain would mean, say, notices only for one region during a pilot; in the course's design it needs its own valid receipt that states the limit, and a notice outside the limit is denied. A hold also needs an end date. In the course's design an unresolved hold lapses to deny, not to allow.
The human gate. Ines, in model risk, outside warranty, holds a registered validator key. Only her signed allow, on passing evidence for the current rule hash, opens notice.issue. An agent may compile the evidence pack; it does not hold a validator key and does not sign.
Why exclude the holder. The Level 300 holder (Marcus, warranty operations) operates the system: receives its alerts, can stop it, releases operational holds. That is Stay at Yes, with an incentive to keep it running. Validation asks whether it should run at all: Get to Yes. Placing both in one role recreates the conflict the gate exists to remove. Marcus keeps his stop power regardless.
Repairing a bad gate. Given approver_role: rule_author and notice.issue: allow, the repair is approver_role: independent_validator with approver_not_in: [rule.authors, system.holder], and notice.issue: deny unless a valid receipt exists (allow, current rule hash, every required check passed, registered validator key).
Gate 2: protected attributes and proxies
The Tweede Kamer's inquiry committee sent Ongekend onrecht (Kamerstuk 35 510, nr. 2) to the Speaker on 17 December 2020; it found that "grondbeginselen van de rechtsstaat zijn geschonden" (fundamental principles of the rule of law were violated), a charge applied to the implementing agency, the legislature and the courts.5 Its remit excluded risk profiles that used nationality, and it notes that the Autoriteit Persoonsgegevens investigated how the tax administration handled (second) nationality in risk selection.5
The AP's fine decision is dated 25 November 2021: administrative fines totalling EUR 2.75 million on the Belastingdienst for processing the (dual) nationality of childcare-benefit applicants in an unlawful, discriminatory and therefore improper way.4 One indicator in a system that automatically selected risky applications was Dutch or not Dutch nationality.4 Separately, the decision finds that Dutch citizens' dual-nationality data was kept without a lawful basis from January 2014.4 Its only scale figure has a different scope: on 25 May 2018, 1.4 million citizens with dual nationality were registered in the system concerned, and the decision states it cannot be established how many were Dutch and had applied for childcare benefit.4 The 1.4 million is a database count, not the unlawfully retained subset and not an affected-applicant count; this gear gives no such count.
This is direct use of a protected attribute, not a proxy case. The gate must cover both. Let $F$ be the rule's feature set and $A$ the protected attributes in scope. The worksheet's gate bars "protected attributes and their proxies" as risk features and requires that "every adverse decision has meaningful human review".8 It has no receipt or deny clause; those below are the course's design.
# protected-attribute-gate.yml (fictional; course design)
- action: decision.rule.publish
decision: deny
unless:
- valid(feature_review, checks: [features_enumerated]) # F enumerated and hashed into rule_sha256
- intersect(features, protected_attributes) == []
- valid(proxy_test, checks: [proxy_bound_met]) # every f in F under the pre-registered bound
- action: notice.issue
decision: hold # wait for a person
approver_role: human_reviewer # meaningful review of each adverse decision
A proxy test asks, for each $f \in F$, how well $f$ (alone or with others) predicts membership of a protected group, against a pre-registered bound; 400.7 covers methods and their limits. The receipt records pass or fail per feature. A receipt recording that the test ran and found a proxy has proxy_bound_met = fail, so valid is false and publication is denied. Removing $A$ from $F$ while leaving a strong proxy in place passes a naive column check and fails this gate.
The notice.issue hold is the Stay at Yes half: for as long as the rule runs, each adverse notice waits for a named reviewer (at Big Widget, Priya in dealer relations) who can stop it. The receipt for each review records the decision and the reviewer's role, not the dealer's details.
Run both gates in the sandbox
The course sandbox, apps/sandbox400/, runs both gates on synthetic Big Widget scenarios. Read its apps/sandbox400/README.md first. From the repository root, with Python 3.9 or later and no network:
python3 apps/sandbox400/sandbox400.py audit premise
python3 apps/sandbox400/sandbox400.py audit protected
audit premise runs nine scenarios through the worksheet's literal receipt_exists and through the valid() predicate above. The draft predicate allows eight of them: the one valid allow, and seven that should fail (a hold receipt, a stale rule hash, an old policy version, a failed equity test, an unregistered key, and an approver who is a rule author or the holder). valid() denies those seven. With no receipt, both deny. audit protected runs four feature lists and fails closed: a direct protected feature, a failed proxy test and a missing proxy test all deny. The prepared outputs are apps/sandbox400/prepared/4009-b-premise-gate.txt and 4009-b-protected-attribute-gate.txt. These scenario receipts are unsigned: the exercise tests the predicate's fields, roles and key registry, not signatures. Signatures are the next section's job.
Two sentences a minister could read
- "No clawback notice is sent until someone who did not write the rule, and does not run it, has checked this exact version and recorded that it is lawful, that it is right often enough, and that it is not wrong more often for some dealers than others."
- "The rule may not use nationality, ethnicity or anything shown to stand in for them, and a named person reviews every notice before it goes out."
Verifying a stranger's bundle
Why the verifier must not depend on the producer
The Court of Appeal found that the Post Office "consistently asserted that Horizon was robust and reliable" while it did not adequately consider or disclose problems with it, and that defendants "were prosecuted, convicted and sentenced on the basis that the Horizon data must be correct, and cash must therefore be missing, when in fact there could be no confidence as to that foundation".3 It found the convictions unsafe in all 39 Horizon cases before it.9 Read this as a failure in those prosecutions, not as proof that Horizon could not be tested in principle: the High Court's Horizon Issues judgment, above, did examine it.2 The design requirement: verification that needs no cooperation, software, account or network from the party whose evidence is being tested. This is the Recover to Yes gate: evidence someone else can test after the fact.
The bundle
Beacon describes its bundle as "the manifest, the NDJSON receipt log, the public keys, the policy YAMLs, the checklists, and a one-page" VERIFY.md, says receipts are signed with an Ed25519 key over JCS canonical bytes (RFC 8785), and says "Anyone with openssl can re-check it. No vendor login. No phoning home."10 The worksheet's procedure: move the bundle to a machine with no network, follow VERIFY.md "without Beacon, without an account, without us", and check the signature, chain continuity ("no gaps in seq / prev_hash") and absence of payload content.8 The worksheet also lists a beacon verify --offline <bundle> command that "does the mechanics in one command", but marks it for the Fall release, not as available today.8 Either way, the VERIFY.md route is the one that does not depend on the producer's tool.
The model
This is the construction the course sandbox's bundles use, as each bundle's VERIFY.md states. Write the receipt log as $r_1, \dots, r_N$ with $r_i = (\text{seq}i, \text{prev\_hash}_i, b_i, \text{sig}_i)$ and unsigned part $u_i = (\text{seq}_i, \text{prev\_hash}_i, b_i)$. Let $h_i = H(\mathrm{canon}(u_i))$, with $\text{prev\_hash}_i = h{i-1}$ and $h_0$ = 64 zero hex characters, the genesis. Each $\text{sig}_i$ is a detached Ed25519 signature over $\mathrm{canon}(u_i)$, as defined above, under the evidence key. A separate packaging key signs the canonical form of the bundle's manifest, which lists every other file with its SHA-256 and size. RFC 8032 verification takes a message, a signature and a public key; if any decoding fails, the signature is invalid.6 The verifier holds, from an authenticated channel independent of the bundle (not a second file shipped with it):
- pinned trust material: the fingerprints of the evidence key $pk^\*$ and the packaging key;
- an expected checkpoint $(n^\, h^\, v^\*)$, plus the policy file's hash: the last sequence number, its record hash and the policy version in scope.
The pinned keys and checkpoint are the course's additions; the worksheet's checklist does not name them. The procedure, numbered as the sandbox verifier numbers it, runs offline throughout:
- Inventory. Exactly the files
VERIFY.mdlists; none missing, none extra. - Keys and manifest. Both keys in the bundle must match the pinned fingerprints. The manifest signature must verify under the pinned packaging key, and every file,
VERIFY.mdincluded, must match its signed hash and size. Every signature is checked under a pinned key, not under the bundle's own key file because it is there. - Signatures and chain. Every receipt fits a closed schema and its signature verifies under $pk^\*$; a missing or invalid signature fails the bundle. For $i = 1..N$: $\text{seq}i = i$ and $\text{prev\_hash}_i = h{i-1}$.
- Checkpoint. A checkpoint in the bundle, signed under $pk^\$, must match $n^\$, $h^\$, $v^\$ and the policy hash received separately, and $N = n^\$, $h_N = h^\$.
- Policy. The policy file hashes to the expected value, and every receipt names it.
- Content. No prompts, outputs or personal data; any payload content fails the bundle.8
- Provenance. Each carried Level 300 item is labelled with where it was made and whether it was signed at source.
What each step proves, and what it does not
Step 1 without the pinned fingerprints proves nothing about identity. The bundle ships its own public keys.10 A signature that verifies under a key from the bundle shows only that the bundle's producer held the matching private key. Identity comes from pinning, and pinning is only as good as the channel it came through.
The chain in step 2 cannot detect truncation. If $r_1..r_N$ satisfies the chain predicate, so does every prefix $r_1..r_k$ for $k < N$: each retained record still points to its predecessor. A producer, or anyone holding the signing key, can therefore drop the tail and present a bundle that verifies internally. Only step 3, against an $n^\$ and $h^\$ obtained independently, rejects it. Matching $n^\$ alone is not enough: the key holder can sign an alternative history of the same length, which only $h^\$ rejects. Certificate Transparency addresses a related problem in public logs: RFC 9162 says "Merkle consistency proofs prove the append-only property of the tree", and warns of a log "that shows different, inconsistent views of itself to different clients".7 Its answer, like ours, depends on the verifier having a view of the log's state that the log did not hand them in the same breath. RFC 9162 is a design reference here; nothing in this module shows that Beacon implements it.
Step 3 bounds the committed history, not reality. A matching checkpoint shows the record is whole up to what the producer committed to. It does not show every real-world action was recorded. Key custody, checkpoint freshness and resistance to showing different checkpoints to different verifiers are operational properties this procedure does not establish.
Signing proves integrity from packaging onward, not truth. RFC 8032 specifies that verification checks a signature on a message under a public key.6 It follows, as our inference rather than a statement in the RFC, that a valid signature says nothing about the accuracy of the signed content or about what happened to it before it was signed. A Level 300 receipt that was unsigned at creation and is signed inside the 400 bundle is protected against change after packaging, and no more.
The lab: verify the sandbox bundles yourself
The course sandbox, apps/sandbox400/, holds the bundles for this lab. Read apps/sandbox400/README.md first: it lists every Level 400 exercise with its command and its prepared output. Everything in it is synthetic. No model is trained or run, and no person or real system produced any receipt. The keys are course test keys derived from public labels in apps/sandbox400/build_fixtures.py, so anyone can sign with them: a pass shows how the check works, not who signed. The sandbox's ed25519.py is the RFC 8032 reference algorithm, slow and not constant-time; never use it to protect a real key. The verifier is not Beacon, and its README records Beacon compatibility as not checked.
The 400.9 bundles are one log, bw-warranty-gate, under policy 0.3:
| Bundle | Log | Expected result |
|---|---|---|
apps/sandbox400/bundles/bundle-a | seq 1..214 | Accept |
apps/sandbox400/bundles/bundle-b | the same log, cut after seq 163 and re-packaged | Reject at step 3: truncated, 51 records missing |
apps/sandbox400/bundles/bundle-c | the same log, rewritten after seq 150 and re-signed with the evidence key | Reject at step 3: same count, different head |
Each bundle holds MANIFEST.json, MANIFEST.sig, VERIFY.md, receipts.ndjson, checkpoints.ndjson, the two public keys in keys/, policy/policy-0.3.yaml, crosswalk.yaml, known-issues.md, eval/eval-report.json and decisions/decision-record.json, and nothing else. VERIFY.md states the construction, the genesis, this closed inventory and the steps.
Your trust material is the file apps/sandbox400/trust/bw-warranty-gate.trust.json. It stands for material received separately from the bundle, and the verifier trusts only it:
- evidence key fingerprint:
f0c5ef8d809e5d8aea4645adae5f2345280d305a677248ca491efe11b77e75e7 - packaging key fingerprint:
d3e7a4b3db6abc28f7bca08c14c29213f36759cb626b22f1088d9c840d905688 - checkpoint $(n^\, h^\, v^\*)$:
214,6e0450d9ba37f8f02d517431172ed1de020bde42ffb98158864e574b1e2a8679,0.3 - policy hash:
ed8e74af08fe530432698e0defe0410502a416be3d8a41517c64f4169908cd06
Compare these values with the trust file before you run anything. In this lab that comparison stands in for an authenticated channel. In a real audit, a second file in the same download is not a second channel.
Switch off the network. From the repository root:
python3 apps/sandbox400/verify_bundle.py apps/sandbox400/bundles/bundle-a --trust apps/sandbox400/trust/bw-warranty-gate.trust.json
Then run the same command for bundle-b and bundle-c. It prints a line per step, a plain-language reason on failure, and exits 0 (accept), 1 (reject at the first failed step) or 2 (missing or invalid trust material). Work out each result before you compare it with the prepared outputs, apps/sandbox400/prepared/4009-c-verify-bundle-a.txt, -b.txt and -c.txt. Bundle A: 214 signatures valid, chain continuous from seq 1, checkpoint at 214 matching head and policy 0.3, one carried Level 300 item not authenticated at source. Bundle B: steps 0 to 2 pass with 163 signatures; step 3 fails with 51 records missing. Bundle C: steps 0 to 2 pass with 214 signatures; step 3 fails because the record hash at seq 214 is not $h^\*$. B and C were re-signed by the producer, which holds the keys, so only the checkpoint received separately catches them.
Next, make tampered copies of Bundle A and check that each fails at its step. This writes the copies to apps/sandbox400/dist/tampered/:
python3 apps/sandbox400/sandbox400.py tamper --bundle apps/sandbox400/bundles/bundle-a --trust apps/sandbox400/trust/bw-warranty-gate.trust.json --case key-swap --case bad-manifest-signature --case altered-record-repacked --case extra-file-repacked --case suffix-cut --case fork-resigned --case nested-payload --case payload-in-eval-report --case person-in-crosswalk
| Case | What it changes | Fails at |
|---|---|---|
key-swap | Records and manifest re-signed with impostor keys, which the bundle now carries | Step 1: the evidence key is not the pinned key |
bad-manifest-signature | One character of MANIFEST.sig changed | Step 1 |
altered-record-repacked | One receipt's decision flipped, manifest re-signed | Step 2: that record's signature is invalid |
extra-file-repacked | An extra notes file, manifest re-signed to list it | Step 0: a file outside the inventory |
suffix-cut | The last 12 records dropped, the rest re-signed | Step 3: truncated |
fork-resigned | One decision flipped and the whole log re-signed | Step 3: head does not match |
nested-payload | A request-text field added inside one receipt body, re-signed | Step 2: field not in the closed schema |
payload-in-eval-report | A model-output list added to the eval report | Step 5: field not in the closed schema |
person-in-crosswalk | An email address added as a comment | Step 5: the read-through scan flags it |
The case code and the step each must fail at are in apps/sandbox400/tamper.py. The README's other tamper command, with no --bundle, runs every case against the 400.P project bundle instead; that is 400.P's exercise, with its own trust file.
How the sandbox makes the content step concrete, as its README describes it:
- Signed inventory.
MANIFEST.jsonlists every file except itself andMANIFEST.sig, with its SHA-256 and size;VERIFY.mdis listed too. A file present but unlisted, listed but absent, or changed fails. So an instruction page changed after signing fails. - Strict parsing. All JSON the verifier reads, including the trust file, rejects duplicate keys at every depth before verification. A parser that keeps only the last of two equal keys would let a payload value sit, unchecked, in front of the real one.
- Closed schemas. Receipts, the eval report and the decision record fit closed schemas, with no free-text fields; the two artifacts must be the ones the receipts name. People appear only as a role and an opaque work-record reference.
- Screen and read-through. The text files are screened for prompt-, output- or person-like content. The README is plain that these checks "cannot prove that free text contains no sensitive content; a person should still read it". Step 5 is complete only when you have read
VERIFY.md,crosswalk.yaml,known-issues.mdand the policy file yourself. An ACCEPT is the result of the script's automated checks, not of yours.
Kofi Mensah, an auditor outside Big Widget, is the worked example: he holds the same fingerprints and checkpoint, received through an authenticated channel before the bundles arrived, and runs A and B.
The verification statement
The worksheet asks for one sentence, with the verifier's name and the date.8 It should be a conjunction of the checks actually passed, scoped to the bundle and the trust material used, with each unverified carried item named. Kofi's:
"I, Kofi Mensah, on 18 November 2026, verified Bundle A offline by its VERIFY.md: all receipt signatures verify under the pinned Big Widget evidence key and the manifest signature under the pinned packaging key, the chain is continuous from sequence 1 to the expected checkpoint (214, hash matching the value received separately) under policy version 0.3, and I found no payload content by the verifier's inventory and schema checks and my own read-through of its free-text files; the carried Level 300 receipt was unsigned at source and I did not verify it."
Overclaims to reject: "the system is lawful", "the decisions were correct", "the record is complete" (without naming the checkpoint), or any assurance level. The worksheet's own declaration is "I have not claimed any level for any system."8 The statement is the verifier's own, signed by them; no tool or agent signs it. Interpreting assurance levels is module 500.6's job.
Write yours only for what you ran. When Bundle A is accepted, the sandbox verifier prints a statement template with blanks for your name, the date and how you received the trust material; fill it in only after your own read-through. If you ran the lab offline, your sentence can say so, and should name the bundle, the trust file and checkpoint you used, and that the bundle and keys are course practice fixtures. If you only read this page, or read the prepared outputs instead of running the verifier, say that instead: "I have not verified a bundle" is an honest answer.
Practice
The Foundation's audit-training page offers a Gate Check of yes-or-no questions "each tied to evidence" and a vendor-RFI tool with "a scored knowledge check, self-audit, and verifiable receipt export".11 Keep workplace personal data out of practice runs.
Summary
| Gate | Yes | Fails closed when | Proves | Does not prove |
|---|---|---|---|---|
| Premise | Get to Yes | No valid receipt: allow, current rule hash, every check passed, registered validator key, approver outside authors and holder | An independent party recorded passing legal, statistical and cohort evidence for this rule version | That the evidence was sufficient; that the law will not change |
| Protected attribute | Get to Yes, Stay at Yes | A protected feature is present, or no valid passing proxy-test receipt exists | The feature list was reviewed and passed the proxy test; each adverse notice waits for a named reviewer | That no unknown proxy exists |
| Stranger's check | Recover to Yes | Key not pinned, signature missing or invalid, chain broken, checkpoint (count, head hash, policy version) mismatch, payload present | Integrity, signer identity (via the pinned key) and completeness up to the committed checkpoint | That any decision in the record was right; that every action was recorded; the accuracy of anything unsigned at source |
Cut as not checked: the worksheet's Robodebt debt count and "unlawful from the start"; the toeslagenaffaire family counts, the low-income risk feature and the cabinet's resignation; the mechanics and timeline of Robodebt's income averaging; the Post Office (Horizon System) Offences Act 2024's effect. The lab runs the course sandbox's A, B and C bundles, its trust file and its tamper cases. That is practice with course fixtures, not a sign-off.