
Project: hold a running system · Standard gear
Reviewed and published, version 1.0.0. As of 18 November 2026. It is not a credential. Big Widget Company, its people, its systems, its policy, its logs and every id in them are made up. The court record, the government record and the guidance documents are real, and each is cited. The court case is given as allegations, not findings; its outcome is not checked. The course sandbox this project's objective names is the offline claims runner in
apps/sandbox/("What you will do" says how to use it).
You will be able to:
- turn a policy into an enforced hold for one sandbox system, with every part the gate asks for, and get it reviewed and marked (300.P-a);
- run that hold as a dry run on the course's synthetic logs, including a missing approval record and the cases that must stop, and report what it caught, when, and what it could not count (300.P-b).
What you will do
Figure data as text
Report the paper run
- REPORT · course procedure · Describe catches and gaps — what, when, stops and pages; reference receipts. Say what you could not count.
- QUALIFY · course procedure · Say which results are paper and which come from sandbox traces — a paper result is not evidence that a connected stop refused anything; a sandbox trace is evidence only about the course’s practice executor. Name untested rules.
- RECOVER · course procedure · If something went wrong: incident note — what happened, when, who was told, what changed; keep references only.
Course dry-run reporting procedure; NIST incident communication (s2) and NCSC incident planning (s3) support the incident note.
Objective: 300.P-b. Text equivalent of figures/report-the-paper-run.svg.
This is the Level 300 project. In 300.8 you learned the operate gate. In 300.4 you wrote an authority table, in 300.5 you counted with honest denominators, and in 300.9 you built a monitor, a stop and a receipt. Here you put all of it together on one system.
You bring two things:
- A hold for one sandbox system: a written set of rules that says who answers for each action, what may run, what stops it and what record it leaves (300.P-a).
- A dry-run report: what the hold would have done when you ran it against the course's synthetic logs (300.P-b).
Everything is synthetic. You work on the course's own practice files, never on a real system, real logs or anyone's private repository. No real system is changed. This is a course rule, decided on 18 November 2026: sandbox, sample policy and logs are synthetic fixtures owned by the course.
Paper first, then the sandbox. The objective asks for an enforced hold in the course's sandbox: a stop that a running executor actually obeys. You work in two steps.
- On paper. Write the hold and run it by hand on the synthetic logs in this module, which match the course's practice prompts. A hand run shows what your rules would decide for each logged action. It does not show that any executor refused anything, so on its own it does not meet the "enforced" part of 300.P-a.
In the sandbox. The course sandbox is an offline Python runner in this repository. Its instructions are in
apps/sandbox/README.md, section "300.P · Hold a running system". Copyapps/sandbox/claims-policy.json, put your hold's holder, expiry, alert route, authority table, budgets and heartbeat setting into the copy, and run the cases from the repo root in the order given under "Run it in the sandbox" below. That order includes a named restart by the holder between cases, because a stop in the sandbox persists from one run to the next.Each run writes a JSON trace and prints where it is. The trace holds the gate's
events, itsreceiptsand a separateexecutor_log. The refusal is shown when a refused proposal has a receipt and no entry in the executor log. That is what makes the sandbox run evidence of enforcement and the hand run only preparation for it.
The sandbox is still a practice system. It uses a synthetic clock, simulated pages and a local recording executor; no real letter, payment, call or message is sent, and approvals, callbacks and restart names are fixture inputs, not real authentication. Its second-channel rules are fixed: you can change the settings in your copy of the policy, not how the call-back works. So a sandbox trace shows that the course's executor obeyed your settings. It does not show that any real system would. In your report, say which results come from paper and which from sandbox traces.
Your starting policy. Use the sample policy below. If you finished 200.P, you may use your own 200.P policy instead, but only once it is share-safe (nothing private, confidential or about a real person) and you have mapped each of its rules onto the synthetic system's actions. A rule you cannot map is a gap you report, not a rule you guess at. If you cannot do both, use the sample policy.
People make every decision here. You write the hold. A second person reviews it. A host marks it. The model in the sandbox decides nothing about your project.
Where this sits in the three yeses. Module 100.9 taught Get to Yes, Stay at Yes and Recover to Yes. Your hold is the Get to Yes for one system: it says exactly what may run, how far, and who answers for it. That yes is conditional. The gate gives one of four answers before every action: go ahead, only this far (the budget), wait for a person (an approval, or a tripped stop) or no. Answering every time is Stay at Yes. The stops, pages and incident notes start Recover to Yes; a stopped system is not yet a recovered one.
The misconception this corrects: "A system with a dashboard is being held." A dashboard shows you numbers. A hold says, for each action, who answers for it, what may run, what stops it automatically, and what record it leaves. It counts as enforced only once the stop has been seen to refuse an action in a running system. Paper shows the rules; the sandbox trace shows the refusal.
The sandbox system and the sample policy
The system is Big Widget's warranty-claims assistant, the same one the course's practice prompts use. It reads incoming warranty claims from dealers and flags the claims it thinks the warranty does not cover. Three reviewers, R1, R2 and R3, then clear the flagged claims: each clearance is one reviewer signing off one flagged claim.
Sample policy (made up, for practice):
The warranty-claims assistant may read claims and the warranty terms, and flag claims for review. A named reviewer must open the claim file and decide every flagged claim. No decision letter goes to a dealer without a reviewer's decision on record. Goodwill payments to dealers above the payment limit need the finance controller's approval and a call-back to the dealer's number on file. The assistant never changes a dealer's bank details and never changes its own rules. Dana Okafor, head of warranty operations, holds the assistant. Her approval is reviewed every six months.
Your job is to turn those sentences into a hold the sandbox can enforce, part by part.
The hold, part by part
The course's gate has eight parts. The objective for this project lists seven that the host checks; the eighth, judgment, sits inside the authority table (step 2). Write each part for the warranty assistant.
1. Holder and expiry
One named person holds each action, and the approval has a date it runs out. "The warranty team" is not a holder. A team can share the work, and an on-call rota can answer an alert at 3 a.m. and escalate it. But someone has to answer afterwards for why this action was allowed to run, and that is one named person. Keep the two jobs apart: the rota handles the alert; the holder answers for the decision.
NIST's AI Risk Management Framework, a voluntary US framework, asks that "Roles and responsibilities and lines of communication related to mapping, measuring, and managing AI risks are documented and are clear to individuals and teams throughout the organization" (GOVERN 2.1).1 The holder line is that idea at the size of one system.
Example (course rule, made up). Holder: Dana Okafor, holder reference H-017. Approval A-17, granted Day 0 by the head of operations (holder reference H-002), expires at the end of Day 180. Alerts go to the warranty on-call rota, which escalates to Dana if nobody answers. Only Dana, or someone she names on the record, restarts a stopped action. If the holder line is ever empty, the page goes to the rota and to H-002, who must name a new holder on the record. (The practice files count in days, not calendar dates, so nothing here looks like a real date.)
2. Authority table
Every action the system can take goes in one of three rows: alone, needs a named approver, or never. Anything not listed is denied. This is the 300.4 table, now enforced.
| Action | Authority | Gate's answer | Notes |
|---|---|---|---|
| Read a claim and the warranty terms | Alone | Go ahead | |
| Flag a claim for review | Alone | Go ahead | No budget in this sample: the budgets count clearances only (step 4) |
| Clear a flagged claim | Named reviewer | Wait for a person; then only this far | The reviewer opens the claim file and decides. The model informs; the reviewer decides (judgment). Budgeted (step 4) |
| Send a decision letter to a dealer | Named reviewer, plus out-of-band check | Wait for a person | Irreversible (a disclosure): see step 3 |
| Pay a goodwill payment above the payment limit | Named approver (finance controller), plus out-of-band check | Wait for a person | Irreversible: see step 3 |
| Change a dealer's bank details | Never | No | |
| Change its own rules, budget or authority table | Never | No | |
| Anything else | Never | No | Unlisted means denied |
Judgment. Where a model informs but a qualified person decides, write it in the table, as in the "clear a flagged claim" row. A person who signs without looking is not deciding (300.8).
If you want flags budgeted too, add a separate flag budget with its own counter, its own limit and its own test. Do not charge flags to the clearance budget: that counter, and the synthetic log, count clearances only.
3. Out-of-band rule for irreversible actions
An irreversible action needs confirmation on a channel the person or system making the request does not control. The sample system has two irreversible actions, and each needs a confirmation that fits it.
The warning case is from Hong Kong. In a written reply to its Legislative Council on 26 June 2024, the Hong Kong Government reported that "In end-January 2024, the Police received one fraud case involving the use of deepfake technology".2 It said "the pre-recorded video conference was generated using downloaded online public video clips", that "the informant eventually gave authorisation to transfer funds to five local bank accounts", and that the informant "suffered a loss of about HK$200 million".2 It added: "Since the meeting was pre-recorded, there was actually no interaction between the informant and the fraudster."2 The record does not name the organisation, and this course does not either. In general advice in the same reply, not tied to this case, the Police listed "verifying with a phone call when a relative or a friend makes a request for remittance via a video or an audio recording".2
Course rule, for the warranty assistant:
- Goodwill payment above the limit. Paid only after the finance controller approves it and someone calls the dealer back on the number already on file, never a number given in the request. A request that arrives by email, chat or video call, however familiar the face, is not the confirmation.
- Decision letter. The requester here is the assistant, which drafts the letter and asks to send it. The confirmation is the reviewer's decision, entered by the reviewer under their own login in the claims system, which the assistant cannot write to. The letter goes only to the dealer's address already on file. A decision that appears only in the assistant's own output is not the confirmation.
A bank-style call-back is not needed for every disclosure. What matters is that the confirmation travels on a channel the requester cannot fake. Write, for each irreversible action, which channel that is and why the requester cannot control it.
4. Aggregation budget
Small acts can add up to a high-stakes act in volume. Set a budget per reviewer and system-wide, per hour, and say what happens when either is exceeded. Count peaks, not averages.
The warning is the Cigna PXDX case. A class action complaint filed on 24 July 2023 alleges that "over a period of two months in 2022, Cigna doctors denied over 300,000 requests for payments using this method, spending an average of just 1.2 seconds “reviewing” each request".3 It alleges that Cigna's doctors "reject claims on medical grounds without ever opening patient files".3 These are a pleading's allegations, not findings; the complaint takes the figures from a ProPublica report it cites.3 How the case has gone since is not checked. The course uses those totals only as a warning. It never divides them into an hourly rate or uses them in your arithmetic.
Example budget (course rule). At most 50 clearances per reviewer per hour, and at most 120 system-wide per hour. Only clearances count. When the per-reviewer line is reached, that reviewer's next clearance is refused for the rest of the hour; when the system-wide line is reached, everyone's is. Either way the holder is paged. These are the numbers in the course's practice prompts. (300.9's stop instead stays tripped until the holder restarts it. Either rule is fine if your hold says which one it uses and your dry-run counts follow from it. The sandbox adds a separate run-level stop on top of either rule; see "Run it in the sandbox".)
5. Fail-closed stop
Write what stops the system, and make it fail closed: if the holder, an approval, a record or the monitor is missing or out of date, the action does not run. It waits, and a person is paged.
Say when each failure becomes knowable. A missing record or an expired approval is visible the moment the action arrives. A silent monitor is not: the gate only learns it has gone quiet once its last heartbeat is too old. Course rule: the monitor counts as up while its last heartbeat is less than 60 seconds old. For up to a minute after it goes quiet, actions are decided on a count that may be stale. That delay is the price of the 60-second setting; say it in your hold.
NIST's framework asks that "Mechanisms are in place and applied, and responsibilities are assigned and understood, to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use" (MANAGE 2.4).1 A stop is only real once you have seen it stop something in a running system. The dry run shows what it would do; the sandbox runs (see "Run it in the sandbox", below) are where you see it.
For the warranty assistant, "stopped" means a flagged claim stays in the queue for a person. Nothing is cleared, denied or paid by default.
6. Content-free audit receipts
Every action the gate decides leaves a receipt with: what happened, who held it, which approval, the result, and how to undo it. A receipt holds references, never private content: no personal data, no prompts, no model output, no document text, no secrets. That includes the holder. The receipt carries the holder's reference (H-017), not her name. Dana's name lives in the approval record, which is kept under control and shown only to people allowed to see it. A claim reference replaces the dealer's or customer's name; an approval id replaces the approval email.
References are not anonymous. Someone allowed to follow them can find the person or the claim behind them; that is the point, because an authorised checker must be able to. They also do not stop inference. Someone who already knows that C-4471 is their own claim can read its outcome off a receipt without ever opening the lookup table, and a stable reference links every receipt that carries it. So keeping content out of the receipt does not make the receipt public. Receipts are still sensitive: control who can read and share them, as well as who can use the lookup table, and do not publish them on the strength of being "content-free".
The UK National Cyber Security Centre's guidelines for secure AI system development, published with the US Cybersecurity and Infrastructure Security Agency and partner agencies, say: "You treat logs as sensitive data and implement controls to protect their confidentiality, integrity and availability."4 They also advise logging inputs "to enable compliance obligations, audit, investigation and remediation in the case of compromise or misuse."4 A content-free receipt lets someone audit the decision without copying the sensitive record into yet another place.
Example receipt (made up; the log gives no claim references, so C-4471 is illustrative).
| Field | Entry |
|---|---|
| Receipt | RC-0193 |
| When | Practice Day 90, 11:00 hour |
| What | Clear flagged claim, claim ref C-4471 |
| Who acted | Reviewer R1, through the assistant's own system identity |
| Holder | H-017 |
| Approval | A-17 (valid to end of Day 180) |
| Result | Stopped: R1 over the 50-per-hour budget; holder paged |
| Undo | None needed; claim C-4471 is still in the review queue |
These are unsigned receipts. They are the artifact that runs up the ladder: 400.P packs them into an evidence bundle that a stranger can check. Even then, a receipt shows what the gate decided, not that the decision was right.
7. Coverage: an honest number
Say how many of the system's actions the gate actually saw, out of how many, and where the "out of how many" comes from. The denominator must come from a record other than the gate's own receipts, such as the system's action log. If no such record exists, write "cannot count yet". That is an honest answer, not a failure (300.5).
Make both sides count the same thing. Write down three choices before you count:
- The unit. Actions that actually ran (executions), or actions that were asked for (attempts). These are different populations, and they need different independent records.
- The scope and window. Which actions, and which period: for example,
claim.clearon Day 90. - The match. For each action in the independent record, find the gate's receipt for that same action, by an action id both records share, and check that the receipt's decision is allow. Count each action once.
This course uses one coverage number: permit-matched execution coverage. It is the number of executed actions whose own receipt says allow, out of the executed actions in the independent record. An action that ran with no receipt for it is unchecked. An action that ran even though its own receipt says deny is worse: it ran after a refusal. Neither counts towards coverage, and you list the second kind separately as a failure of the stop, not hide it inside the coverage number. A legitimate refusal never appears in the independent record of executions, because it never ran.
Do not divide the gate's receipt total by the log's total: a receipt for a refused request, or for some other action, can stand in for an execution the gate never saw. Two refusals and two unchecked executions would show 2 out of 2 that way, when the gate permitted none of the executions (0 out of 2). If you also want attempt coverage, label it separately and say where the attempts were independently recorded.
To check your counting, run python3 apps/sandbox/sandbox.py --trace-dir practice-run claims coverage, in the sequence under "Run it in the sandbox" (after the restart that follows claims authority; coverage itself leaves the run ready). Its trace works through small made-up cases: a matched allow with a duplicate and a legitimate refusal (1 of 1), unrelated refusals (0 of 2), an execution whose own receipt says deny (0 of 1, listed under executions_after_refusal), the wrong scope or day, and four cases that must come out as "cannot count yet" (no independent log, no shared id, a missing execution id, a denominator made of receipts). Work out each answer before you read the trace.
Cross-check against NIST
NIST's framework is voluntary and this course does not claim your hold meets it. It is a useful checklist that the hold names an owner, a measure and a response:
| Hold part | NIST AI RMF 1.0 |
|---|---|
| Holder and expiry | GOVERN 2.1: roles and responsibilities documented and clear1 |
| Budget and monitor | MEASURE 2.4: "monitored when in production"1 |
| Stop | MANAGE 2.4: "supersede, disengage, or deactivate"1 |
| Receipts and incident note | MANAGE 4.3: incidents and errors communicated, and "Processes for tracking, responding to, and recovering from incidents and errors are followed and documented"1 |
The dry run
A dry run replays synthetic logs through your hold and records what the hold would have done. No real system acts. A paper exercise does prove something: it shows whether your rules give a clear answer for each logged action, and what the stops and receipts should look like. It does not prove that a live system obeys them, how fast a stop acts, or that the log is complete. Say that in your report.
Run every condition your hold claims to handle. A condition you wrote down but never ran is written, not tested: report it that way.
The synthetic log
The numbers are invented for practice and say nothing about any real company. Each cell is the number of flagged claims a reviewer cleared in that hour, on Practice Day 90.
| Hour | R1 | R2 | R3 | System total |
|---|---|---|---|---|
| 09:00 | 12 | 10 | 11 | 33 |
| 10:00 | 14 | 9 | 13 | 36 |
| 11:00 | 71 | 12 | 10 | 93 |
| 12:00 | 8 | 11 | 9 | 28 |
The log gives hourly totals only: no claim references, no minutes, and no order between reviewers.
Run 1: the per-reviewer budget
Check every hour, for each reviewer and for the system total:
- Per reviewer (50 per hour). R1 cleared 71 in the 11:00 hour. That breaks the budget. Every other cell is 14 or less.
- System-wide (120 per hour). The highest total is 93, at 11:00. The system-wide line holds.
- Why averages hide it. Over four hours the three reviewers cleared 190 claims in 12 reviewer-hours, about 16 each. R1's own average is 105 in four hours, about 26. Both are well under 50. Only the hourly, per-reviewer count shows the breach: R1, 71 clearances against a limit of 50 in one hour.
What the hold does: R1's first 50 clearances in the 11:00 hour fit the budget. The hold refuses the 51st and every later R1 clearance that hour, so 21 claims stay in the queue, and it pages the holder. The other reviewers carry on, because the system-wide line still holds. Of the 190 clearances in the log, the hold would allow 169 and stop 21.
When did it catch it? Somewhere in the 11:00 hour, at R1's 51st clearance. The log gives hourly totals only, so you cannot count the exact minute. Say so.
Run 2: the system-wide budget
Run 1 never tests the system-wide line: the busiest hour is 93, well under 120. A hold with the system-wide line switched off would give the same Run 1 result. So the course gives a second synthetic fixture, invented for practice:
| Hour (Practice Day 91) | R1 | R2 | R3 | System total |
|---|---|---|---|---|
| 11:00 | 45 | 45 | 45 | 135 |
No reviewer reaches 50, so the per-reviewer line never fires. The system total is 135, over 120. The hold allows the first 120 clearances of the hour, refuses the 121st and every later one that hour (15 claims stay in the queue), and pages the holder. Which reviewers' clearances were refused depends on the order they arrived in, and the fixture does not record order: write "cannot count yet".
Run 3: the missing approval record
One clearance has no approval record: the reviewer's decision is not on file. Your hold must not allow it. It stops that action, and pages the holder. Its receipt says which record is missing:
| Field | Entry |
|---|---|
| What | Clear flagged claim, claim ref C-4502 |
| Holder | H-017 |
| Approval | Missing: no reviewer decision on record |
| Result | Stopped; holder paged |
| Undo | None needed; claim C-4502 is still in the queue |
A hold that logs a warning and lets the action through has failed open.
Run 4: the cases that must stop
Each of these must fail closed: the action does not run, the receipt names the cause, and a person with the authority to fix it is paged. The system does not carry on until someone notices.
- Expired approval. Approval A-17 ran to the end of Day 180, and the log is from Day 181. The next action of any kind that A-17 covered is refused; the receipt says "Stopped: approval A-17 expired end of Day 180". Dana, or whoever can renew or replace A-17, is paged. Every later action under A-17 is refused too, until a renewal is on the record.
- Holder missing. On Day 150 Dana moves to another job, and the holder line is emptied with no replacement named. The next action under the hold is refused; the receipt says "Stopped: no holder on record". The page goes to the on-call rota and to H-002, who must name a new holder on the record.
- Monitor silent. On Day 92 the budget monitor's last heartbeat arrives at 11:20:00, then nothing. A clearance at 11:20:30 is decided on the count the gate already has, and on every other check, because the gate cannot yet know the monitor is gone. That is not an automatic yes: if the cached count already shows R1 at 50 that hour, the clearance is refused for the budget. Only the monitor-unavailable stop has not started. From 11:21:00, when the last heartbeat is 60 seconds old, every clearance is refused; the receipt says "Stopped: budget monitor unavailable"; the holder is paged. If the monitor comes back, clearances stay stopped until the holder restarts them.
Run it in the sandbox
Do the four runs by hand first and write down what you expect. Then run the same cases in the course sandbox, from the repo root.
A sandbox stop persists. The sandbox keeps a run-level stop in practice-run/stop-state.json. Any stop inside a case (a budget stop, a missing record, the holder's pause) also latches this outer stop, and it stays latched when the program exits and starts again. The next case will not run until the policy holder, H-017, or a delegate H-017 has recorded for the current stop, records a restart with a name and a reason. Anyone else's restart is refused and the run stays stopped. So between cases you act as the holder: read the trace, then record the restart. Use this sequence, copied from apps/sandbox/README.md, section "300.P · Hold a running system", with one state path throughout:
python3 apps/sandbox/sandbox.py --trace-dir practice-run status
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims authority
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed authority"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims coverage
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F1
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F1-live
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F1-live"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F2
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F2-live
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F2-live"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F3
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F3"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-expired
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-holder
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F4-holder"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-monitor
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F4-monitor"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-monitor-cap
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F4-monitor-cap"
python3 apps/sandbox/sandbox.py --trace-dir practice-run status
If status shows the run still stopped from an earlier exercise, review that stop and record the holder's restart first. To use your own settings, add --policy YOUR_COPY.json after the case, and pass the same --policy to status and restart; if your copy names a different holder.ref, that holder restarts. Do not delete the state files or switch to a fresh directory to get past a stop: that bypasses the stop rather than releasing it, and the README's rule is one state path per run.
If you skip a restart, the case does not run. It exits with input error and its trace says persisted stop requires a named restart. That trace is a preflight refusal, not the case's receipts or executor log; record the restart and run the case again before you report it.
Two layers of stop. The rules in this lesson, and the table below, describe what happens inside a case: a budget stop ends at the hour boundary and R2 and R3 carry on; the holder or her recorded delegate can restart a scoped stop inside the fixture. The run-level stop sits outside those rules. It holds the whole claims run until the holder's restart, even after the in-case stop has ended, and it does not change the case's counts (Run 1 is still 169 allowed and 21 stopped). The fixture's project_delegates entry drives the in-case delegate check only; it does not let H-018 restart the run-level stop unless H-017 records a delegation for that stop.
What the run-level stop does not prove: the names are synthetic references, not authenticated identities. The state file, its .head file and an append-only .journal are hash-chained, so an edited, truncated, deleted or restored older copy of the state is detected and blocks the run. Someone who can rewrite or restore all three files together, or delete all three and every earlier claims output, can still defeat it. It is local tamper evidence for one writer at a time, not a signature or production durability.
| Run | Case | What to check in the trace |
|---|---|---|
| 1 | claims F1, then claims F1-live | Day 90: 169 allowed, 21 stopped. Each of the 21 refusals has a receipt and no executor entry. R1 can clear again in the next hour. |
| 2 | claims F2, then claims F2-live | Day 91: 120 allowed, 15 stopped; no reviewer reaches 50. |
| 3 | claims F3 | C-4502 has no reviewer decision: refused, H-017 paged, a receipt with no approval reference. A reviewed restart does not replay the original proposal. |
| 4, expired | claims F4-expired | Refused on Day 181 and paged until a renewal is recorded. |
| 4, holder | claims F4-holder | The rota and H-002 are paged; only an appointment on the record clears the stop. |
| 4, monitor | claims F4-monitor, then claims F4-monitor-cap | At heartbeat age 30 the cached count and every other check decide; with R1 already at 50, the clearance is refused. From age 60, clearances stop. A returning heartbeat does not restart them; only the holder or her recorded delegate does. |
| Part 3 | claims authority | Forbidden and unlisted actions are refused; the letter and the above-limit payment each need their own second-channel record. |
Two of these need care. F1 and F2 are summaries of this module's hourly logs, not executor traces: they invent no claim references, minutes or reviewer order. F1-live and F2-live replay new, ordered, made-up events with the same hourly totals, so their claim references, times and which reviewer was refused in Run 2 are facts about those practice files, not about the Day 90 or Day 91 log. Cite the fixture and the receipt references you read. Any condition you did not run in the sandbox is written, not tested.
What you could not count
The report ends with the honest gaps. For these logs, at least:
- Whether reviewers opened the files. The log counts clearances. It does not record whether each reviewer opened the claim file, or how long they spent. So you cannot tell real review from a rubber stamp from this log. Write "cannot count yet", and add the field to the log design. The missing field is not proof of a rubber stamp either.
- The exact time of each breach, and which reviewers Run 2 stopped (hourly totals only, no order).
- Receipts for each event. A hand run over hourly totals gives counts per hour (Run 1: 169 allowed, 21 stopped), not a receipt for each clearance: the log has no claim references to put in them. Write receipts for the events you can describe (R1's 51st clearance, C-4502, the first Day-181 action, the first refused action in each must-stop case) and report the rest as counts. Do not invent claim references or times to fill the gaps. The sandbox's
-liveruns issue one receipt per decision, but for their own made-up events, not for the Day 90 log. - Coverage. The replay accounts for all 190 clearances in the Day 90 log: 190 of 190. That is the coverage of your replay over its input, not permit-matched coverage of actions a running gate saw, and it says nothing about actions the log never recorded. The flags and letters are not in this log at all, so for those: "cannot count yet".
The dry-run report
One page:
- What the hold caught: R1's breach at 11:00, the system-wide breach on Day 91, the missing approval, and each must-stop case.
- When: the hour, and the receipt id for each catch.
- What it did: the stops and pages, with receipts.
- What it could not count, and why; and any condition that is written but not tested.
- Which results are paper and which come from sandbox traces. A paper result is not evidence that a connected stop refused anything; a sandbox trace is evidence only about the course's practice executor.
- What you would change in the hold or the log.
When something goes wrong, add a short incident note: what happened, when, who was told, and what changed. Keep it to references, as in the receipts. NIST asks that "Incidents and errors are communicated to relevant AI actors" (MANAGE 4.3)1, and NCSC asks for incident plans that "reflect different scenarios and are regularly reassessed".4 Writing the note is part of Recover to Yes. The system is back at yes only when the fix has been tested and the holder restarts it on the record.
Review and marking
1. Readiness check (auto-graded, first). Short questions on the hold and the dry run, for example: given a sample hold with no expiry, pick the line that fails the gate; or, given a log line with no approval record, say whether the hold allows, stops or pages, and what its receipt says. You pass this before anyone marks your project. It is a check before human marking, not a mark.
2. A second person reviews it. Ask someone who may see this practice work to check:
- every action in the authority table has a row, and unlisted actions are denied;
- the holder is a named person with an expiry, and alerts have a route that does not depend on the holder answering every page;
- each irreversible action has an out-of-band check on a channel the requester does not control;
- the budget counts per reviewer and system-wide, by hour;
- the stop fails closed in every run, and any condition not run is reported as written, not tested;
- no receipt carries personal data, a name, prompts, output or document text, and the receipts are still treated as sensitive;
- the coverage number matches executed actions one by one against an independent record and counts only those whose own receipt says allow, lists any action that ran after a refusal, or says "cannot count yet";
- the report separates paper results from sandbox traces.
They sign and date it with what they found. Their signature says they reviewed it. It is not an approval from any company.
3. A host marks it on the Foundation's Thursday call. The host checks that every hold part is present, the readiness check passed, and your dry-run receipts match the synthetic logs and the sandbox traces. You do not have to host a call. A host marks a paper run as a paper run: written expected receipts are not evidence of connected enforcement. The enforced part of 300.P-a is marked from sandbox traces.
What the host's mark means. It means a host checked your hold and dry run against this table. It is not a certification, a vendor credential, or proof that any real system is safe or lawful.
| Check | Passes when |
|---|---|
| Holder and expiry | A named person, an approval id and an expiry for every action; an alert route and escalation |
| Authority table | Every action in one of alone, approver, never; unlisted is denied; judgment written in |
| Out-of-band | Each irreversible action needs a confirmation on a channel the requester does not control |
| Budget | Per reviewer and system-wide, per hour, with what happens when exceeded |
| Stop | Fails closed when a holder, approval, record or monitor is missing; says when each becomes knowable |
| Receipts | What, holder, approval, result, undo; references only |
| Coverage | Unit, scope and window stated; executions with their own allow receipt out of an independent record, executions after a refusal listed separately, or "cannot count yet" |
| Dry run | All four runs done on paper and in the sandbox; receipts match the synthetic logs and traces; gaps and untested conditions reported; paper and sandbox evidence kept apart |
Further learning (public university course materials)
These are optional background, not sources for this module. Reading public course pages does not mean you are enrolled in, or earn credit for, either course.
- Carnegie Mellon University, Machine Learning in Production (https://mlip-cmu.github.io/) (300.P-a, 300.P-b).
- Stanford University, CS 329S: Machine Learning Systems Design (https://stanford-cs329s.github.io/) (300.P-a).