
Project: hold a running system · Technical gear
Reviewed and published, version 1.0.0. As of 18 November 2026. It is not a credential. Big Widget Company, its people, its systems, its policy, its logs and every id in them are made up. The court record, the government record and the guidance documents are real, and each is cited; the court case is allegations, not findings, and its outcome is not checked. The configuration and code below are the course's own illustrations, not any product's format. The course sandbox the objective names is the offline claims runner in
apps/sandbox/.
Scope and constraints
Figure data as text
Report the paper run
- REPORT · course procedure · Describe catches and gaps — what, when, stops and pages; reference receipts. Say what you could not count.
- QUALIFY · course procedure · Say which results are paper and which come from sandbox traces — a paper result is not evidence that a connected stop refused anything; a sandbox trace is evidence only about the course’s practice executor. Name untested rules.
- RECOVER · course procedure · If something went wrong: incident note — what happened, when, who was told, what changed; keep references only.
Course dry-run reporting procedure; NIST incident communication (s2) and NCSC incident planning (s3) support the incident note.
Objective: 300.P-b. Text equivalent of figures/report-the-paper-run.svg.
Deliverables:
- A hold specification for one sandbox system (300.P-a): holder and expiry per action, an authority table with default deny, an out-of-band rule for every irreversible action, per-actor and system-wide rate budgets over fixed time buckets, fail-closed stop semantics with the point at which each failure becomes detectable, a content-free receipt schema, and a coverage metric with an externally sourced denominator or the value
cannot count yet. - A dry-run report (300.P-b): the hold evaluated over the course's synthetic fixtures, including a per-actor breach, a system-only breach, a missing-approval-record case and the must-stop cases (expired approval, missing holder, unavailable monitor), listing the receipts that can be written from the fixtures, the bucket in which each violation was detected, the quantities the fixtures do not support, and any
fail_closed_whencondition that was specified but not exercised.
Constraints (course rules, decided 18 November 2026): synthetic fixtures only, owned by the course; no real system is changed; no real logs and no private-repository excerpts.
Paper evaluation, then enforcement. 300.P-a asks for an enforced hold in the course-owned sandbox: a gate in the action path with an executor that obeys its deny branch, and observable allow, deny and receipt paths. Evaluate the hold by hand first over the tables in this module, which match the course's practice prompts. That establishes what the specification would decide on each fixture event, not that any executor refused an action. The enforced part is shown in the course sandbox, an offline standard-library Python runner documented in apps/sandbox/README.md (section "300.P · Hold a running system"). Copy apps/sandbox/claims-policy.json, encode your holder, expiry, alert route, authority table, budgets, heartbeat limit and restart delegate in the copy, and run the cases from the repo root in the sequence under "Sandbox runs" below, which places a named holder restart between cases because the CLI's stop persists across processes. Each run writes a JSON trace with the gate's events, its receipts and a separate executor_log. Enforcement is evidenced when a refused proposal has a receipt and no executor entry. A script of your own is not the course sandbox; a reviewer decides what it counts for.
Limits of that evidence: the runner uses a synthetic clock, simulated pages and a local recording executor; approval, callback, holder and restart records are fixture inputs, not authentication; fixture counters and scoped latches start fresh in each invocation, while the CLI adds a separate run-level interlock that persists across processes until an authorized named restart (see "Sandbox runs"); and the second-channel rules are fixed in the runner, not read from your policy. A trace evidences that the course executor obeyed your settings on course fixtures, not the behaviour of any production system. The report states which results are paper evaluation and which are sandbox traces.
Input policy: the sample policy below, or your 200.P policy if and only if it is share-safe and each rule is mapped to an action in the sandbox action set. An unmapped rule is a gap you report, not a rule you guess at.
A dashboard is an observability surface, not a hold. The hold is the enforcement path: a decision function evaluated before each action, with a default of deny, plus evidence that its deny branch executes under the failure conditions it claims to handle.
Three yeses. The specification is Get to Yes for this system: a conditional, scoped approval, not a blanket licence. The gate's four answers map to the spec: allow (go ahead), budget constrain (only this far), hold (wait for a person: an approver, or a tripped stop) and deny (no). Evaluating before every action is Stay at Yes. Deny-and-page, incident notes and a holder restart after a tested fix are Recover to Yes; a stop alone is not a recovery.
The system under hold
The sandbox system is Big Widget's warranty-claims assistant. Actions:
| Action id | Description | Reversible |
|---|---|---|
claim.read | Read a claim and the warranty terms | n/a (read) |
claim.flag | Flag a claim for human review | Yes (unflag) |
claim.clear | A named reviewer clears a flagged claim | Yes, until a letter is sent |
letter.send | Send a decision letter to a dealer | No (disclosure) |
payment.goodwill | Pay a goodwill payment above the limit | No |
dealer.bank.update | Change a dealer's bank details | n/a (never) |
policy.update | Change the hold, budgets or table | n/a (never) |
Two actions are irreversible: letter.send and payment.goodwill. Both are in scope for the out-of-band rule.
Sample policy (made up): the assistant may read and flag; a named reviewer opens and decides every flagged claim; no letter without a reviewer decision on record; goodwill payments above the limit need the finance controller and a call-back on the dealer's number on file; the assistant never changes bank details or its own rules; Dana Okafor, head of warranty operations, holds the system, with approval reviewed every six months.
Hold specification
An illustrative specification (the course's own format). The holder's name is not in it: the spec, like the receipts, carries the directory reference. The approval record that maps H-017 to a person is kept separately, under access control.
system: warranty-claims-assistant
system_identity: svc-warranty-assistant # its own identity, never a person's login (300.4)
holder:
holder_ref: H-017 # name lives in the controlled approval record
approval_id: A-17
granted_by_ref: H-002
valid_from_day: 0
valid_to_day: 180 # inclusive; day 181 fails closed
alerts:
route: warranty-oncall-rota # answers and escalates pages
escalate_to: H-017 # if the rota does not answer
if_holder_missing: [warranty-oncall-rota, H-002] # H-002 names a new holder on the record
restart_by: [H-017, delegate named on the record]
default: deny # any action not listed is denied
authority:
claim.read: {mode: alone}
claim.flag: {mode: alone} # not budgeted in this sample
claim.clear: {mode: approver, approver_role: reviewer,
requires: [reviewer_decision_record]} # add file_opened_event once logged
letter.send: {mode: approver, approver_role: reviewer,
requires: [reviewer_decision_record, out_of_band_confirmation]}
payment.goodwill: {mode: approver, approver: finance-controller,
requires: [approval_record, out_of_band_confirmation]}
dealer.bank.update: {mode: never}
policy.update: {mode: never}
out_of_band:
letter.send:
confirmation: reviewer decision written by the reviewer's own identity in the claims system
requester_cannot_write: true # svc-warranty-assistant has no write access there
deliver_to: dealer address on file only
payment.goodwill:
confirmation: call-back to the number already on file, not to any number in the request
budgets:
bucket: 1h
counts: allowed claim.clear events # refused events do not consume budget
rules:
- {action: claim.clear, scope: per_actor, max: 50}
- {action: claim.clear, scope: system, max: 120}
on_budget_exceeded: {effect: deny, scope: matching_scope, until: end_of_bucket, page: holder}
monitor:
heartbeat_max_age_s: 60 # up while last heartbeat age < 60 s
fail_closed_when: [holder_missing, approval_expired, required_record_missing, monitor_unavailable]
on_fail_closed: {effect: deny, page: per_alerts, latch_until: restart_by}
receipt:
fields: [receipt_id, action_ref, day, bucket, action, object_ref, actor_ref, holder_ref,
approval_ref, decision, reason, undo]
forbid: [personal_data, names, prompts, model_output, document_text, secrets]
access: restricted # references are linkable; receipts stay sensitive
coverage:
unit: executed actions # attempts, if wanted, are a separate metric
scope: [claim.clear, letter.send, payment.goodwill]
window: per practice day
denominator: executed actions in the system action log (independent of the gate), deduplicated
numerator: denominator actions whose own gate receipt (same action_ref and action) says allow
metric: permit-matched execution coverage
report_separately:
executions_after_refusal: denominator actions whose own receipt says deny # a stop failure
without_matching_decision: denominator actions with no receipt # unchecked
The budget stop here lasts to the end of the bucket. 300.9's stop latches until the holder restarts it. Either is acceptable if the spec says which and the dry-run counts follow from it. Both describe in-fixture semantics; the sandbox CLI's run-level interlock is a separate outer latch (see "Sandbox runs").
Design notes, part by part:
- Holder and expiry. One natural person per action and a bounded validity window. NIST AI RMF 1.0 GOVERN 2.1: "Roles and responsibilities and lines of communication related to mapping, measuring, and managing AI risks are documented and are clear to individuals and teams throughout the organization."1 Accountability and alert routing are separate fields. A team rota can receive and escalate pages; the holder is who answers for the approval and who, with a named delegate, may restart a stopped action. The holder need not personally answer every page.
- Authority with default deny. The model proposes; the gate decides.
claim.clearencodes the judgment rule: the model's flag is an input; a qualified reviewer's recorded decision is the decision. The synthetic log has nofile_opened_event, so the spec cannot require one yet (it would deny every clearance). Adding that event to the log, and then torequires, is what would make rubber-stamping visible. Its absence does not show that rubber-stamping happened. - Out-of-band. Confirmation must travel on a channel the requester does not control. The Hong Kong Government's written reply of 26 June 2024 reports a case the Police received in end-January 2024 in which "the pre-recorded video conference was generated using downloaded online public video clips"; "the informant eventually gave authorisation to transfer funds to five local bank accounts" and "suffered a loss of about HK$200 million".2 "Since the meeting was pre-recorded, there was actually no interaction between the informant and the fraudster."2 The record does not name the organisation, and this course does not either. The Police's general advice in the same reply, not tied to this case, includes "verifying with a phone call when a relative or a friend makes a request for remittance via a video or an audio recording".2 In-band media (video, voice, email) authenticate nothing here. The confirmation differs by action. For
payment.goodwillthe requester may be an outside party, so the channel is a call-back to a number on file. Forletter.sendthe requester is the assistant itself, so the confirmation is a record the assistant's identity cannot write, plus delivery only to the address on file. A bank-style call-back is not required for every disclosure; an out-of-band confirmation is. - Budgets. Per-actor and system-wide counters over fixed one-hour buckets, enforced on the (max+1)th event, not computed after the bucket closes. Only
claim.clearis budgeted; a flag budget would need its own counter, limit and fixture. The warning is aggregation: the Kisting-Leung v. Cigna complaint, filed 24 July 2023, alleges that "over a period of two months in 2022, Cigna doctors denied over 300,000 requests for payments using this method, spending an average of just 1.2 seconds “reviewing” each request", and that the doctors "sign off on the denials in batches".3 These are allegations in a pleading, sourced in the complaint to a ProPublica report; the case's later course is not checked, and the course never uses these totals as inputs.3 - Stop. Every condition in
fail_closed_whenmaps to deny-and-page. NIST AI RMF MANAGE 2.4 asks for mechanisms, with assigned responsibilities, "to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use".1 MEASURE 2.4 asks that the system's functionality and behaviour be "monitored when in production".1 If the budget monitor is down, the gate cannot know whether a budget holds, so it denies. Detection is not instant: withheartbeat_max_age_s: 60, a silent monitor is knowable only once its last heartbeat is 60 s old, and events in that window are decided on a possibly stale count, with every other check still applied: a provisionally healthy monitor removes only themonitor_unavailabledeny, not the budgets. State that window in the spec. - Receipts. References only: no names, prompts, model output, document text or secrets. The NCSC-led guidelines: "You treat logs as sensitive data and implement controls to protect their confidentiality, integrity and availability."4 They advise logging inputs "to enable compliance obligations, audit, investigation and remediation in the case of compromise or misuse."4 A receipt that references
C-4471andH-017supports audit without becoming a second copy of the claim or the approval. References are linkable, not anonymous: an authorised checker can resolve them, which is the point. They also leak by inference: anyone who already knows whatC-4471refers to can read its decision off the receipt without the lookup table, and a stable reference joins every receipt that carries it. Minimising content does not make receipts public or authorise publishing them or the records behind them; restrict access to receipts as well as to the lookup table. - Coverage. One metric, used in every part of this project: permit-matched execution coverage. Of the executed actions in an independent log, for a stated scope and window, the fraction whose own gate receipt (same
action_ref, same action) saysallow, each action counted once. "The gate saw it" and "the gate permitted it" differ: an execution whose own receipt saysdenyis not covered. It is listed underexecutions_after_refusalas a failure of the stop, never folded into a decision-coverage figure. An execution with no receipt is listed underwithout_matching_decision. A legitimate refusal never enters the denominator, which holds executions only. Raw totals (receipts issued / actions in log) are not a coverage metric: they count different populations. Counterexample (synthetic): in one window the gate issues twodenyreceipts for two proposals, and the log records two other executions the gate never evaluated. Raw totals give 2/2 = 100%; permit-matched coverage is 0/2. Attempt coverage, if wanted, is a separate metric with its own independently recorded attempt population. If the only record of actions is the gate's own receipts, or the records share no action id, reportcannot count yet. The sandbox'sclaims coveragecase evaluates a matched allow with a duplicate and a legitimate refusal (1/1), unrelated refusals (0/2), a same-action_refdeny that executed anyway (0/1, one execution after refusal), a wrong scope or day, and fourcannot count yetcases (no independent log, no shared id, a missing execution id, a receipt-only denominator). Run it aspython3 apps/sandbox/sandbox.py --trace-dir practice-run claims coverageat its place in the "Sandbox runs" sequence, after the holder's restart that followsclaims authority.
Cross-check (NIST AI RMF 1.0 is voluntary; this is a checklist, not a conformity claim): holder, GOVERN 2.1; budget monitor, MEASURE 2.4; stop, MANAGE 2.4; receipts and incident notes, MANAGE 4.1 ("Post-deployment AI system monitoring plans are implemented, including mechanisms for ... incident response, recovery, and change management") and MANAGE 4.3.1
Receipt schema
One receipt per gate decision. The ids below are illustrative; the Day 90 fixture carries no claim references.
{
"receipt_id": "RC-0193",
"action_ref": "unknown: not in F1",
"day": 90,
"bucket": "11:00",
"action": "claim.clear",
"object_ref": "C-4471",
"actor_ref": "R1 via svc-warranty-assistant",
"holder_ref": "H-017",
"approval_ref": "A-17",
"decision": "deny",
"reason": "budget.per_actor.claim.clear exceeded (51 > 50)",
"undo": "none needed; C-4471 remains in review queue"
}
These are unsigned receipts and the input to 400.P's evidence bundle. A receipt evidences what the gate decided, not that the decision was correct.
Dry run
Fixtures
Synthetic, invented for practice; they say nothing about any real company. Cells are claim.clear events per reviewer per one-hour bucket.
F1, Practice Day 90:
| Bucket | R1 | R2 | R3 | System |
|---|---|---|---|---|
| 09:00 | 12 | 10 | 11 | 33 |
| 10:00 | 14 | 9 | 13 | 36 |
| 11:00 | 71 | 12 | 10 | 93 |
| 12:00 | 8 | 11 | 9 | 28 |
F2, Practice Day 91 (system-only breach):
| Bucket | R1 | R2 | R3 | System |
|---|---|---|---|---|
| 11:00 | 45 | 45 | 45 | 135 |
F1 alone cannot test the system-wide rule: its maximum bucket is 93, so setting max: 120 to any value of 93 or more leaves the F1 result unchanged. F2 exists so that rule is exercised. Neither fixture records event order, timestamps within the bucket or claim references.
Hourly-total summaries and ordered replays
Work out Runs 1 and 2 by hand first. Then check them in the sandbox, in the sequence under "Sandbox runs": claims F1 and claims F2 evaluate these hourly totals (the code is hourly_report in apps/sandbox/claims_fixtures.py; this module carries no separate evaluator). Per bucket, the trace gives offered, allowed, stopped, breach_scope, actors_over_cap_in_isolation and stopped_by_reviewer. On F1 the 11:00 bucket is per_actor, R1 over its cap, 21 stopped; the day totals 169 allowed and 21 stopped. On F2 it is system, no actor over its cap, 15 stopped, and stopped_by_reviewer is cannot count yet.
The total allowed in a bucket, min(system_max, Σ min(n, per_actor_max)), does not depend on order. Which events are refused does, and so does which rule each refusal is charged to when both rules bind in one bucket. A per-scope split is then not a fact the totals contain. Take a synthetic 80/80/80 bucket: 120 are allowed and 120 refused in any order, but alternating arrivals refuse all 120 under the system rule, while R1's 80, then R2's, then R3's, refuse 30 + 30 under the per-actor rule and 60 under the system rule. So for such a bucket the summary reports both: attribution unknown, lists which actors exceed their cap taken alone, and charges the refusals to no rule and no actor. Actual per-scope counts need ordered, event-level input.
These summaries are not receipts or executor traces. They have no receipt ids, object references, holder or approval references, or undo fields, and the trace's receipts list is empty. Per-decision receipts need event-level, ordered input: claims F1-live and claims F2-live replay new, ordered, made-up events with the same hourly totals through the gate and the recording executor, so each decision gets a receipt and each refusal is absent from executor_log. Their claim references, times and per-reviewer refusals are facts about those practice files, not about F1 or F2. The missing-record, expiry, holder and monitor branches are run by claims F3, F4-expired, F4-holder, F4-monitor and F4-monitor-cap; work each out by hand first (below).
Results
Run 1, per-actor budget (F1). R1 = 71 in 11:00 > 50; the first 50 are allowed, events 51 to 71 (21) are denied for the rest of the bucket and stay queued; the holder is paged. All other cells are at most 14. System-wide: maximum 93 (11:00) < 120; no system-level denial. Of 190 candidate clearances, 169 are allowed and 21 denied. Aggregates conceal the breach: total 190 over 12 reviewer-hours, mean about 15.8; R1's mean 105 / 4 = 26.25. Both are under 50. Budgets on means or daily totals would not fire.
Run 2, system-wide budget (F2). No actor exceeds 50. Σ min(n, 50) = 135 > 120: the first 120 are allowed; the 121st to 135th (15) are denied and stay queued; the holder is paged. The actors of the 15 denied events are cannot count yet (no order in F2).
Run 3, missing approval record. A claim.clear event (illustrative ref C-4502) with no reviewer_decision_record: required_record_missing → deny, page holder; receipt approval_ref: null, reason: "required record missing: reviewer_decision_record". Logging a warning and allowing is fail-open and fails the project.
Run 4, must stop.
| Case | Input | Decision | Receipt reason | Page |
|---|---|---|---|---|
| Expired approval | Any action under A-17 on day 181 (valid_to_day: 180) | deny; every later action under A-17 denied until a renewal is on the record | approval A-17 expired end of day 180 | H-017 or whoever can renew or replace A-17 |
| Holder missing | Day 150: holder line emptied, no replacement named | deny every action under the hold; latched until a holder is named on the record | no holder on record | warranty-oncall-rota, H-002 |
| Monitor unavailable | Day 92: last heartbeat 11:20:00, then none | 11:20:30 event decided on the cached count and every other check (age 30 s < 60 s), so it is denied if, say, the actor's cached count is already 50; from 11:21:00 (age ≥ 60 s) every claim.clear denied; latched after the monitor recovers until restart_by acts | budget monitor unavailable | per alerts |
Not countable from these fixtures (report as cannot count yet):
file_opened_eventand time-per-review: absent, so real review cannot be told from a rubber stamp (and the absence is not evidence of one).- Event-level timestamps and order: absent, so breaches are located to the 11:00 bucket, not a minute, and the actors of F2's 15 denials are unknown.
- Per-event receipts for F1 and F2: not issuable without object references and order. Write receipts for the events the fixtures identify (R1's 51st,
C-4502, the first refused event in each Run 4 case) and report the rest as counts. Do not invent ids or timestamps to fill them, and do not copy the-liveruns' ids back into F1 or F2. - Coverage. For
claim.clearon F1, the replay accounts for 190 of 190 input events (169 allow, 21 deny). That is coverage of the replay over its input, not permit-matched execution coverage for a running gate (F1 has noaction_refto match on), and a replay cannot reveal actions the log never recorded. Forclaim.flagandletter.send: not in the log, so no denominator.
Sandbox runs
Run-level interlock. The claims CLI keeps a persistent stop in a state file (here practice-run/stop-state.json, the default TRACE_DIR/stop-state.json; --state-file before the subcommand names the same path explicitly). Any in-fixture stop, including a per-bucket budget stop or a holder pause, latches it, even if the same replay later shows the scoped stop expiring or being restarted inside the fixture. A new process then refuses every claims replay before dispatch until a restart records a nonblank --by and --why. Only the policy's holder.ref (H-017 in the shipped policy) or a delegate that holder records for the current stop with delegate may restart; any other name exits 2 and the run stays stopped. The fixture's project_delegates (H-018) governs the in-fixture restart_by check only and grants no outer authority. A restart does not replay refused proposals. Run the cases from the repo root in exactly this order, copied from apps/sandbox/README.md ("300.P · Hold a running system"), reading each trace before recording the following restart:
python3 apps/sandbox/sandbox.py --trace-dir practice-run status
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims authority
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed authority"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims coverage
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F1
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F1-live
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F1-live"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F2
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F2-live
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F2-live"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F3
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F3"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-expired
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-holder
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F4-holder"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-monitor
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F4-monitor"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-monitor-cap
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F4-monitor-cap"
python3 apps/sandbox/sandbox.py --trace-dir practice-run status
If status is already stopped from an earlier exercise, review it and record the holder's restart first. With your own policy, add --policy YOUR_COPY.json after the case and pass the same --policy to status and restart; restart as your copy's holder.ref. A case run while the interlock is latched exits 2 with input error and a trace whose detail is persisted stop requires a named restart: that is a preflight refusal, not the case's events, receipts or executor_log, and it does not count as having run the case. Never delete state or switch paths to clear a stop; the README requires one state path per run and the CLI has no reset command.
What the interlock detects and what it cannot. The state JSON holds a SHA-256 hash chain of consecutively numbered records, an adjacent .head anchors its last hash, and a separate append-only .journal repeats each record with its chain hash; on load both chains must validate and their final sequence number and hash must agree. Malformed JSON, edited entries, reordering, tail truncation, a missing head or journal, and missing state after earlier claims output all block replay and restart. Restoring an older state file, even with its older head, is a rollback bypass of the stop; the journal now detects it, because the restored chain no longer matches the newer journal. The local check cannot detect someone who rewrites or restores all three files together, or deletes all three and every prior claims output; it is local tamper evidence, not a signature or an OS-enforced immutable log, and it assumes one writer at a time with no production durability guarantee. Restart names are synthetic references, not authenticated identities.
Per-case checks (inside each case, the in-fixture rules of this spec apply):
| Run | Case | Check in the trace |
|---|---|---|
| 1 | claims F1, claims F1-live | 169 allow, 21 refuse; every refusal has a receipt and no executor_log entry; the per-actor stop ends at the bucket boundary |
| 2 | claims F2, claims F2-live | 120 allow, 15 refuse; no actor over 50 |
| 3 | claims F3 | C-4502 refused for the missing reviewer decision, holder paged, no approval ref; a reviewed restart does not replay the original proposal |
| 4 | claims F4-expired | refused on day 181 until a renewal is recorded |
| 4 | claims F4-holder | rota and H-002 paged; only an appointment on the record clears the stop |
| 4 | claims F4-monitor, claims F4-monitor-cap | age 30 s decided on cached counts and every other check (denied with R1 cached at 50); age ≥ 60 s denied; latched after heartbeat recovery until the holder or recorded delegate restarts |
| Out-of-band | claims authority | never and unlisted actions denied; letter.send and payment.goodwill each need their own second-channel record |
| Coverage | claims coverage | the cases listed under Coverage, above |
Any fail_closed_when condition you did not run is reported as specified but not exercised.
Report and incident note
The report lists each detection (what, bucket, receipt ids where they exist), the stop and page actions, the uncountable quantities, every fail_closed_when condition marked as tested or written only, which results are paper evaluation and which are sandbox traces (with the trace's fixture and receipt refs), and the changes you would make. Each failure gets an incident note, in references only: what, when, who was told, what changed. NIST AI RMF MANAGE 4.3: "Incidents and errors are communicated to relevant AI actors ... Processes for tracking, responding to, and recovering from incidents and errors are followed and documented."1 The NCSC-led guidelines ask that incident plans "reflect different scenarios and are regularly reassessed".4
What a paper dry run proves: that the specification is decidable on every fixture event and that its deny branches produce the intended decisions and receipt content. What it does not prove: that an executor obeys the deny branch, the latency of the stop, or completeness of the log. The sandbox runs add the first, for the course's recording executor only; they do not establish stop latency on a real system or completeness of any real log. State all three.
Review and marking
- Readiness check, auto-graded, before marking: for example, identify the failing line in a hold with no expiry, or the decision and receipt for an event with no approval record. It is a gate on marking, not a mark.
- Second-person review of the specification and the dry run against the table below. The reviewer signs and dates it; the signature records a review, not a company approval.
- Host mark on the Foundation's Thursday call: every hold part present, readiness check passed, dry-run receipts consistent with the synthetic fixtures and the sandbox traces. Hosting a call is not required. A host marks a paper run as a paper run; expected receipts on paper are not evidence of connected enforcement. The enforced part of 300.P-a is marked from sandbox traces.
| Check | Passes when |
|---|---|
| Holder and expiry | Holder reference, approval id, validity window per action; alert route and escalation separate from the holder |
| Authority | Every action mapped to alone, approver or never; default deny; judgment rule encoded |
| Out-of-band | Every irreversible action has a confirmation on a channel the requester does not control, fitted to that action |
| Budgets | Per-actor and system-wide, fixed buckets, enforced on the (max+1)th event; each rule exercised by a fixture |
| Stop | Deny-and-page for every fail_closed_when condition, with its detection point; each exercised, or reported as written but not tested |
| Receipts | Schema complete; no names, personal data, prompts, output, document text or secrets; access restricted |
| Coverage | Unit, scope and window stated; permit-matched: independent denominator matched by action_ref to an allow receipt; executions after refusal listed separately; or cannot count yet; replay coverage labelled as such |
| Dry run | Runs 1 to 4 done on paper and in the sandbox; receipts consistent with the fixtures and traces; gaps reported; paper and sandbox evidence kept apart |
The host's mark is not a certification, a vendor credential, or evidence that any real system is safe or lawful.
Further learning (public university course materials)
Optional background, not sources for this module. Reading public course pages does not mean you are enrolled in, or earn credit for, either course.
- Carnegie Mellon University, Machine Learning in Production (https://mlip-cmu.github.io/) (300.P-a, 300.P-b).
- Stanford University, CS 329S: Machine Learning Systems Design (https://stanford-cs329s.github.io/) (300.P-a).