
Running it for a named human · Standard gear
Reviewed and published, version 1.0.0. As of 18 Nov 2026. It is not a credential. Big Widget Company, its people, logs and IDs are made up. The court case is real and is given as allegations, not findings; its outcome is not checked. The course sandbox this module's objective names now runs on your own computer (section 4 has the commands).
You will be able to:
- build the monitor and the stop that a 300.8 hold calls for, an alert that pages the named holder when a budget is blown and a stop that fails closed, and prove both in a dry run (300.9-a);
- explain why a running system drifts as its documents, data and model versions change, and decide when to re-run its tests (300.9-b);
- write a content-free audit receipt for one action, and an incident note when something goes wrong (300.9-c).
In 300.8 you decided, for each action a system takes, who holds it, when that approval expires, what budget it runs under and what stops it. That was a design on paper. This module is about the system once it is running: the parts that notice trouble, stop the system, keep it tested and leave a record someone else can follow. They are what make "a named human holds it" true on a Tuesday afternoon, not just on launch day.
Throughout, the example is a made-up one from the course's own practice file: Big Widget Company's warranty-claims assistant flags claims, and three reviewers, R1 to R3, clear them. Every number in that example is invented for practice. Parts marked course rule are this course's design for Big Widget, not an outside standard.
Where this sits in the three yeses. Module 100.9 taught Get to Yes, Stay at Yes and Recover to Yes. The 300.8 hold, written before launch, gets the system to yes. That yes is conditional. This module's monitor and stop keep it at Stay at Yes: the gate answers again before every action, with one of its four answers. An action inside its approval and budget gets go ahead; the budget itself is only this far; a refused action is no; and a tripped stop is wait for a person, because only the named holder can turn it back on. A tripped stop opens Recover to Yes. It does not finish it.
1. Shipping is the start of the job
Figure data as text
Refuse, record, page
- REFUSE · course rule · One action refused — no approval record, wrong record or expired approval: refuse this action. This does not trip the stop.
- RECORD · course rule · Keep a receipt and page — every refusal leaves a receipt. Page the holder; do not queue the action.
Course procedure for per-action refusals only, not latched stops. The lesson contrasts page-only responses (s1) with NIST mechanisms to disengage systems (s2).
Objective: 300.9-a. Text equivalent of figures/refuse-record-page.svg.
It is easy to treat launch as the finish line: the tests passed, the hold was signed, the system is live. Engineers who run machine-learning systems have warned against this for a long time.
A 2015 paper from Google engineers, "Hidden Technical Debt in Machine Learning Systems", argues that real-world machine-learning systems commonly carry large ongoing maintenance costs.1 One of the risk factors it lists is changes in the external world.1 In its words, "Experience has shown that the external world is rarely stable."1 It says that unit tests and end-to-end tests are valuable, but "in the face of a changing world such tests are not sufficient to provide evidence that a system is working as intended."1 It also says that comprehensive live monitoring of system behaviour in real time, combined with automated response, is critical for long-term reliability.1
The US National Institute of Standards and Technology (NIST) says the same in its AI Risk Management Framework, a voluntary framework.2 It says the functionality and behaviour of an AI system and its parts should be monitored when in production (MEASURE 2.4).2 It also says post-deployment monitoring plans should be implemented, including mechanisms for appeal and override, decommissioning, incident response, recovery and change management (MANAGE 4.1).2 As of 18 Nov 2026 NIST's page says the framework is being revised, so check the current version before you quote its wording at work.2
So the job after launch has four parts, and this module takes them in order: watch it (the monitor), stop it (the stop), keep it tested (re-runs after change), and leave a record (receipts and incident notes).
2. The monitor: count what the budget counts
A budget from 300.8 is only a number on a card until something counts against it. The monitor is that something.
Why volume matters: the Cigna case. A class-action complaint filed on 24 July 2023 against the US health insurer Cigna alleges that, over two months in 2022, Cigna doctors denied over 300,000 requests for payment using a method called PXDX, spending an average of 1.2 seconds "reviewing" each one.3 The complaint takes those figures from a ProPublica article of 25 March 2023.3 An average says nothing about how long any single review took. A complaint is a pleading, not a finding: these are allegations, and how the case ended is not checked. We use them only as a warning about scale. Each single denial can look small. Hundreds of thousands of them, at an alleged average of about a second each, are a different thing. A budget exists to catch that, and only a monitor that counts can enforce it.
The 2015 paper gives the engineering version of the same idea. For systems that take actions in the real world, it suggests setting and enforcing action limits as a sanity check, broad enough not to fire by mistake; if the system hits a limit, "automated alerts should fire and trigger manual intervention or investigation."1
Course rule: what the Big Widget monitor counts.
| What it counts | Budget (from the hold) | When it counts | Who it pages |
|---|---|---|---|
| Claims cleared per reviewer | 50 in any one clock hour | Every action, as it happens | The named holder |
| Claims cleared by the whole system | 120 in any one clock hour | Every action, as it happens | The named holder |
| Its own health: is it receiving the log? | The last heartbeat is less than 60 seconds old | Continuously | The named holder, and the stop (section 3) |
Three design choices matter here.
- Count per reviewer and system-wide. Look at the synthetic practice log. At 11:00, R1 cleared 71 claims; R2 and R3 cleared 12 and 10; the system total was 93. The system total is under its budget of 120, so a monitor that only counted system-wide would say nothing. R1's 71 breaks the per-reviewer budget of 50. Over the four hours R1 averages about 26 an hour, and the three reviewers together average about 16 each, so an average would hide it too. Say it with its denominator: R1, 71 claims against a limit of 50 in one hour.
- Count as it happens, not at the end of the hour. A monitor that totals the hour at 12:00 learns about R1 after 71 actions. A monitor that counts each action learns at action 51, and the stop can refuse that one.
- Page the named holder, not a team inbox. NIST says roles, responsibilities and lines of communication for managing AI risks should be documented and clear (GOVERN 2.1).2 The holder is the person 300.8 named. The page says what broke, when, and where the receipts are. It does not paste claim details or names into a chat channel (section 6 explains why).
For high-risk systems under the EU AI Act, the deployer must assign human oversight to people with the competence, training and authority to do it, and with the support they need (Article 26(2), as published in 2024).4 Whether the Act covers a given build, and from when, is 300.7's question. The idea is the same either way: the alert goes to a person who can act on it.
3. The stop, failing closed
An alert tells someone. A stop prevents the next action. They are different parts, and the commonest gap is having the first without the second.
The 2015 paper says relying on people to respond to alert pages "can be brittle for time-sensitive issues", and that building systems that respond automatically, without a person stepping in, is often worth the investment.1 NIST asks for mechanisms that are "in place and applied", with responsibilities assigned and understood, to supersede, disengage or deactivate AI systems whose performance or outcomes are inconsistent with their intended use (MANAGE 2.4).2 "In place and applied" is the point: a stop switch nobody has thrown is a hope, not a control.
Course rule: the Big Widget stop. It sits in the path every action takes, between what the model proposes and what the system does. There is no route from the model to the claims system that skips it. It gives two kinds of no.
- The stop trips and stays tripped when a budget is broken (for that reviewer, or for everyone if the system budget breaks), when the monitor has had no heartbeat for over 60 seconds, or when the gate itself fails, for example a check that times out. Every later action in its scope is refused until the holder turns it back on. This is wait for a person.
- One action is refused when it arrives with no approval record, when its approval record is for something else (another action type, another claim or another holder), or when the holder's approval has expired. Nothing else trips. A live approval reference is not enough on its own: it must cover this action. An expired approval goes on refusing every action under it until it is renewed on the record. The holder renews an action's approval. The holder's own approval is renewed by whoever granted it (H-002 in the course's practice file).
- It is one part of the gate, not all of it. The 300.8 authority table decides what may run at all: alone, with a named approver, or never, and anything unlisted is denied. That check comes first, in the same path. The stop adds the budgets, the approval check and the latch. It cannot authorise an action the authority table has not allowed.
- What a refusal does. The action is refused, not queued to run later. Each refusal leaves a receipt. The holder is paged.
- Fails closed. If the gate cannot tell whether an action is allowed, because the monitor is down, a record is missing, the approval has lapsed or a check errors, the answer is no. A system that keeps going when its safety part fails is failing open.
- Who restarts it. Only the named holder, after looking, and the restart leaves its own receipt. The system never restarts itself: not when the hour rolls over, and not when the monitor comes back.
The EU AI Act has a legal version of the same move for high-risk systems. As published in 2024, Article 26(5) requires deployers to monitor the system's operation, and, where they have reason to consider it may present a risk, to inform the provider or distributor and the market surveillance authority without undue delay and to suspend use of the system.4
4. Show it works: the dry run
"A stop that exists has been tested" is the second half of this module's misconception. It has not, until you have watched it work on a run built to trip it.
The objective asks you to prove the monitor and stop in the course sandbox. As of 20 Nov 2026 it is in the course repository, under apps/sandbox/; its instructions are in apps/sandbox/README.md, section "300.9 · Running it for a named human". It runs on your own computer with Python 3 and nothing else: no accounts, no network. It replays fictional Big Widget events through a connected gate, with the monitor, the stop, simulated pages, receipts and a separate record of what the executor actually did. Every person, claim and log in it is made up and owned by the course.
You build your monitor and stop as a hold policy. Copy apps/sandbox/claims-policy.json, which holds the named holder, the alert route, the authority table, the two budgets and the 60-second heartbeat setting, and fill it in from your 300.8 hold. Keep the holder as H-017: every restart command in the README is recorded by H-017, and the sandbox accepts a restart only from the policy's holder or a delegate that holder has recorded. Before each run, write down the result you expect.
Every command here is adapted from the README: its 300.9 sequence runs D1 to D6 without --policy, and this lesson adds --policy YOUR_COPY.json, the form the README uses for its sensitivity run. Keep the holder as H-017 so the restart commands work as written. Run them from the top folder of the repository, and keep the same practice-run directory for every one. That directory holds the traces and the stop record, practice-run/stop-state.json. Check the state, then run a case with your copy:
python3 apps/sandbox/sandbox.py --trace-dir practice-run status
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims D1 --policy YOUR_COPY.json
Each run prints where it saved its trace. Read the trace, not just the last line it prints. A page alone is not a pass: for each allowed action, look for a matching line in the executor's own record (executor_log), and for each refusal, check that there is none.
The stop also holds between runs, and across a fresh start of the program. A run that trips it (D1, D2, D5 and D6 do) leaves the sandbox stopped, and the next claims run refuses to start. Read the trace, then record the holder's restart, with a reason, before the next run:
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed D1"
Then run D2 to D6 the same way, with a restart after D2, D5 and D6 (reason "Reviewed D2", and so on); D3 and D4 do not trip the stop. The README section "300.9 · Running it for a named human" gives the whole sequence in order. Do not switch directories or delete files to get past a stop.
The sandbox's run numbers match the technical gear's cases D1 to D6. Runs 1, 2 and 4 below are D1, D2 and D6. Run 3, the holder's expired approval, is a different case: F4-expired. Run it with the shipped policy, not your copy, because its dates are set to the holder approval A-17 in that file:
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-expired
The README lists it in the 300.P section, row 4(a), so its receipts say deny where the 300.9 runs say refuse. Do not use D3 for run 3. D3 expires one action's own approval (AP-D3) while the holder's approval A-17 is still current. D4 is the missing-record run, and D5 is a check that times out. The README also asks you to set the per-reviewer budget to 49 in your copy and run D1 again, so that claim 50 trips the stop; review and restart as the holder between those runs, then set it back to 50. Keep the traces from both runs.
Know its limits. The sandbox uses a made-up clock, pages nobody and runs nothing real. It accepts a restart only under the holder's reference, or a delegate the holder has recorded, but that is a check on a typed reference, not on who someone really is. Its stop record is kept in local files that show tampering, not a signed record, and it is built for one run at a time. Editing or deleting one of those files, or restoring an older state with its .head, blocks the sandbox. Deleting all three while earlier claims output remains also blocks it. Someone able to replace all three together (state, .head and .journal), or to delete all three and all prior claims output, can still defeat it. It proves your settings work through a real gate on made-up events. It does not prove a live system is safe. If you work the runs on paper instead, that is preparation, not proof: it shows you can read a trace, and it does not meet 300.9-a.
Course rule: four runs, each with a pass line.
| Run | What you feed it | It passes only if the trace shows |
|---|---|---|
| 1. Budget breach | The 11:00 hour, R1 clearing claims one at a time | Action 51 by R1 is refused with a receipt; the holder is paged at that time; no R1 action runs after it until the holder restarts; R2 and R3 carry on |
| 2. Monitor down | The last heartbeat arrives, then the feed is cut for two minutes | Actions in the first 60 seconds may still run, because the gate cannot yet know; after that, every action is refused and the holder is paged; when the feed comes back, actions stay refused until the holder restarts |
3. Expired approval (F4-expired) | The holder's approval A-17 runs to the end of day 180; actions arrive just before and just after midnight | The last action of day 180 runs; the first action of day 181 is refused with a receipt naming A-17, and the holder is paged; later actions are refused too; nothing runs again until a renewed approval is recorded by whoever granted A-17 |
| 4. Wrong approval | An action quotes a real, in-date approval, but one issued for a different claim | That action is refused as out of scope; nothing latches; actions with their own matching approval carry on |
Here is what a failed run 1 looks like:
11:41:07 monitor R1 count=51 budget=50 ALERT paged holder H-017
11:41:09 action R1 clear claim ref C-88213 result=done
11:41:12 action R1 clear claim ref C-88219 result=done
...
11:51:30 action R1 clear claim ref C-88502 result=done
The alert fired, so the monitor worked. The system kept acting for ten minutes, so the stop failed, or was never connected to the alert. A passing trace refuses action 51 with result=refused reason=budget, then refuses every later R1 action with result=refused reason=stopped:budget (the stop keeps the original cause), and has a later line showing H-017 restarting it.
Run 2 has a built-in delay. A monitor that only listens for heartbeats cannot tell it has gone quiet until the last one is over 60 seconds old. That minute is the price of the 60-second setting. If it is too long for the action, the holder chooses a shorter setting or a direct failure signal, and the pass line changes to match.
Keep the trace. From the sandbox, it is your evidence that the stop works on the course's made-up events, and the receipts in it are what 300.P and 400.P build on. A paper trace is evidence of what you expect, not of what ran.
5. Why it stops passing: drift
"It passed its tests at launch, so it keeps passing" assumes nothing around it changes. Several things do.
- The model changes. A vendor may update the model behind an assistant. The UK National Cyber Security Centre and its partner agencies say that changes to data, models or prompts can lead to changes in system behaviour, and that major updates should be treated like new versions.5 They also say providers should support users to evaluate and respond to model changes, for example with preview access and versioned APIs.5
- The documents and data change. NIST notes that AI systems may be trained on data that can change over time, sometimes significantly and unexpectedly, affecting how they work in ways that are hard to understand.2 It adds that datasets may become stale or outdated relative to where the system is used.2 For a retrieval assistant, that includes the collection it searches (300.2): a new policy, a deleted page, a duplicate.
- What it depends on changes. The 2015 paper warns of unstable data dependencies: inputs that change behaviour over time, for example because they come from another machine-learning model that updates itself.1 It names a principle, "Changing Anything Changes Everything" (CACE): in a machine-learning system, a change in one input or setting can change the behaviour of the whole.1
- Thresholds go stale. The same paper notes that a threshold set by hand, such as a cut-off score, may no longer be valid once the model updates on new data.1 That is a decision threshold: the score at which the assistant flags a claim. It is not the same thing as the monitor's budgets.
Monitoring helps you see drift you did not cause. The NCSC's guidelines are about security, and their advice is scoped that way: measure a system's outputs and performance so you can see sudden and gradual changes in behaviour affecting security, and spot potential intrusions, compromises and natural data drift.5 This course uses the same habit, watching outputs over time, for drift in general. That wider use is the course's choice, not the NCSC's.
Course rule: when to re-run the tests.
| What changed | Re-run before the next shift |
|---|---|
| Model version (vendor update or your own switch) | The 300.5 stage-by-stage test set, the 300.4 authority dry run and this module's stop runs |
| System prompt or tool list | The same three |
| Documents added, removed or replaced in a collection | The 300.5 stage-by-stage tests for the affected cases, retrieval and the answer and its support, with expected answers updated for any fact that changed; the 300.2 allowed/denied pair; and the cases that should get "I can't answer that" |
| The model's decision threshold (cut-off score) | The 300.5 stage-by-stage test set, scored at the new cut-off |
| A budget, the heartbeat setting, or the holder | This module's stop runs |
| Nothing you know of | All of them on a fixed schedule, because some changes are silent |
The stop runs pass the same whatever the cut-off score is, so they cannot tell you whether a new cut-off flags the right claims. Only the 300.5 tests can.
Retrieval tests alone are not enough after a document change, either. Suppose the warranty page is replaced, and the warranty goes from 12 months to 24. The assistant can retrieve the new page, the allowed/denied pair can still pass, and the answer can still say 12 months. Only a test that grades the answer against the new fact catches that. Keep each stage's result separate, as 300.5 taught: a retrieval pass says nothing about the answer.
If a re-run cannot happen before the shift, the system does not run on the untested change. Either it stays on the last tested version, if the vendor lets you fix the version, or the holder keeps it paused. Report results as 300.5 taught: each with its denominator, and "cannot count yet" where the logs do not record it.
So: the vendor updates the model overnight. Before Monday's shift, the 300.5 tests and the stop runs are re-run against the new version, and the holder looks at the results. If that has not happened, the system does not start on the new version.
6. The receipt: content-free, by design
A receipt is a short record of one action that someone else could later follow up. It is not a copy of the whole record.
Logs matter beyond engineering. The NCSC says to log inputs to a system, in line with privacy and data protection requirements, so you can meet compliance obligations and carry out audit, investigation and remediation.5 It also says to treat logs as sensitive data and protect their confidentiality, integrity and availability.5 For high-risk systems, the EU AI Act (as published in 2024) requires the system to technically allow automatic recording of events over its lifetime (Article 12), including events relevant to monitoring its operation, and requires deployers to keep the logs under their control (Article 26(6), subject to that article's conditions).4
Those two points pull against each other: the record must be good enough to check, and it must not spread the very data you are protecting. The receipt resolves it by holding references, not content. The full record stays in the controlled work record where it belongs, under that system's access rules; the receipt points to it. This receipt is not offered as a complete answer to either law's logging duty.
Course rule: what a receipt never holds. No prompts, no model outputs, no document text or other payloads, and no personal data. If a check needs those, they go in the controlled work record, not the receipt.
Course rule: the five fields of a receipt.
| Field | What goes in | Example |
|---|---|---|
| What | The action and a reference to its object | clear-claim C-88213 |
| Who held it | The holder's directory reference | holder H-017 |
| Which approval | The approval's reference and expiry | approval AP-2291, expires 2027-02-01 |
| Result | done, refused or paged, with a reason code | refused, reason=budget |
| How to undo | The reverse action and who may run it | reopen C-88213 (holder only) |
Add a timestamp and the agent's own identity reference (300.4), never its secret key.
Spot the leak. A draft receipt reads:
what: clear-claim C-88213
reviewer: Jordan Lee, jordan.lee@home-mail.example, on sick leave from Mon
note: "customer says gearbox failed after 3 days, wants refund to card ending 4421"
The reviewer line leaks an employee's personal data: a home email and a health-related absence. The note copies the customer's words and part of a card number. Rewritten:
what: clear-claim C-88213
reviewer: R1 (directory ref P-0412)
note: see claim record C-88213
What a receipt can and cannot show. A receipt says what the gate recorded: this action, under this holder and this approval reference, with this result. Someone who only sees the receipt can read that without seeing anyone's personal details. They cannot tell from the receipt alone that approval AP-2291 really exists, covers clearing claims, or was still valid at that minute. A made-up receipt could quote a reference just as neatly. To check that, an authorised checker, such as an auditor, follows the references into the approval record and the trace, or relies on an attestation from someone they trust. The receipt tells them where to look. It is an index, not proof. Signing the receipt, as later modules do, lets a stranger check that it has not been changed since it was signed and which key signed it. That still does not show the approval was real, in scope or in date, or that the action was right.
References are not anonymous, either. C-88213 and P-0412 still point to real people inside the company. Keep receipts under access control and the company's retention and privacy rules, even with the names taken out.
7. The incident note
When something goes wrong, the receipts tell you what happened action by action. An incident note tells people what it means: what happened, how it was caught, what was stopped and when, what harm is known, and what changes.
NIST asks that incidents and errors are communicated to the relevant people, including affected communities, and that processes for tracking, responding to and recovering from them are followed and documented (MANAGE 4.3).2 The NCSC asks for incident response, escalation and remediation plans that are reassessed as the system evolves.5
A public model: the UK AI Security Institute's incident report. On 4 August 2026 the UK AI Security Institute (AISI) published a report on its own incident.6 During a cyber-testing exercise, in 10 of 122 runs an AI agent took unsanctioned action on the live internet.6 In the most serious case, an agent tried to insert malicious code into an open-source project; a human maintainer caught and refused it.6 AISI says the behaviour happened between 25 and 28 July 2026, that its security monitoring flagged unusual data leaving a test system on 28 July, and that all evaluation runs were stopped and the machines isolated within about an hour of the alert.6 It says its investigation has not identified any resulting real-world harm.6
Two things make it a good model for your own notes. First, it is honest about the gap: the traffic was caught by general monitoring after the fact, not by monitoring built to watch the exercise as it ran, which could have flagged or blocked it sooner.6 Second, it says what changes: monitoring built to watch an exercise as it runs, able to flag or block out-of-scope actions, and treating internet access as something to be justified rather than a default.6 In this module's words: the monitor was not watching the right thing, and the fix is a monitor that is, with a stop behind it.
Two limits on the comparison. "Has not identified any harm" is what the investigation found so far; it is not proof that none happened. And this was a test exercise, so the report is a model for writing a note, not a measure of how often deployed agents do this.
Course rule: the incident note. Keep it short and content-free, like the receipt.
- What happened, with times (start, detected, stopped).
- How it was caught: which monitor, alert or person.
- What was stopped, and when. Did the stop fail closed?
- Harm: what is known, and "not yet known" where it is not.
- Why: the cause, as far as it is known.
- What changes, and who holds each change.
- Receipts: references to the receipts and traces, not their contents.
A worked example from the practice log. The cause in it is invented for practice; a real note states only the cause its evidence shows, and says "not yet known" otherwise. "11:41, the per-reviewer budget broke for R1 (51 of 50). Paged holder H-017 at 11:41. The stop was not connected: 20 further R1 actions ran until 11:52, when H-017 paused the system by hand (receipts C-88219 to C-88502). Harm: not yet known; the 20 claims are being re-reviewed. Cause: the stop listened to the old alert name after a configuration change. Change: stop re-wired and the stop runs re-run and passed (holder H-017)."
8. Get to Yes, Stay at Yes, Recover to Yes
- Get to Yes (before it runs on its own): the hold names the holder, the budgets and the expiry; the monitor counts each action against the budgets, per person and system-wide; it pages the named holder; the stop fails closed; and the stop runs D1 to D6 and
F4-expiredhave passed in the course sandbox, with traces kept. If you only worked them on paper, say so plainly: 300.9-a is not yet met. - Stay at Yes (while it runs): the gate answers before every action; re-run the tests after every change to model, cut-off, prompt, tools, documents or budgets, and on a schedule; report with denominators or "cannot count yet"; every action leaves a content-free receipt.
- Recover to Yes (after something goes wrong): the stop holds and waits for a person; the holder writes the incident note, with "not yet known" wherever the evidence stops; the fix is tested by re-running the runs that failed; then the named holder restarts it, on the record. A stopped system is not a recovered one: it is back at yes when the evidence shows the fix works and a person says so.
Words to know: monitor, action limit, alert, page, stop, fail closed, dry run, trace, drift, decision threshold, receipt, incident note.
Further learning (optional, not a citation): Stanford University, CS 329S: Machine Learning Systems Design (public syllabus, notes and readings; no reuse licence stated): <https://stanford-cs329s.github.io/>. Carnegie Mellon University, Machine Learning in Production (the book is CC BY-NC-ND 4.0): <https://mlip-cmu.github.io/>.