
Project: hold a running system · Plain gear
Reviewed and published, version 1.0.0. As of 18 November 2026. It is not a credential. Big Widget Company, its people, its systems, its rules and its logs are made up. The court record, the government record and the guidance documents are real. The court case is a set of claims, not findings. How it ended is not checked.
Your project
Figure data as text
Report the paper run
- REPORT · course procedure · Describe catches and gaps — what, when, stops and pages; reference receipts. Say what you could not count.
- QUALIFY · course procedure · Say which results are paper and which come from sandbox traces — a paper result is not evidence that a connected stop refused anything; a sandbox trace is evidence only about the course’s practice executor. Name untested rules.
- RECOVER · course procedure · If something went wrong: incident note — what happened, when, who was told, what changed; keep references only.
Course dry-run reporting procedure; NIST incident communication (s2) and NCSC incident planning (s3) support the incident note.
Objective: 300.P-b. Text equivalent of figures/report-the-paper-run.svg.
This is the last project in Level 300. You take one practice AI system and you hold it.
Holding a system means writing down, for every job it does: who answers for it, what it may do, what makes it stop, and what record it leaves. Then you test what your rules would do.
You bring two things:
- Your hold. The written rules for one practice system (300.P-a).
- Your test report. What your rules would have done on a practice log (300.P-b).
Everything is pretend. You never touch a real system or real records. The course gives you a made-up system, made-up rules and made-up logs.
Test on paper first, then on the practice computer. First you do the test by hand, with the logs in this lesson. A paper test shows what your rules would do. It cannot show that a computer did stop. The project asks for a stop that a computer actually obeys. So next you run the same tests in the course's practice computer setup, called the sandbox. It is a small program in this course's files. The steps are in apps/sandbox/README.md, in the part called "300.P · Hold a running system".
The sandbox is still pretend. It sends no real letters, payments, calls or messages. When it stops, it stays stopped until the holder starts it again (see "Now run it in the sandbox" below). It only shows that the course's practice computer obeyed your rules. In your report, say which results came from paper and which came from the sandbox.
Your starting rules. Use the sample rules below. If you did the 200.P project, you can use your own rules instead. But only if nothing in them is private, and only if you match each rule to the practice system's jobs. If a rule does not match any job, say so. Do not guess. If you can't do both, use the sample rules.
People make every choice here. You write the hold. A second person checks it. A host marks it.
The three yeses. In 100.9 you met Get to Yes, Stay at Yes and Recover to Yes. Your hold is the Get to Yes. It says what may run and who answers for it. Before every job, the hold gives one of four answers: go ahead, only this far, wait for a person, or no. Answering every time is Stay at Yes. Stopping, paging a person and writing a note start Recover to Yes. A stopped system is not fixed yet.
Here is a common mistake: "If a system has a dashboard, someone is holding it." A dashboard only shows numbers. A hold says who answers, what may run, and what stops it. And someone has seen the stop work on a running system.
The practice system
Big Widget Company makes widgets. Dealers send in warranty claims when a widget breaks. An AI helper reads the claims. It marks, or flags, the ones it thinks the warranty does not cover.
Three reviewers, called R1, R2 and R3, look at the flagged claims. When a reviewer signs off one flagged claim, we call that a clearance.
The sample rules (made up):
The AI helper may read claims and the warranty rules, and flag claims. A named reviewer must open each flagged claim and decide it. No letter goes to a dealer unless a reviewer decided. Big goodwill payments need the finance boss to say yes, and a phone call back to the dealer's number on file. The helper never changes a dealer's bank details. It never changes its own rules. Dana Okafor runs warranty work and holds the helper. Her yes is checked every six months.
Your job is to turn these rules into a hold.
The seven parts of a hold
1. One named person, and an end date
One real person holds each job. "The team" is not enough. A team can share the work. A team rota can answer an alarm at 3 a.m. and pass it up. But one named person must answer for why the job was allowed to run. The rota answers the alarm. The holder answers for the yes.
Each yes also has an end date, because people move jobs.
A US science agency, NIST, wrote a guide to AI risk. It says who does what should be written down and clear to everyone.1
Example: Dana Okafor holds the helper. Her code on receipts is H-017. Her yes is called A-17. It ends after Day 180. Alarms go to the warranty team rota first. If nobody answers, they go to Dana. If there is ever no holder at all, the alarm also goes to the boss who gave Dana her yes (code H-002). That boss must name a new holder. (The practice files count days, not real dates.)
2. The "who may do what" table
Every job goes in one of three boxes: alone, needs a named person to say yes, or never. Anything not on the list is a no.
| Job | Box | The hold's answer |
|---|---|---|
| Read a claim | Alone | Go ahead |
| Flag a claim | Alone | Go ahead |
| Clear a flagged claim | A named reviewer, who opens the claim and decides | Wait for a person, then only this far |
| Send a letter to a dealer | A named reviewer, whose choice is on record, plus a second check | Wait for a person |
| Pay a big goodwill payment | The finance boss, plus a call back | Wait for a person |
| Change a dealer's bank details | Never | No |
| Change its own rules | Never | No |
| Anything else | Never | No |
The AI helps. A person decides. A person who signs without looking has not decided.
The limit in part 4 counts clearances only. Flags have no limit in these rules. If you want one, give flags their own limit and their own count.
3. A second check for things that cannot be undone
Some acts cannot be undone. This system has two. Sending money is one. Sending a letter is the other: once a dealer reads it, you cannot take it back. For both, the check must come through a second channel that the one asking does not control.
Here is why. In 2024, the Hong Kong Government reported a fraud the Police got in late January 2024. Crooks used deepfakes, which are fake videos made with AI. They made a fake video meeting from video clips found online.2 The person they tricked said yes to sending money to five local bank accounts. The loss was about HK$200 million.2 The meeting was recorded ahead of time. Nobody real was talking back.2 The government did not name the company, and neither do we. In general advice, not about this case, the Police told people to check with a phone call when a relative or friend asks for money by video or audio.2
So a face on a video call does not prove who is asking.
For the helper:
- Money. Call the dealer back on the number you already have, not a number in the request.
- Letters. The helper asks to send the letter. So the check is the reviewer's choice, typed in by the reviewer under their own login. The helper cannot write there. And the letter only goes to the dealer's address already on file.
4. A limit on how many
Small acts can add up to something big. So set a limit for each hour: one for each reviewer, and one for the whole system. Look at the busiest hour, not the average.
Here is a warning. In 2023, people sued an insurance company called Cigna. Their court papers say its doctors turned down over 300,000 payment requests in two months in 2022. They say the doctors spent an average of about 1.2 seconds on each.3 They also say the doctors did not open the patient files.3 These are claims in a lawsuit, not findings by a court. How the case ended is not checked. We use them as a warning, not as numbers to do maths with.
Example limit: each reviewer may clear up to 50 an hour. The whole system may clear up to 120 an hour. Go over, and clearances stop for the rest of that hour, and Dana gets a page.
5. A stop that fails closed
"Fails closed" means: if something is missing, the answer is no. If the holder, the yes, a record or the alarm is missing, the job waits. A person gets a page.
Some problems show up at once, like a missing record. Others take a while. The alarm sends a "still here" signal. In these rules, if the last signal is a minute old, the alarm counts as gone. So for up to a minute, the hold does not know yet. Write that down.
NIST's guide says there should be ways to switch an AI system off when it is not working as planned, with someone in charge of doing it.1 A stop only counts once you have seen it stop something on a running system.
For the helper, "stopped" means the claim waits for a person. Nothing is turned down or paid by itself.
6. A receipt with no private details
Every time the hold decides something, it leaves a receipt. The receipt says: what happened, who held it, which yes, what the result was, and how to undo it.
Use codes, never names or the claim's words. That goes for Dana too. The receipt says H-017, not "Dana Okafor". Her name stays in the approval record, which only allowed people can see. The UK's cyber-security agency says to treat logs as sensitive and protect them.4 A receipt with only codes lets someone check the choice without copying private details.
Codes are not secret names. A person who is allowed to can look them up. Codes also do not hide everything. Someone who already knows a claim's code from somewhere else can still match it to a receipt. And the same code on many receipts links them together. So codes keep names and words out of the receipt, but the receipts are still sensitive. Keep them where only allowed people can read them, and do not post them anywhere.
Example receipt (made up): RC-0193. Clear claim C-4471. Reviewer R1. Holder H-017. Yes A-17. Result: stopped, R1 is over 50 this hour, holder paged. Undo: nothing to undo, the claim is still waiting.
These receipts go up to Level 400. There, you pack them so a stranger can check them. A receipt shows what the hold decided. It does not prove the choice was right.
7. An honest count
Say how many of the system's jobs your hold saw, out of how many. The "out of how many" must come from some other record, not from your hold's own receipts. If you don't have one, write "cannot count yet". That is an honest answer.
Count the same kind of thing on both sides. Say first what you are counting: jobs that actually ran, or jobs that were asked for. Then match them one by one. Take each job in the other record and ask: did the hold leave a receipt for this exact job, and does that receipt say yes? Only those jobs count. A receipt for a job the hold said no to does not count for a different job that ran. Just dividing one total by the other can look like 100% when the hold said yes to none of the jobs that ran.
Watch for one more case. A job ran, and its own receipt says no. That job does not count. It is worse than a missing receipt: the stop did not work. List it on its own line in your report.
The test run
Now you test your hold on the practice logs. Nothing real happens. You just write down what your hold would have done. A paper test does prove something. It shows whether your rules give a clear answer each time, and what the stops and receipts should look like. It does not prove that a live system obeys them. Say that in your report.
Test every problem your hold says it handles. If you wrote a rule but did not test it, say "written, not tested".
The practice log
These numbers are made up. Each box is how many claims a reviewer cleared in that hour, on Day 90.
| Hour | R1 | R2 | R3 | Whole system |
|---|---|---|---|---|
| 9:00 | 12 | 10 | 11 | 33 |
| 10:00 | 14 | 9 | 13 | 36 |
| 11:00 | 71 | 12 | 10 | 93 |
| 12:00 | 8 | 11 | 9 | 28 |
The log only has hourly totals. It has no claim codes, no minutes, and no order.
Test 1: each reviewer's limit
- Each reviewer (50 an hour). R1 cleared 71 in the 11:00 hour. That is over the limit.
- Whole system (120 an hour). The most is 93. That is under the limit.
- Why the average hides it. All three reviewers cleared 190 claims in 4 hours. That is about 16 each, each hour. R1 alone averaged about 26 an hour. Both look fine. Only the hour-by-hour count shows the problem.
What the hold does: R1's first 50 clearances that hour are fine. At number 51 the hold stops R1 for the rest of the hour. The other 21 claims wait. Dana gets a page. So of 190 clearances, 169 go ahead and 21 wait.
When did it catch it? In the 11:00 hour. The log only has hourly totals, so you cannot tell the exact minute. Say so.
Test 2: the whole-system limit
Test 1 never gets near 120. So it does not test the whole-system limit at all. Here is a second made-up hour, on Day 91:
| Hour | R1 | R2 | R3 | Whole system |
|---|---|---|---|---|
| 11:00 | 45 | 45 | 45 | 135 |
No reviewer gets to 50. But the whole system cleared 135, over 120. The hold lets the first 120 through. It stops the other 15. Dana gets a page. Whose clearances were stopped? The log does not show who came in what order. Write "cannot count yet".
Test 3: a missing record
One clearance has no record of the reviewer's choice. The hold must not let it through. It stops it and pages Dana. The receipt says: "Missing: no reviewer choice on record."
If the hold just writes a warning and lets it through, it failed open. That is a fail.
Test 4: it must stop
Each of these must stop. The receipt says why. Someone who can fix it gets a page. Nothing carries on by itself.
- The yes ran out. Dana's yes, A-17, ended after Day 180. The log is from Day 181. The next job stops. The receipt says the yes has run out. Someone who can renew it gets a page.
- No holder. On Day 150, Dana moves to a new job. Nobody new is named. The next job stops. The receipt says there is no holder. The team rota and H-002 get a page.
- The alarm goes quiet. On Day 92, the alarm's last "still here" signal comes at 11:20:00. Then nothing. At 11:20:30 the hold cannot know yet that the alarm is gone. So a clearance then is decided the normal way: on the count the hold already has, and with every other check. If R1 is already at 50 that hour, it still stops. Only the "alarm is gone" stop has not started yet. From 11:21:00, every clearance stops. Dana gets a page. If the alarm comes back, clearances stay stopped until Dana starts them again.
Now run it in the sandbox
Write down what you expect first. Then run the tests from the top folder of the course files.
The sandbox remembers a stop. When a test makes the sandbox stop, it stays stopped. Closing it and starting it again does not clear the stop. The stop is saved in a file, practice-run/stop-state.json. Only the holder, H-017, can start it again, by typing her code and a reason. (She can also name someone on the record to do it for her.) Nobody else can. So between tests, you play Dana: you look at what the last test did, then you record the restart. Never delete the file or switch to a new folder to get around a stop. That is a way around the stop, not a fix.
Type these lines in this order. They are copied from apps/sandbox/README.md, part "300.P · Hold a running system":
python3 apps/sandbox/sandbox.py --trace-dir practice-run status
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims authority
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed authority"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims coverage
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F1
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F1-live
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F1-live"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F2
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F2-live
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F2-live"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F3
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F3"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-expired
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-holder
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F4-holder"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-monitor
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F4-monitor"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-monitor-cap
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F4-monitor-cap"
python3 apps/sandbox/sandbox.py --trace-dir practice-run status
status tells you if the sandbox is ready or stopped. If it is still stopped from an earlier exercise, look at why, and record the holder's restart first.
Which line goes with which test:
| Test | Case names |
|---|---|
| 1 | F1, then F1-live |
| 2 | F2, then F2-live |
| 3 | F3 |
| 4 | F4-expired, F4-holder, F4-monitor, F4-monitor-cap |
| Second check (part 3) | authority |
| Honest count (part 7) | coverage |
Each run tells you where it saved its record. Open the record. A stop worked if the job has a "no" receipt and is missing from the list of jobs that ran (executor_log).
If you skip a restart, the next test does not run at all. It prints "input error", and its record says "persisted stop requires a named restart". That record is not a test result. Record the restart and run that test again.
To try your own rules, copy apps/sandbox/claims-policy.json and change your copy. Add --policy and your copy's name after the case name. Add the same thing to status and restart too. If you change the holder's code in your copy, restart with that code.
Two kinds of stop. Inside one test, the stops follow the rules in this lesson. A budget stop ends when the hour ends, and R2 and R3 carry on. The sandbox's saved stop is a second, outside lock. It holds the whole sandbox until the holder restarts it, even when the stop inside the test has already ended. It does not change the test's numbers: Test 1 is still 169 go ahead and 21 wait.
The sandbox is honest about its limits. It does not check who really typed "H-017"; it only checks the code. It notices if someone edits, cuts short, deletes or swaps in an old copy of the saved stop file. But someone who replaces all of its files together, or deletes all of them and every past record, can still fool it. It is a practice lock, not a real one.
F1 and F2 only add up the hourly totals. The -live runs use new made-up jobs with the same totals, one by one. Their codes and times belong to those practice runs, not to the Day 90 log.
What you could not count
End with the gaps. For these logs:
- Did the reviewers open the claims? The log does not say. So you cannot tell real checking from just signing. Write "cannot count yet". (That does not prove they just signed, either.)
- The exact minute of each problem, and whose clearances Test 2 stopped.
- A receipt for every clearance. The log has no claim codes. Write receipts for the cases you can describe, and give the rest as counts. Do not make up codes or times.
- Your honest count. Your test went through all 190 clearances in the Day 90 log: 190 out of 190. That counts your test, not a running system. And the log does not list flags or letters at all. For those, write "cannot count yet".
If something goes wrong, write a short note: what happened, when, who you told, and what you changed. Use codes, not names. NIST says problems should be shared with the right people and written down.1 The UK guide says to plan for different kinds of problems and to check the plans again and again.4 The system is back at yes only when the fix is tested and the holder says so.
Checking and marking
- A quick quiz first. The computer marks it. For example: a hold has no end date; which line fails? You pass this before anyone marks your project. The quiz is not your mark.
- A second person checks your hold. They look at all seven parts and your four tests. They check that no receipt has names or private details, and that you said which results are paper and which came from the sandbox. Then they sign and date it.
- A host marks it on the Foundation's Thursday call. The host checks that all seven parts are there, your quiz passed, and your receipts match the practice logs and the sandbox records. You do not have to run a call. The host marks a paper test as a paper test. Only a sandbox record shows that the stop was obeyed.
The host's mark means a host checked your work. It is not a certificate. It does not prove any real system is safe or legal.