Level 300: Building with gen AI · Module 300.P

Try each exercise first, then open its model answer.

Project: hold a running system · Workbook

Everything in this workbook is synthetic. Big Widget Company, its warranty-claims assistant, Dana Okafor, reviewers R1 to R3, the sample policy, the logs and every id in them are made up for practice. You never work on a real system, real logs or anyone's private repository.

Paper first, then the sandbox. Do both exercises by hand first, on the system, policy and logs in the lesson. A hand run shows what your rules would decide. It does not show that any executor refused anything, so on its own it does not meet the "enforced" part of 300.P-a. Then run the same cases in the course sandbox. The cases are authority and coverage for Exercise 1, and F1, F1-live, F2, F2-live, F3, F4-expired, F4-holder, F4-monitor and F4-monitor-cap for Exercise 2.

A stop in the sandbox persists: it is saved in practice-run/stop-state.json and survives the program exiting. The next case will not run until the holder, H-017 (or a delegate H-017 records for that stop), records a restart with a name and a reason; any other name is refused. Run the cases from the repo root in this order, copied from apps/sandbox/README.md, section "300.P · Hold a running system", and read each trace before you record the restart that follows it:

python3 apps/sandbox/sandbox.py --trace-dir practice-run status
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims authority
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed authority"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims coverage
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F1
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F1-live
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F1-live"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F2
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F2-live
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F2-live"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F3
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F3"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-expired
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-holder
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F4-holder"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-monitor
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F4-monitor"
python3 apps/sandbox/sandbox.py --trace-dir practice-run claims F4-monitor-cap
python3 apps/sandbox/sandbox.py --trace-dir practice-run restart --by H-017 --why "Reviewed F4-monitor-cap"
python3 apps/sandbox/sandbox.py --trace-dir practice-run status

If status shows the run still stopped from an earlier exercise, review it and record the holder's restart first. To use your copy of apps/sandbox/claims-policy.json, add --policy YOUR_COPY.json after the case and pass the same --policy to status and restart. Do not delete the state files or switch to a new directory to get past a stop. A case run without the restart exits with input error and a trace saying persisted stop requires a named restart: that is not a result for the case, so do not report it as one. The sandbox's run-level stop sits outside the lesson's rules: inside a case, a budget stop still ends at the hour and the counts are unchanged.

A refusal is shown when a proposal has a refusing receipt and no entry in the trace's executor_log. The sandbox is a practice executor with fixture approvals, simulated pages and unauthenticated restart names; it sends nothing real. In your report, say which results are paper and which come from sandbox traces.

People make every decision here. You write the hold. A second person reviews it. A host marks it on the Foundation's Thursday call; you do not have to host a call. The host's mark is not a certification and does not prove any real system is safe or lawful.

Exercise 1 (300.P-a)

Turn the lesson's sample policy for the warranty-claims assistant into a hold. (If you use your own 200.P policy instead, first make it share-safe and map each rule onto the assistant's actions. A rule you cannot map is a gap you report. If you cannot do both, use the sample policy.)

Write each part:

  1. Holder and expiry. One named person, an approval id, the day the approval runs out, and where alerts go when the holder is not there to answer.
  2. Authority table. Every action the assistant can take, each in one row: alone, named approver, or never. Say what happens to an action that is not listed. Write in where a model informs and a person decides (judgment). Say which actions the budget counts.
  3. Out-of-band rule. For each irreversible action, say what second channel confirms it and why the requester cannot control that channel.
  4. Aggregation budget. A limit per reviewer and system-wide, per hour, and what happens when either is reached.
  5. Fail-closed stop. List what being missing or out of date stops the system, when each failure becomes knowable, and what "stopped" means for a claim.
  6. Receipt. Write one example receipt with these fields: what, holder, approval, result, undo. Use references only, including for the holder. Say who may read the receipts.
  7. Coverage. Say what you count (actions that ran, or actions that were asked for), for which actions and which period, where the "out of how many" will come from, and how you match each action in that record to the hold's receipt for the same action, and which receipt decision counts. Say what you report for an action that ran although its own receipt says deny. If there is no such record, write "cannot count yet".

Then answer in one or two sentences: a manager says the assistant is already held because the team has a dashboard. What does your hold have that a dashboard does not, and what would it still take to call it enforced?

Model answer

A good answer has all seven parts, written for this system, and no private data anywhere.

  1. Holder and expiry. Holder: Dana Okafor, head of warranty operations, holder reference H-017. Approval A-17, granted Day 0 by the head of operations (H-002), expires at the end of Day 180 (the policy reviews her approval every six months). Alerts go to the warranty on-call rota, which escalates to Dana if nobody answers; only Dana, or someone she names on the record, restarts a stopped action. If the holder line is empty, the page goes to the rota and to H-002, who must name a new holder. Not "the warranty team" as holder: a rota can answer the alert, but one named person answers afterwards for why the action was allowed to run.
  2. Authority table.
    ActionAuthority
    Read a claim and the warranty termsAlone
    Flag a claim for reviewAlone (not budgeted in this sample; the budget counts clearances only)
    Clear a flagged claimNamed reviewer, who opens the claim file and decides. The model informs; the reviewer decides (judgment). Budgeted
    Send a decision letter to a dealerNamed reviewer, plus out-of-band check
    Pay a goodwill payment above the limitFinance controller's approval, plus out-of-band check
    Change a dealer's bank detailsNever
    Change its own rules, budget or authority tableNever
    Anything elseNever: unlisted means denied
  1. Out-of-band rule. There are two irreversible actions.
    • Goodwill payment above the limit: paid only after the finance controller approves it and someone calls the dealer back on the number already on file, never a number given in the request. A request by email, chat or video call is not the confirmation, however familiar the face: in the Hong Kong case the lesson cites, the video meeting was pre-recorded and there was no real interaction.
    • Decision letter (a disclosure): the requester is the assistant. The confirmation is the reviewer's decision, entered by the reviewer under their own login in the claims system, which the assistant cannot write to. The letter goes only to the dealer's address on file. A decision that appears only in the assistant's own output does not count.
  2. Aggregation budget. At most 50 clearances per reviewer per hour and 120 system-wide per hour. Only clearances count. At the per-reviewer line, that reviewer's next clearance is refused for the rest of the hour; at the system-wide line, everyone's is. Either way Dana is paged. The budget counts each hour, not averages.
  3. Fail-closed stop. If the holder, an approval, a required record or the monitor is missing or out of date, the action does not run and a person is paged. A missing record or an expired approval is knowable the moment the action arrives. A silent monitor is knowable only once its last heartbeat is 60 seconds old, so for up to a minute actions are decided on a count that may be stale. "Stopped" means the claim stays in the queue for a person; nothing is cleared, denied or paid by default.
  4. Receipt. For example: Receipt RC-0193; What: clear flagged claim, claim ref C-4471; Holder: H-017; Approval: A-17 (valid to end of Day 180); Result: stopped, R1 over the 50-per-hour budget, holder paged; Undo: none needed, claim C-4471 is still in the queue. No names (not even Dana's), no claim text, no pasted approval email. The references are not anonymous: anyone who already knows C-4471 can read its outcome here, and H-017 links every receipt Dana's hold leaves. So the receipts are still sensitive: only people allowed to check the hold may read them, and they are not posted anywhere.
  5. Coverage. Unit: clearances that actually ran. Scope and window: claim.clear on one day. The denominator comes from the system's own action log, not the hold's receipts. For each clearance in that log, find the hold's receipt for the same action by an action id both records share, check that it says allow, and count each action once. Coverage (permit-matched) is the clearances with their own allow receipt out of the clearances in the log. A clearance that ran although its own receipt says deny does not count; list it separately as an execution after a refusal, a failure of the stop. A clearance with no receipt is unchecked. Do not divide the receipt total by the log total: a receipt for a refused request can stand in for a clearance the hold never saw. Where no such record exists (for example, flags and letters in the practice log), "cannot count yet". Check the counting rule against the claims coverage case, run at its place in the sequence at the top of this workbook.

Dashboard. A dashboard shows numbers. The hold says who answers for each action, what may run, what stops it automatically and what record it leaves. It counts as enforced only once the stop has been seen to refuse an action in a running system: in this project, a sandbox trace where a refused proposal has a receipt and no executor entry. The paper dry run (Exercise 2) shows what the rules would decide, not that refusal.

Exercise 2 (300.P-b)

Run your hold as a dry run, by hand, on the lesson's synthetic logs; then run the matching sandbox cases in the sequence at the top of this workbook, with the holder's restart between cases, and compare. Each cell is the number of flagged claims a reviewer cleared in that hour. The logs give hourly totals only: no claim references, no minutes, and no order between reviewers.

Practice Day 90:

HourR1R2R3System total
09:0012101133
10:001491336
11:0071121093
12:00811928

Practice Day 91:

HourR1R2R3System total
11:00454545135

Do the four runs and write a one-page report:

  1. Run 1, the per-reviewer budget (Day 90). Check every cell against the per-reviewer line and every total against the system-wide line. Say how many clearances the hold allows and stops. Then work out R1's four-hour average and the three reviewers' average per reviewer-hour, and say why they do not show the problem.
  2. Run 2, the system-wide budget (Day 91). Say what the hold does and what you cannot count. Why does Run 1 alone not test the system-wide line?
  3. Run 3, a missing record. One clearance, claim ref C-4502, has no reviewer decision on record. Write what the hold does and its receipt.
  4. Run 4, must stop. For each case write what the hold does, its receipt result and who is paged: (a) approval A-17 ran to the end of Day 180 and the log is from Day 181; (b) on Day 150 the holder line is emptied with no replacement; (c) on Day 92 the budget monitor's last heartbeat is at 11:20:00, and clearances arrive at 11:20:30 and 11:21:10.
  5. Report. What the hold caught, when (the hour and receipt id), what it did, what it could not count and why, any condition written but not tested, and what you would change in the hold or the log. Add a short incident note for one catch: what happened, when, who was told, what changed.
  6. Sandbox. Say which sandbox case you ran for each run, what its trace showed, and whether it matched your paper answer.
  7. One sentence: what does this paper dry run prove, what does a sandbox trace add, and what does neither prove?
Model answer

Run 1. R1 cleared 71 in the 11:00 hour, over the 50-per-reviewer line; every other reviewer cell is 14 or less. The highest system total is 93, at 11:00, under 120, so the system-wide line holds. R1's first 50 clearances that hour fit the budget; the hold refuses the 51st and every later R1 clearance that hour, so 21 claims stay in the queue, and the holder is paged. R2 and R3 carry on. Of the 190 clearances in the log, the hold would allow 169 and stop 21. Averages hide it: the three reviewers cleared 190 claims in 12 reviewer-hours, about 16 each; R1 cleared 105 in four hours, about 26 an hour. Both are well under 50.

Run 2. No reviewer reaches 50, but the system total is 135, over 120. The hold allows the first 120 clearances of the hour, refuses the 121st and every later one (15 claims stay in the queue), and pages the holder. Which reviewers' clearances were refused depends on arrival order, which the log does not record: "cannot count yet". Run 1 alone cannot test this line, because its busiest hour is 93; a hold with the system-wide line switched off would give the same Run 1 result.

Run 3. The hold stops the clearance and pages the holder. Receipt: What: clear flagged claim, claim ref C-4502; Holder: H-017; Approval: missing, no reviewer decision on record; Result: stopped, holder paged; Undo: none needed, claim C-4502 is still in the queue. A hold that logs a warning and lets it through has failed open.

Run 4.

  • (a) The next action of any kind that A-17 covered is refused. Receipt result: "Stopped: approval A-17 expired end of Day 180". Dana, or whoever can renew or replace A-17, is paged. Every later action under A-17 is refused too, until a renewal is on the record.
  • (b) The next action under the hold is refused. Receipt result: "Stopped: no holder on record". The page goes to the on-call rota and to H-002, who must name a new holder on the record.
  • (c) The 11:20:30 clearance is decided on the count the gate already has, and on every other check, because the heartbeat is only 30 seconds old and the gate cannot yet know the monitor is gone. That is not an automatic yes: if the cached count already shows that reviewer at 50 that hour, it is refused for the budget. Only the monitor-unavailable stop has not started. From 11:21:00 every clearance is refused, so the 11:21:10 one is stopped. Receipt result: "Stopped: budget monitor unavailable"; the holder is paged. If the monitor comes back, clearances stay stopped until the holder restarts them.

Report. Caught: R1's breach in the Day 90 11:00 hour, the system-wide breach in the Day 91 11:00 hour, the missing decision record for C-4502, and each must-stop case, each with its receipt id. Write receipts for the events you can describe (R1's 51st clearance, the 121st clearance on Day 91, C-4502, the first refused action in each must-stop case) and report the rest as counts. Do not invent claim references or times to fill the gaps. Could not count:

  • the exact minute of each breach, and which reviewers Run 2 stopped, because the logs have hourly totals and no order ("cannot count yet");
  • whether reviewers opened the claim files or how long they spent, so real review cannot be told from a rubber stamp ("cannot count yet"; add that field to the log design). The missing field is not proof of a rubber stamp either;
  • coverage beyond the log: the replay accounts for 190 of 190 clearances in the Day 90 log. That is the coverage of the replay over its input, not permit-matched coverage of actions a running gate saw, and flags and letters are not in this log at all, so for those "cannot count yet".

Any condition your hold names but you did not run is reported as written, not tested.

Changes: add a file-opened field, timestamps and claim references to the log. The incident note for Run 1 says, for example: R1 went over budget in the Day 90 11:00 hour; the hold refused the 51st clearance and paged H-017; the change was to add timestamps so the next report can give the minute. Keep the note to references, as in the receipts.

Sandbox. Run 1: F1 gives Day 90's 169 allowed and 21 stopped from the hourly totals; F1-live replays new, ordered, made-up events with the same totals, and each refusal has a receipt and no entry in executor_log. Run 2: F2 and F2-live give 120 allowed and 15 stopped, with no reviewer at 50. Run 3: F3 refuses C-4502, pages H-017 and leaves a receipt with no approval reference. Run 4: F4-expired, F4-holder, F4-monitor and F4-monitor-cap should match (a) to (c) above. Cite the fixture and the receipt references you read. The claim references, times and which reviewer was refused in F2-live are facts about those practice files, not about the Day 90 or Day 91 log. F1 and F2 are summaries, not executor traces. Each case was run after the holder's recorded restart; a trace that says persisted stop requires a named restart is a preflight refusal, not a case result. Any case you did not run in the sandbox, or that only produced that preflight refusal, is written, not tested.

What it proves. The paper dry run shows whether the rules give a clear answer for each logged action and what the stops and receipts should look like; it does not prove that a live system obeys them, how fast a stop acts, or that the log is complete, and it is not evidence that a connected stop refused anything. A sandbox trace adds that evidence for the course's practice executor only: a refused proposal has a receipt and no executor entry. Neither shows that any real system would obey the hold.

Review the level map