Try each exercise first, then open its model answer.
Hold: the operate gate · Workbook
Four exercises, one for each thing this module teaches. Big Widget Company, its people, its systems and its logs are made up. The court cases are real and are given as allegations, not findings. Do not use real names, customers, logs or documents from your own work. Write short answers. "Not checked" or "cannot count yet" is an honest answer when the record does not tell you.
Your answers are a private learning record. Do not paste personal, customer or claim details into them.
Exercise 1 (300.8-a)
Write the hold, then test it. Big Widget's warranty-claims assistant can take these actions:
- flag a claim for review;
- send a rejection letter;
- pay a refund of over $10,000 to a dealer;
- change a dealer's bank details.
- For each action, write the holder, the expiry and the stop that fails closed. Use the people in the module (Dana Okafor, head of warranty operations; Sam Reyes, finance controller) or say which action the assistant must never take.
- Which action needs an out-of-band check, and what is it? Use the Hong Kong deepfake case to say why a video call is not enough.
- Test the hold. For each case, say whether the action runs, and what happens instead if it does not:
- A. Dana leaves the company in January 2027. Nobody updates the hold.
- B. It is 1 June 2027, and nobody renewed the approval for flagging claims.
- C. Dana's standing approval for rejection letters is current. A letter is ready, but there is no reviewer's named decision on that claim on record.
- D. A colleague says: "Warranty operations owns all of this, and our on-call rota answers every alert. We don't need a name on each line."
- E. The budget monitor goes down at 02:00 and the gate pauses rejection letters. At 06:00 the monitor is back up. Nobody has looked at the pause yet.
- The gate pauses rejection letters. Who can end the pause, and which of the three yeses does the pause start? What does the receipt for the pause hold, and what must it never hold?
- One day the gate checked 10 requests and turned all 10 down, and the system took 10 actions. A colleague reports "coverage 10 ÷ 10 = 100%". What is wrong, and how should coverage be counted?
Model answer
One good hold (the module's own table):
Action Holder Expires Stop (fails closed) Flag a claim for review Dana Okafor 31 May 2027 No current approval or holder: flagging pauses; claims go to the normal queue unflagged Send a rejection letter Dana Okafor 31 May 2027 No reviewer's named decision on this claim: the letter is not sent, even with Dana's standing approval current Refund over $10,000 to a dealer Sam Reyes 28 Feb 2027 No call-back confirmation of this refund on the dealer's number on file: the refund is not paid Change a dealer's bank details Never by the assistant — Not on the list, so denied Your dates can differ. Check that each line has one named person (not a team), a real end date, and a stop that says what happens when something is missing: the action does not run, and a person is paged.
- The refund over $10,000: money that has left the company cannot be recalled by changing a setting. The check is a call back on the dealer's number already on file, a channel the person asking does not control, and it confirms this payment: this payee, this amount. A tick in a box that anyone could set is not a confirmation. In the case the Hong Kong Police received in end-January 2024, fraudsters used deepfake technology to fabricate a pre-recorded video conference; the person targeted authorised transfers to five local bank accounts and lost about HK$200 million. Because the meeting was pre-recorded, there was no real interaction on the call at all. A face or voice on a call is not identity. (The police's own advice, to verify with a phone call when a relative or friend asks for money, was general advice; the call-back rule for payments is the course's broader rule.)
- Tests:
- A: does not run. The holder no longer exists, so nobody answers for those actions, whatever the expiry date says. The actions that needed Dana's approval pause, and the gate pages Dana's manager, or someone else with the authority, to name a new holder.
- B: does not run. The approval expired at the end of 31 May. Flagging pauses at expiry, claims go to the normal queue unflagged, and the holder is paged. Running on "until someone notices" is failing open.
- C: does not run. The standing approval says the assistant may send this kind of letter at all. It is not a decision on this claim. With no reviewer's named decision on record, the letter is not sent, the holder is paged, and the receipt says which record is missing.
- D: the colleague is half right. Shared ownership is fine for the work, and a rota with an escalation route is a good way to make sure someone always answers the alert. But a team or a rota cannot be the one who said this action may run, who can stop it, and who is asked afterwards why they said yes. Each action needs one name. The rota handles the alert; the holder answers for the decision.
- E: still does not run. The letters paused when the monitor went down, and the pause does not end by itself when it comes back. The monitor coming back is a fact; saying yes again is a person's decision. The letters stay paused until Dana, or someone she names on the record, looks at what happened and releases the pause on the record. Picking up where it left off is failing open.
- Dana, or someone she names on the record. The AI never releases its own pause. The pause is the start of Recover to Yes: a named person looks at what stopped and why, fixes it, and says yes again on the record. A paused system is not yet a recovered one. The receipt holds references, not content: which action, which hold entry (an id such as
HOLD-WC-03), which approval and decision records, the gate's answer, and how to undo it. It never holds the claim text, the letter, a customer's details, or a person's name or login, Dana's included; those stay in the controlled hold and work records the receipt points to. The id keeps the name out of the receipt, but it does not make the receipt anonymous: anyone allowed to open the hold can see thatHOLD-WC-03is Dana's. - Two totals side by side do not show coverage. The gate could have turned down its ten requests while the system ran ten other actions with no check at all. Coverage asks, of the actions that actually ran, how many the gate checked and allowed first, matched one action at a time. The ten refused requests are counted separately, because they never ran. Any action that ran with no check, or after the gate said wait or no, is listed by itself as a breach, not folded into a smaller percentage. Every run counts, not just the first: each run is checked against what the gate said just before it, so a second run after a pause is a breach too. An action with more than one run is also listed, so a person can find out whether it really ran twice or was only logged twice. If the records cannot be matched yet, the hold says "cannot count yet".
Exercise 2 (300.8-b)
Set a budget on a made-up log. Big Widget's reviewers R1 to R3 decide on the claims the assistant flags. Each number is how many decisions a reviewer made in that hour. This log is invented for practice, and it shows what happened with no live stop running.
| Hour | R1 | R2 | R3 | System total |
|---|---|---|---|---|
| 09:00 | 12 | 10 | 11 | 33 |
| 10:00 | 14 | 9 | 13 | 36 |
| 11:00 | 71 | 12 | 10 | 93 |
| 12:00 | 8 | 11 | 9 | 28 |
The budget is at most 50 decisions per reviewer per hour, and 120 system-wide per hour.
- Which hour, if any, breaks the per-reviewer line? Which breaks the system-wide line?
- Work out R1's average per hour over the four hours, and the average per reviewer per hour across all three. Say why neither shows the problem.
- A colleague proposes this stop: "When a reviewer reaches 50, stop sending them new flags." What is wrong with it? Write a stop that works, and say what happens to the claims already in R1's queue and who is paged.
- The Cigna case is this module's warning about volume. In two or three sentences, say what was reported and alleged, what Cigna said, and why this course does not use the figures in the budget above.
Model answer
- Per reviewer: 11:00, when R1 made 71 decisions, over the 50 line. System-wide: no hour. The busiest was 93 at 11:00, under 120. A budget with only a system line would have missed R1 entirely.
- R1: 12 + 14 + 71 + 8 = 105, and 105 ÷ 4 is about 26 an hour. All three: 190 in total over 12 reviewer-hours, about 16 per reviewer per hour. Both are well under 50. An average spreads one bad hour across the quiet ones; the budget is checked on each hour, because the peak is what broke it.
- The budget counts decisions, so the stop must stop decisions. Stopping new flags only stops new work arriving: R1 could keep clearing cases already in the queue and go past 50 anyway. A stop that works: at R1's 51st decision in the hour, R1's further decisions that hour do not take effect and no rejection letter goes out on them; they wait for a person. Cases already in R1's queue stay where they are, and nothing is approved or rejected automatically. The gate pages the holder, Dana Okafor, who moves the waiting cases to a reviewer under budget or leaves them to wait, and records which she chose. New flags stop going to R1 until Dana releases the pause; the count starts again at noon, but a new hour does not end the pause. If the system line is crossed, the same happens for everyone. With this stop running, the log would show 50 for R1 at 11:00, not 71.
- ProPublica reported in March 2023, citing corporate documents and interviews with former Cigna officials, that over two months Cigna doctors denied over 300,000 payment requests using its PXDX method, spending an average of 1.2 seconds on each. The Kisting-Leung class action, filed in July 2023, alleges the same figures and cites the ProPublica report for them, so it is not a second, independent count. Asked whether PXDX lets its doctors reject claims without examining them, Cigna said that description was incorrect (it was not speaking of the totals), and said reviews happen after the service and do not result in denials of care. These are claims, not findings: the course uses them only as a warning that small acts add up, and does not turn them into an hourly rate or an input to the budget.
Exercise 3 (300.8-c)
Real review or rubber stamp? Big Widget's log for one reviewer's Tuesday shows this:
- 412 flagged claims cleared as rejections;
- 0 overrides of the assistant's flag;
- every decision timed at 3 or 4 seconds;
- decisions signed in four batches, one action each;
- the log has no field for whether the reviewer saw the case, by any route.
- What do these traces point to? Name the evidence, with its denominator, and say what it does and does not prove.
- What can you not tell from this log, and what should the hold say about it?
- Explain, using the EU AI Act's Article 14, why a name on each decision is not human oversight.
- Name two more things you would check before saying the reviewer could really have decided otherwise. Use the Lokken complaint to show why.
- A manager suggests: "Make every reviewer overturn at least 5% of flags, then we know they review." Answer them.
Model answer
- The traces of a rubber stamp: "0 overrides out of 412 decisions", seconds each and nearly identical, and whole batches signed with one action. Read together, they are strong reasons to ask questions. They do not prove what the reviewer was thinking: a low override count can also mean the system is usually right. Real review usually varies in time with the case, records a reason when it disagrees, and is signed one at a time. A person reads the traces together and decides.
- Whether the reviewer saw any case. With no record of case views at all, the log cannot tell a real review from a rubber stamp, and the hold should say so: "case views: cannot count yet". A missing record is not a zero. And even a recorded view shows the case was opened, not that it was understood.
- Article 14 says people assigned to oversee a high-risk AI system must be enabled, as appropriate and proportionate, to remain aware of the tendency to rely or over-rely on its output (automation bias), and to decide in any particular situation not to use it or to disregard, override or reverse the output. A reviewer who never sees the case cannot know when to override. A human decided only if they had the information, the time and the authority to decide otherwise, and used them.
- Whether the reviewer had enough time per case, and whether they can say no without penalty, written down. The Lokken complaint alleges that the insurer set targets to keep skilled-nursing stays within 1% of the days the AI model projected, and that employees who deviated from its projections were disciplined and terminated. Those are allegations, not findings, but they show what to check: targets or pressure to match the system make overriding unlikely, whatever the paperwork says.
- A quota does not fix it. There is no right rate of disagreement: a reviewer who overturns flags to hit 5% is not reviewing either. Check the conditions for real review instead (seeing the case, enough time, freedom to say no, signing one at a time) and read the traces as questions, not scores.
Exercise 4 (300.8-d)
Allegation, statement or gate? Sort each statement about the nH Predict case into allegation (from the Lokken complaint), company statement (UnitedHealth's spokesperson, as reported by STAT) or gate (the course's own rule). Then answer the questions.
- A. The model "has a 90% error rate".
- B. "Coverage decisions are based on CMS coverage criteria and the terms of the member's plan".
- C. A model may inform but not replace clinical judgment in a coverage decision.
- D. Coverage decisions were made "not based on individual patient's needs, but based on the outputs of the nH Predict AI Model".
- E. The "lawsuit has no merit".
- Sort A to E.
- A colleague says: "The model is accurate on average, so it can decide each case. And the lawsuit proves it was wrong anyway." Answer both halves.
- Explain why the gate asks the same thing whoever turns out to be right. Does the company's statement show the gate is met?
- Big Widget's service planner says a repair needs 4 hours. The engineer who examined the widget says 7. Who decides, and what goes in the record? Over 2,000 jobs, engineers have never departed from the planner. What does the hold do with that?
Model answer
- Allegation: A and D (the complaint filed on 14 November 2023 in federal court in Minnesota). Company statement: B and E (the emailed statement STAT reported the same day). Gate: C (the Foundation's teaching rule, not a finding about the case, not something the company said, and not a quote from any law).
- An average can inform a case, but it does not decide the patient in front of you; this is the aggregation problem turned around. And a lawsuit is not a finding: a complaint is one side's account, filed to start a case, and the company's statement is the other side's. The module states neither as fact, does not check the 90% figure's basis, and does not say how the case ended.
- If the complaint is right, the model's output replaced clinical judgment, which the gate forbids. If the company is right, the tool is a guide and coverage decisions rest on the coverage criteria and the plan's terms. That account fits the gate, but the statement alone does not show the gate is met: it does not say who reviews each case. The hold should be able to show it: who decided, whether they saw the case, and how often they departed from the model. Either way, when the model's predicted stay and the treating clinician disagree, a qualified person reviews that patient's case and decides, and the record shows who and why. The gate does not say whose figure must win.
- Under Big Widget's own rule for its repairs (not a rule about medicine or insurance), the planner's estimate informs and the engineer who examined the widget decides: the 7 hours stands, with the reason recorded. If there is no engineer's decision on record, the job is not closed; it fails closed. "Never, out of 2,000" is not a success, and not proof of a problem either. It is a question for the holder, Dana Okafor: are engineers seeing the jobs, with time and freedom to depart from the planner?