
Hold: the operate gate · Technical gear
Reviewed and published, version 1.0.0. As of 19 Nov 2026. It is not a credential. Big Widget Company, its people, systems and logs are made up. Court cases are cited as pleadings and reporting. Nothing here is a finding, and case outcomes are not checked. The code is a teaching sketch. It has not been run against a real executor, it does not handle concurrency, and it is not a library to deploy.
Objectives. 300.8-a: bind each action of a running system to a named holder, an approval expiry and a fail-closed stop. 300.8-b: model aggregation and enforce a per-principal and system-wide budget over time buckets, on peaks, not means. 300.8-c: detect rubber-stamp review from log evidence and explain why it is not oversight. 300.8-d: keep allegation, company statement and gate as separate claim types, and show the gate's requirement is invariant to which account is true.
The operate gate is the course's rule set (the Practice 300 Hold, as the course-owned prompts in levels/300/practice/hold-prompts.md). It is not a statute. Where law states a related duty, it is quoted and cited.
Lifecycle and decision rights
In the course's three yeses (module 100.9), the hold written before launch is Get to Yes. The operate gate, evaluated before every execution, is Stay at Yes. A stop opens Recover to Yes, which closes only when a named person has inspected the cause, repaired it and recorded a new yes. A paused system is not evidence of recovery, and a dependency that comes back is not a new yes.
The predicate returns one of the course's four gate answers: Allow (go ahead), Constrain (only this far), Pause (wait for a person) and Deny (no). This module writes the third as Pause to keep it distinct from the hold record. The gate decides by rule. A system-level Pause is latched: it persists across evaluations until a named person (the holder, or someone the holder names on the record) records a release. The model or agent never releases it, and neither does the recovery of whatever caused it.
The model proposes; an executor runs only what the gate allowed. There is no direct path from the model to the tool for these actions, and a system prompt is not that boundary.
1. The hold as a policy over actions
Figure data as text
Who holds this action?
| Focus | Lesson point |
|---|---|
| HOLDER · COURSE RULE · Name one person | Write who answers for this action and when their approval expires. |
| STOP · COURSE RULE · Fail closed | Holder, approval or record missing or expired? Do not run; page a person. |
| LEGAL CONTEXT [s6] · High-risk systems must be stoppable | Article 14: overseers must be able to intervene or halt it safely, as appropriate and proportionate. |
The holder, expiry and fail-closed rule are course-owned procedure. Article 14 supplies the separate high-risk oversight context (s6); this summary does not show the whole operate gate or establish legal compliance.
Objective: 300.8-a. Text equivalent of figures/who-holds-this-action.svg.
Definitions
Let a system expose a finite action set A (the authority table from 300.4: alone, approver, never; anything not in A is denied). A hold is a map from each a ∈ A to a record:
# hold.yml — one entry per action (synthetic, Big Widget warranty-claims assistant).
# A controlled record: it names people, so it is not shared the way a receipt is.
- id: HOLD-WC-03 # the reference a receipt carries instead of a name
action: send_rejection_letter
authority: approver # alone | approver | never
holder: dana.okafor # one natural person, resolvable in the directory
release_by: [dana.okafor] # the holder, plus anyone she names here on the record
alert_route: warranty-oncall # rota with escalation to the holder; routing, not accountability
approval_ref: APR-2026-0412 # standing approval to operate; a reference, never its content
expires: 2027-05-31
decision: per_case # approver: a named reviewer's decision bound to this case
budget: {per_reviewer_per_hour: 50, system_per_hour: 120}
out_of_band: false
stop:
on: [holder_inactive, approval_expired, approval_record_missing, budget_exceeded,
budget_unavailable, monitor_down, latch_unreadable, fault_journal_unreadable]
action: pause_latch_and_page # never "log_and_continue"; stays paused until a named release
# (journal unreadable: the executor halts until a named restart)
per_case: {on: [decision_missing], action: pause_this_case}
- id: HOLD-FN-01
action: refund_dealer_over_10k
authority: approver
holder: sam.reyes
release_by: [sam.reyes]
approval_ref: APR-2026-0388
expires: 2027-02-28
decision: per_case
out_of_band: true # call-back on the number already on file, bound to this refund
stop: {on: [approval_expired, holder_inactive], action: pause_latch_and_page}
per_case: {on: [oob_unconfirmed, decision_missing], action: deny_or_pause_this_case}
- id: HOLD-WC-09
action: change_dealer_bank_details
authority: never
Two kinds of stop are in play. A system-level stop (holder gone, approval missing or expired, budget breached or unreadable, monitor down, latch store unreadable) latches: it pauses this action for every case until a named release is recorded. A per-case stop (no decision on this case, no call-back for this refund) holds only that case. It ends when the missing human record exists, which is itself a named person's act.
Two approvals are in play and they are different objects. The standing approval (approval_ref, expires) authorises the system to perform this class of action at all. The per-case decision authorises this execution: a named, active reviewer's decision, recorded against this action's case. For authority: approver, a current standing approval without a per-case decision is not sufficient.
The holder is a single accountable person; alert_route may be an on-call rota with escalation. Paging a team is technically routine. Named accountability is required because someone must own the decision to let the action run and the decision to stop or resume it, not because a team cannot receive an alert.
Fail-closed evaluation
The gate is a predicate evaluated before each execution. Its default is deny; every missing or unreadable input evaluates to "not held". A system-level stop is written to a persistent latch store before the gate answers, and every later evaluation reads the latch first.
def held(action, now, directory, approvals, decisions, budget, monitor, confirmations,
latches, faults):
h = hold.get(action.name)
if h is None or h.authority == "never":
return Deny("not in authority table") # default deny
dec = decisions.get(action.case_id) if h.authority == "approver" else None
scopes = [h.id] + ([(h.id, dec.reviewer)] if dec else []) # action-wide, then per reviewer
def stop(reason, page, scope=h.id): # latch first, then answer
latches.open(scope, reason=reason, at=now) # raises if it cannot persist
return Pause(reason, page=page)
def released(state): # a valid release, recorded after
rel = state.release # the stop by an active person
return not (rel is None or rel.at <= state.opened_at # on release_by
or rel.by not in h.release_by
or not directory.is_active(rel.by))
f = faults.read(h.id) # 0. a fault in the latch store is
if f is None: # kept in a separate journal
raise StoreUnavailable("fault journal unreadable") # executor halts (see below)
if f != "clear":
if not released(f):
return Pause("store fault: waiting for a named release", page=h.holder)
faults.close(h.id, release=f.release.id)
for s in scopes: # 1. an open latch stays open
state = latches.read(s) # "clear", a Latch, or None
if state is None: # record the fault outside the
faults.open(h.id, reason="latch store unreadable", at=now) # failed store; raises
return Pause("latch store unreadable", page=h.holder) # if it cannot persist
if state != "clear":
if not released(state):
return Pause("paused: waiting for a named release", page=h.holder)
latches.close(s, release=state.release.id) # released; re-check everything below
if not directory.is_active(h.holder): # 2. system-level stops latch
return stop("holder inactive", page=directory.manager_of(h.holder))
rec = approvals.get(h.approval_ref) # standing approval
if rec is None:
return stop("approval record missing", page=h.holder)
if now.date() > h.expires:
return stop("approval expired", page=h.holder)
if not monitor.healthy():
return stop("monitor down", page=h.holder) # unobserved == not held
if h.authority == "approver": # 3. per-case: holds this case only
if (dec is None or dec.action != action.name
or dec.case_id != action.case_id
or not directory.is_active(dec.reviewer)):
return Pause("no named decision for this case", page=h.holder)
if h.budget is not None: # live budget verdict
v = budget.verdict(action.name, reviewer=dec.reviewer if dec else None, at=now)
if v is None:
return stop("budget verdict unavailable", page=h.holder)
if v.system_exceeded:
return stop("system budget exceeded", page=h.holder)
if v.reviewer_exceeded:
return stop("reviewer budget exceeded", page=h.holder, scope=(h.id, dec.reviewer))
if h.out_of_band: # bound to this transaction
c = confirmations.get(action.id)
if (c is None or c.contact_ref != directory.contact_on_file(action.payee)
or c.payee != action.payee or c.amount != action.amount):
return Deny("out-of-band confirmation missing or not for this action")
return Allow(receipt=Receipt(action_id=action.id, hold_ref=h.id, approval_ref=rec.id,
decision_ref=dec.id if dec else None, answer="allow",
hold_version=hold.version_hash()))
A release is honoured only if it was recorded after the latch opened, by a person on release_by who is still active. Closing the latch does not by itself allow anything: the evaluation carries on through every check below it, so a release recorded while the cause is still present latches again at once.
An unreadable latch store is itself a system-level stop, so it needs a record that survives the store coming back. Refusing the first evaluation is not enough: if the store later reads "clear", nothing in it shows that a stop ever happened, and the next evaluation would allow. So the gate writes the fault to a separate fault journal (a different store, not a second table in the failed one) and reads that journal before anything else. The fault clears only by the same named release as a latch.
That moves the question one step back: what if the journal fails too? Any store can fail, so the last step cannot be another store inside the gate. It is an executor interlock:
def execute(action, gate_inputs, executor):
if executor.halted(action.name): # a halt ends only by a person's restart
return Pause("executor halted: waiting for a named restart")
try:
answer = held(action, **gate_inputs)
except Exception as err: # unreadable journal, failed latch write, ...
executor.halt(action.name, reason=repr(err)) # no automatic restart
return Pause("gate error: executor halted")
if isinstance(answer, Allow):
return executor.run(action, receipt=answer.receipt)
return answer # Constrain handling is not shown here
The executor's halt is the place where "a dependency came back" must not turn into "go". Its restart is a person's act, checked the way a release is: by someone on release_by, active, recorded after the halt. How the executor keeps its halted state through its own crash or redeploy is a property of the real executor, and this sketch does not show it. If it cannot keep that state, it must start halted. The sketch does not prove that it does.
Counterexamples a test must include, each removing or changing exactly one input:
| Input removed or changed | Expected result |
|---|---|
Standing approval current, no per-case decision for send_rejection_letter | Pause("no named decision for this case"); no latch |
| Decision exists but for a different case or action | Pause(...): a decision is not reusable |
Monitor healthy, budget.verdict reports reviewer or system exceeded | Pause(... "budget exceeded"), latched |
budget.verdict returns None (count unavailable) | Pause("budget verdict unavailable"), latched |
| Confirmation present but for a different amount or payee | Deny(...) |
| Holder inactive, approval missing, approval expired, monitor down | Pause(...) as listed, latched |
| Latch store unreadable | Pause("latch store unreadable"); fault written to the journal |
| Fault journal unreadable | StoreUnavailable raised; execute halts the executor and pauses |
And one sequence, the restart test, run on the same action and case:
| Step | Inputs | Expected result |
|---|---|---|
| 1 | Budget verdict None | Pause("budget verdict unavailable"); latch opens |
| 2 | Verdict readable and under budget; no release recorded | Pause("paused: waiting for a named release") |
| 3 | Release recorded before the latch opened, or by someone not on release_by | Still paused |
| 4 | Release recorded after the latch opened by Dana, who is active | Latch closes; checks re-run; Allow |
The same sequence with monitor.healthy() going false, then true, must give the same results. Without the latch, step 2 would allow: the dependency recovered, but no person said yes again.
Two more sequences cover the stores themselves, starting with no latch open:
| Step | Inputs | Expected result |
|---|---|---|
| 1 | latches.read returns None | Pause("latch store unreadable"); fault opens in the journal |
| 2 | Latch store readable again and "clear"; no release recorded | Pause("store fault: waiting for a named release") |
| 3 | Release recorded after the fault opened, by Dana, who is active | Fault closes; checks re-run; Allow |
| 1′ | faults.read returns None | execute halts the executor; Pause("gate error: executor halted") |
| 2′ | Journal and latch store both readable and clear; no restart recorded | Pause("executor halted: waiting for a named restart") |
| 3′ | Restart recorded after the halt, by Dana, who is active | Halt ends; checks re-run; Allow |
Step 2 is the case the earlier sketch got wrong: it paused once while the store was unreadable, then allowed as soon as the store read "clear", with no release.
monitor.healthy() reports availability; it does not enforce a volume limit. A healthy monitor observing a breach must still change the result, which is why the budget verdict is a separate input. The out-of-band confirmation is a record looked up from the call-back channel and bound to the action's id, payee and amount, not a boolean the requester can set.
The fail-open bug to look for is any branch where an exception, timeout or None falls through to Allow (for example a try/except: pass around the approval lookup). Test each on: condition with a dry run that removes exactly that input and asserts the action does not execute; module 300.9 builds and proves that stop. This sketch does not establish that every exception path is caught.
Worked case (quiz idea): the approval was granted eight months ago by a holder who has since left. directory.is_active(holder) is false, so the predicate pauses and latches regardless of expires, and pages someone who can name a new holder. A system whose holder is inactive is not held. The holder cannot release this latch (they are inactive), so the only way back is a person with that authority naming a new holder and release_by in the hold itself, then recording a release.
Receipts and coverage
The Receipt carries references and hashes only: the action's id, the hold entry's id (HOLD-WC-03), the approval and decision record ids, the gate's answer, a hash of the hold version in force and an undo reference. It carries no names, directory keys, contact details, claim text, letter text, prompts or model output. The holder's and reviewer's identities stay in the hold and decision records, which are access-controlled work records that the references resolve to for people entitled to see them. A reference is not a licence to disclose what it points to.
This is the course's boundary, not a privacy-law ruling. Replacing dana.okafor with HOLD-WC-03 keeps the name out of the receipt; it does not make the receipt anonymous. Anyone who can open the hold can resolve the reference to Dana, and whether such a reference counts as personal data under a given law is not checked here. That question belongs to the organisation's privacy owner. If an organisation wanted holder identities inside its receipts, that would be an exception to the course rule, decided and recorded by a person, not a quiet change to the schema. This sketch also does not sign the receipt; signing and offline verification are not shown in this module.
Coverage, matched action by action
The hold also reports coverage. Count it by action identity, not by comparing two totals, and check every execution record, not just the first one for each id:
- execution coverage = execution records whose action's latest gate evaluation before that execution was
AlloworConstrain÷ all execution records, per bucket; - contrary to the gate: execution records whose latest prior evaluation was anything else (
Pause,Deny). These are violations, reported by id and log position, never folded into a percentage; - no evaluation: execution records with no prior evaluation for their id at all. These are bypasses, also reported by id and log position;
- ran more than once: ids with more than one execution record. Each record has already been checked on its own above. A person then finds out whether the action really ran twice or was logged twice; the helper cannot tell those apart.
An earlier version of this sketch kept only the first execution of each id. Then Allow, run, Pause, run again on one id reported 1 / 1 covered and no violation: the second run, after the Pause, disappeared. Checking each execution against its own latest prior evaluation is what makes that second run visible.
Proposals the gate refused never ran, so they belong in neither the numerator nor the denominator. Count them separately, by answer.
"Latest" and "before" need an order the records can prove. Clock times cannot: two records can carry the same time, and then the answer would depend on which one the code happened to read first. The contract here is that the gate and the executor both write to one append-only log, and the log gives each record a seq number that only goes up. Timestamps stay for people to read; order comes from seq. Each record must also carry the action's id, issued when the action was proposed. The function checks both before counting, and says "cannot count yet" rather than guessing.
def coverage(evaluations, executions):
evaluations, executions = list(evaluations), list(executions)
seqs = []
for r in evaluations + executions: # validate before counting anything
if getattr(r, "action_id", None) in (None, "") or getattr(r, "seq", None) is None:
return "cannot count yet: a record has no action id or no log position"
seqs.append(r.seq)
if len(seqs) != len(set(seqs)):
return "cannot count yet: two records share a log position, so their order is unknown"
if not executions:
return "nothing executed in this bucket" # not 100%, and not 0%
covered, contrary, unchecked, runs = 0, [], [], {}
for e in sorted(executions, key=lambda r: r.seq): # every execution, not just the first
runs[e.action_id] = runs.get(e.action_id, 0) + 1
prior = [g for g in evaluations if g.action_id == e.action_id and g.seq < e.seq]
if not prior:
unchecked.append((e.action_id, e.seq))
elif max(prior, key=lambda g: g.seq).answer in ("Allow", "Constrain"):
covered += 1
else: # Pause, Deny, or anything not a yes
contrary.append((e.action_id, e.seq))
return {"covered": covered, "executed": len(executions),
"contrary_to_gate": contrary, "no_evaluation": unchecked,
"ran_more_than_once": sorted(a for a, n in runs.items() if n > 1)}
| Synthetic case | Result |
|---|---|
| Gate evaluates A (denied) and B (paused); executor runs C and D | 0 / 2 covered; no_evaluation = C and D, each with its seq. Comparing totals would have said 2 ÷ 2 = 100%. |
| Gate evaluates A (allowed), B and E (refused); executor runs A | 1 / 1 covered. B and E are refused proposals, reported apart. |
| Executor runs B after the gate paused it | 0 / 1 covered; contrary_to_gate = B, with its seq |
A: Allow (seq 1), run (2), Pause (3), run again (4); read in either input order | 1 / 2 covered; contrary_to_gate = A at seq 4; ran_more_than_once = A |
| A: Allow (1), run (2), run (3), with no evaluation between | 2 / 2 covered; ran_more_than_once = A, for a person to find out whether it ran twice or was logged twice |
| Nothing executed | "nothing executed in this bucket" |
Executor does not log action ids, or logs None for some | "cannot count yet"; records with no id are never joined as one action |
A record has no seq | "cannot count yet" |
| A: Allow and Pause at the same clock time, then executed; read in either input order | Same answer both ways, decided by seq: if the Pause has the higher seq, 0 / 1 and contrary_to_gate = A |
Two records with the same seq | "cannot count yet": order unknown |
The tied-time row must be run twice, with the two evaluations swapped in the input, and give the same result both times; so must the run-Pause-run row, with each list reversed. The checks above catch missing ids and repeated positions. The ran_more_than_once list does not decide whether a repeated id means two real effects; it makes sure a person is asked. They cannot show that an id was issued honestly or that the log's order matches what happened; those are properties of the real log and executor, not shown here.
For Constrain, checking that the executed action stayed inside the constraint is a further check, not shown here. Coverage can only see executions the executor logs: an action run by a path that writes no log is invisible to it. An empty breach list over an uninstrumented period is not evidence of coverage.
Out-of-band confirmation
An out-of-band check moves identity verification to a channel the requester does not control. The reference failure: the Hong Kong Government's 26 June 2024 Legislative Council reply describes a case received by the Police in end-January 2024 in which deepfake technology was used to "fabricate a pre-recorded video conference"; "the informant eventually gave authorisation to transfer funds to five local bank accounts" and "suffered a loss of about HK$200 million".1 "Since the meeting was pre-recorded, there was actually no interaction between the informant and the fraudster."1 The record does not name the organisation, and neither does this course. Separately, as general advice, the Police recommend "verifying with a phone call when a relative or a friend makes a request for remittance".1 The course generalises that narrower advice into its own rule: in the hold, out_of_band: true means a confirmation record from a call-back to a contact already on file, bound to this transaction, and recorded by reference in the receipt.
Relation to Article 14 of the EU AI Act
For high-risk systems, Article 14(1) requires design such that they "can be effectively overseen by natural persons during the period in which they are in use", and Article 14(4)(e) requires that the persons assigned oversight be enabled, as appropriate and proportionate, "to intervene in the operation of the high-risk AI system or interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state."2 The hold's holder and stop fields are the course's operational form of that idea; whether Article 14 binds a given system, and from what date, is 300.7's analysis. Implementing this gate is not a claim of compliance with the Act.
2. Aggregation and budgets
Aggregation
Let each action carry a small individual risk. The risk of the system is a function of volume and concentration, not only of the per-action risk: n low-stakes acts by one principal in one window can constitute a high-stakes act (for example, a pattern of denials with no individual review). A single action can of course already be serious on its own. The course calls the volume effect aggregation.
The warning case: ProPublica reported on 25 March 2023, from "corporate documents and interviews with former Cigna officials", that "Over a period of two months last year, Cigna doctors denied over 300,000 requests for payments using this method", "spending an average of 1.2 seconds on each case".3 Asked whether PXDX lets Cigna doctors reject claims without examining them, Cigna told ProPublica that description was "incorrect"; it also said that "Reviews occur after the service has been provided to the patient and does not result in any denials of care".3 Cigna's "incorrect" is addressed to that description of the review process, not to the totals. The class-action complaint in Kisting-Leung v. Cigna Corp. (E.D. Cal., filed 24 July 2023) alleges the same totals at ¶2 and cites ProPublica for them (footnote 3).4 The two are therefore one count, reported then pleaded, not two independent measurements, and neither is a finding. The course treats them as alleged totals only. It does not derive a rate from them and does not use them as inputs below.
Budget specification
A budget is a set of constraints over a bucketed count c(p, t) of counted actions by principal p in bucket t (here, one hour). In the example the counted action is a reviewer's decision taking effect.
- per-principal: for all p, t: c(p, t) ≤ B<sub>p</sub>
- system-wide: for all t: Σ<sub>p</sub> c(p, t) ≤ B<sub>sys</sub>
Both are evaluated on each bucket (the maximum), never on the mean over the window.
Detection and enforcement are different controls. Detection is retrospective: compute c from the log and list breaches. Enforcement is prospective: the gate consults a live count before each counted action (the budget.verdict call in section 1) and returns Pause for the action that would make c(p, t) exceed B<sub>p</sub>, or the system sum exceed B<sub>sys</sub>. The stop must act on the counted action itself. Withholding new arrivals from p is a different, weaker control (an arrivals budget): p could still clear an existing queue past B<sub>p</sub>. On a per-principal breach:
- further decisions by p do not take effect (no letter is sent on them), and the latch keeps it that way after the bucket ends, until a named release;
- cases already queued to p stay queued; none is auto-approved or auto-rejected;
- the holder is paged and decides, on the record, whether to reassign queued cases to principals under budget or let them wait;
- new arrivals to p stop until the holder releases the pause.
A system-wide breach applies the same to all principals. A new hour starting is not a release: the count resets, the latch does not. This sketch does not solve concurrent decisions racing the counter; a real implementation needs an atomic reserve-then-commit or equivalent, and that is not specified here.
Worked example on the synthetic log
Course-owned synthetic log (hold-prompts.md); the numbers are invented. Each cell counts decisions that took effect, recorded with no live enforcement running, so this is a detection exercise. B<sub>p</sub> = 50, B<sub>sys</sub> = 120.
| Hour | R1 | R2 | R3 | System total |
|---|---|---|---|---|
| 09:00 | 12 | 10 | 11 | 33 |
| 10:00 | 14 | 9 | 13 | 36 |
| 11:00 | 71 | 12 | 10 | 93 |
| 12:00 | 8 | 11 | 9 | 28 |
log = {"09:00": {"R1": 12, "R2": 10, "R3": 11},
"10:00": {"R1": 14, "R2": 9, "R3": 13},
"11:00": {"R1": 71, "R2": 12, "R3": 10},
"12:00": {"R1": 8, "R2": 11, "R3": 9}}
B_P, B_SYS = 50, 120
breaches = [(t, p, n) for t, row in log.items() for p, n in row.items() if n > B_P]
sys_breaches = [(t, sum(row.values())) for t, row in log.items() if sum(row.values()) > B_SYS]
# breaches == [("11:00", "R1", 71)]
# sys_breaches == [] # peak system hour is 93
- Per-principal maximum: 71 (R1, 11:00) > 50: breach.
- System-wide maximum: 93 (11:00) ≤ 120: no breach. A system-only budget misses the event.
- Means: 190 actions over 12 reviewer-hours ≈ 15.8 per reviewer-hour; R1 alone 105 / 4 = 26.25 per hour. Both are under 50. The mean dilutes a single-bucket spike by the number of buckets averaged.
- With enforcement live, R1's 51st decision at 11:00 would have paused, and the 11:00 cell would read 50.
These comprehensions detect; they do not stop anything. An empty breaches list over an empty or partial log means no breach was observed, not that the system was within budget.
Design notes: choose the bucket to match the harm (an hour for reviewer overload; a day may hide it); use fixed buckets for reporting and a sliding window for enforcement so a spike across a bucket boundary (e.g. 10:30 to 11:30) is not split (the fixed-hour exercise above does not implement this); report every count with its denominator and bucket.
3. Rubber-stamp review
Definition
A review is a rubber stamp when the recorded decision is attributed to a person but the person did not form an independent judgment: the output of the system passes through unchanged and unexamined. Formally, oversight requires that the reviewer had (i) access to the case evidence, (ii) time to use it, (iii) authority to depart without penalty, and (iv) that departures are possible and recorded. A rubber stamp lacks at least one in practice.
The Cigna complaint alleges the pattern: "Cigna's doctors instantly reject claims on medical grounds without ever opening patient files" (¶2), and Cigna "allowed its doctors to sign off on the denials in batches" (¶5).4 ProPublica quotes a former Cigna doctor: "We literally click and submit" and "It takes all of 10 seconds to do 50 at a time."3 Cigna disputes the description.3 The Lokken complaint alleges a constraint on condition (iii): "targets to keep stays at skilled nursing facilities within 1% of the days projected by the AI Model", and that "Employees who deviate from the nH Predict AI Model projections are disciplined and terminated".5 All of these are allegations.
Why it is not oversight
Article 14(4)(b) of the AI Act requires that persons assigned oversight of a high-risk system be enabled "to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)", in particular for systems "used to provide information or recommendations for decisions to be taken by natural persons"; 14(4)(d) requires that they be able "to decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output".2 A reviewer without the evidence, the time or the freedom to override cannot exercise (d), so the signature records attribution, not oversight.
Detection from logs
Given a review log with events evidence_view(case, reviewer, ts, channel) (a file open, or any other route by which the case evidence is presented) and decision(case, reviewer, ts, outcome, model_outcome), compute, each with its denominator:
| Metric | Definition | Rubber-stamp signal |
|---|---|---|
| View rate | cases with ≥ 1 evidence_view before decision ÷ decided cases | near 0 |
| Dwell time | decision.ts − first evidence_view.ts per case; report the distribution | low median, low variance |
| Override rate | cases where outcome ≠ model_outcome ÷ decided cases | 0 over a large n |
| Batch share | decisions sharing one submit event ÷ decided cases | high |
| Load | decisions per reviewer per bucket (section 2) | over budget |
None is proof alone, and together they are evidence for a person to read, not a verdict. A correct model can have a low override rate; there is no target override rate, and a quota would itself corrupt review. A view event shows the evidence was presented, not that it was understood. If no evidence_view is logged, the view rate and dwell time are "cannot count yet", and the hold must say so rather than infer review or its absence. A fallback such as time since the reviewer's previous decision is an inter-decision interval, a proxy; report it under that name and never as measured reading time.
4. Claim types: allegation, statement, gate
The record
| Claim type | Source | Content |
|---|---|---|
| Allegation | Estate of Gene B. Lokken v. UnitedHealth Group, Inc., No. 0:23-cv-03514 (D. Minn.), complaint filed 14 Nov 20235 | ¶1: "illegal deployment of artificial intelligence (AI) in place of real medical professionals to wrongfully deny elderly patients care owed to them under Medicare Advantage Plans by overriding their treating physicians' determinations as to medically necessary care based on an AI model that Defendants know has a 90% error rate." ¶6: coverage determinations "not based on individual patient's needs, but based on the outputs of the nH Predict AI Model". |
| Company statement | UnitedHealth spokesperson, emailed to STAT, 14 Nov 20236 | The naviHealth predict tool is not used to make coverage determinations; "The tool is used as a guide to help us inform providers, families and other caregivers"; "Coverage decisions are based on CMS coverage criteria and the terms of the member's plan"; "lawsuit has no merit and we will defend ourselves vigorously". |
| Company assertion, reported by STAT | Same STAT article, separate from the emailed statement6 | "UnitedHealth has asserted in response to STAT’s reporting that its physician medical reviewers provide a check" "by making a final determination about whether a patient is issued a payment denial". STAT adds, in its own voice: "But those physicians are reviewing denial recommendations sent to them by clinical case managers" "who are subject to the company’s 1% performance target". |
| Gate | The course (Practice 300 Hold, hold-prompts.md) | A model may inform but not replace clinical judgment in a coverage decision. |
The 90% figure is the complaint's allegation; its basis is not checked here. The outcome of the case is not checked and is not stated.
Treat claim type as data. In notes and incident reports (work records, not receipts), every case fact carries its type and source, and nothing typed allegation or statement is ever restated as finding.
Invariance of the gate
Let H<sub>C</sub> be "the model's output replaced clinical judgment" (the complaint) and H<sub>U</sub> be "the tool is a guide; coverage decisions rest on CMS coverage criteria and plan terms" (the company's statement). The gate G requires: for each case where the model's predicted stay m and the treating clinician's determination d differ, a qualified person reviews that case and decides, and the record names who and why. G does not prescribe that d prevails; it prescribes who decides.
- Under H<sub>C</sub>, G is violated, and the hold would have stopped or flagged it.
- Under H<sub>U</sub>, G is consistent with the described process but not shown by it. The emailed statement does not say who reviews each case. The company has also asserted, as STAT reports, that physician medical reviewers make the final determination on a payment denial; STAT reports that those physicians review recommendations from case managers subject to a 1% target.6 Both are accounts, not records of each case. The hold makes the question checkable per case (who decided, whether they saw the evidence, view rate, override rate, section 3).
G's requirement is the same under both, so the builder need not resolve the dispute to implement it.
Against "accurate on average": a model's aggregate error rate is a population statistic. It can inform an individual case but does not decide it, because the gate is evaluated per case. This is section 2 inverted: aggregation hides individual harms just as averages hide peaks.
Big Widget implementation
- id: HOLD-WC-05
action: approve_repair_hours
authority: approver
holder: dana.okafor # in the controlled hold; receipts carry HOLD-WC-05, not the name
release_by: [dana.okafor]
expires: 2027-05-31
decision: per_case
judgment: {informs: service_planner_estimate, decides: examining_engineer}
on_disagreement: engineer_figure_stands; reason_code_required # Big Widget's own repair rule
stop: {on: [no_engineer_decision_on_record], action: hold_job_open}
report: [override_rate_with_denominator, view_rate_or_cannot_count_yet, coverage_or_cannot_count_yet]
An override rate of 0 out of 2,000 jobs is a signal to investigate (section 3), not a success metric and not, on its own, proof of a rubber stamp.
Further learning (free university courses)
Optional background, not sources: University of Helsinki, Ethics of AI (https://ethics-of-ai.mooc.fi/), on accountability for automated decisions; MIT OpenCourseWare, 6.S897 Machine Learning for Healthcare (Spring 2019) (https://ocw.mit.edu/courses/6-s897-machine-learning-for-healthcare-spring-2019/), on clinical models.