Level 300: Building with gen AI · Module 300.8

At the café counter, a man in an apron hands a brass key with a blank tag to a colleague, beside a blank clipboard. The chalkboard reads “Who holds it?”
Who holds it?

Hold: the operate gate · Technical gear

Reviewed and published, version 1.0.0. As of 19 Nov 2026. It is not a credential. Big Widget Company, its people, systems and logs are made up. Court cases are cited as pleadings and reporting. Nothing here is a finding, and case outcomes are not checked. The code is a teaching sketch. It has not been run against a real executor, it does not handle concurrency, and it is not a library to deploy.

Objectives. 300.8-a: bind each action of a running system to a named holder, an approval expiry and a fail-closed stop. 300.8-b: model aggregation and enforce a per-principal and system-wide budget over time buckets, on peaks, not means. 300.8-c: detect rubber-stamp review from log evidence and explain why it is not oversight. 300.8-d: keep allegation, company statement and gate as separate claim types, and show the gate's requirement is invariant to which account is true.

The operate gate is the course's rule set (the Practice 300 Hold, as the course-owned prompts in levels/300/practice/hold-prompts.md). It is not a statute. Where law states a related duty, it is quoted and cited.

Lifecycle and decision rights

In the course's three yeses (module 100.9), the hold written before launch is Get to Yes. The operate gate, evaluated before every execution, is Stay at Yes. A stop opens Recover to Yes, which closes only when a named person has inspected the cause, repaired it and recorded a new yes. A paused system is not evidence of recovery, and a dependency that comes back is not a new yes.

The predicate returns one of the course's four gate answers: Allow (go ahead), Constrain (only this far), Pause (wait for a person) and Deny (no). This module writes the third as Pause to keep it distinct from the hold record. The gate decides by rule. A system-level Pause is latched: it persists across evaluations until a named person (the holder, or someone the holder names on the record) records a release. The model or agent never releases it, and neither does the recovery of whatever caused it.

The model proposes; an executor runs only what the gate allowed. There is no direct path from the model to the tool for these actions, and a system prompt is not that boundary.

1. The hold as a policy over actions

Card 1, HOLDER · COURSE RULE: Name one person. Write who answers for this action and when their approval expires. Card 2, STOP · COURSE RULE: Fail closed. Holder, approval or record missing or expired? Do not run; page a person. Card 3, LEGAL CONTEXT [s6]: High-risk systems must be stoppable. Article 14: overseers must be able to intervene or halt it safely, as appropriate and proportionate.
The holder, expiry and fail-closed rule are course-owned procedure. Article 14 supplies the separate high-risk oversight context (s6); this summary does not show the whole operate gate or establish legal compliance.
Figure data as text

Who holds this action?

FocusLesson point
HOLDER · COURSE RULE · Name one personWrite who answers for this action and when their approval expires.
STOP · COURSE RULE · Fail closedHolder, approval or record missing or expired? Do not run; page a person.
LEGAL CONTEXT [s6] · High-risk systems must be stoppableArticle 14: overseers must be able to intervene or halt it safely, as appropriate and proportionate.

The holder, expiry and fail-closed rule are course-owned procedure. Article 14 supplies the separate high-risk oversight context (s6); this summary does not show the whole operate gate or establish legal compliance.

Objective: 300.8-a. Text equivalent of figures/who-holds-this-action.svg.

Definitions

Let a system expose a finite action set A (the authority table from 300.4: alone, approver, never; anything not in A is denied). A hold is a map from each a ∈ A to a record:

# hold.yml — one entry per action (synthetic, Big Widget warranty-claims assistant).
# A controlled record: it names people, so it is not shared the way a receipt is.
- id: HOLD-WC-03                 # the reference a receipt carries instead of a name
  action: send_rejection_letter
  authority: approver            # alone | approver | never
  holder: dana.okafor            # one natural person, resolvable in the directory
  release_by: [dana.okafor]      # the holder, plus anyone she names here on the record
  alert_route: warranty-oncall   # rota with escalation to the holder; routing, not accountability
  approval_ref: APR-2026-0412    # standing approval to operate; a reference, never its content
  expires: 2027-05-31
  decision: per_case             # approver: a named reviewer's decision bound to this case
  budget: {per_reviewer_per_hour: 50, system_per_hour: 120}
  out_of_band: false
  stop:
    on: [holder_inactive, approval_expired, approval_record_missing, budget_exceeded,
         budget_unavailable, monitor_down, latch_unreadable, fault_journal_unreadable]
    action: pause_latch_and_page  # never "log_and_continue"; stays paused until a named release
                                  # (journal unreadable: the executor halts until a named restart)
  per_case: {on: [decision_missing], action: pause_this_case}
- id: HOLD-FN-01
  action: refund_dealer_over_10k
  authority: approver
  holder: sam.reyes
  release_by: [sam.reyes]
  approval_ref: APR-2026-0388
  expires: 2027-02-28
  decision: per_case
  out_of_band: true              # call-back on the number already on file, bound to this refund
  stop: {on: [approval_expired, holder_inactive], action: pause_latch_and_page}
  per_case: {on: [oob_unconfirmed, decision_missing], action: deny_or_pause_this_case}
- id: HOLD-WC-09
  action: change_dealer_bank_details
  authority: never

Two kinds of stop are in play. A system-level stop (holder gone, approval missing or expired, budget breached or unreadable, monitor down, latch store unreadable) latches: it pauses this action for every case until a named release is recorded. A per-case stop (no decision on this case, no call-back for this refund) holds only that case. It ends when the missing human record exists, which is itself a named person's act.

Two approvals are in play and they are different objects. The standing approval (approval_ref, expires) authorises the system to perform this class of action at all. The per-case decision authorises this execution: a named, active reviewer's decision, recorded against this action's case. For authority: approver, a current standing approval without a per-case decision is not sufficient.

The holder is a single accountable person; alert_route may be an on-call rota with escalation. Paging a team is technically routine. Named accountability is required because someone must own the decision to let the action run and the decision to stop or resume it, not because a team cannot receive an alert.

Fail-closed evaluation

The gate is a predicate evaluated before each execution. Its default is deny; every missing or unreadable input evaluates to "not held". A system-level stop is written to a persistent latch store before the gate answers, and every later evaluation reads the latch first.

def held(action, now, directory, approvals, decisions, budget, monitor, confirmations,
         latches, faults):
    h = hold.get(action.name)
    if h is None or h.authority == "never":
        return Deny("not in authority table")                 # default deny

    dec = decisions.get(action.case_id) if h.authority == "approver" else None
    scopes = [h.id] + ([(h.id, dec.reviewer)] if dec else [])  # action-wide, then per reviewer

    def stop(reason, page, scope=h.id):                       # latch first, then answer
        latches.open(scope, reason=reason, at=now)            # raises if it cannot persist
        return Pause(reason, page=page)

    def released(state):                                      # a valid release, recorded after
        rel = state.release                                   # the stop by an active person
        return not (rel is None or rel.at <= state.opened_at  # on release_by
                    or rel.by not in h.release_by
                    or not directory.is_active(rel.by))

    f = faults.read(h.id)                                     # 0. a fault in the latch store is
    if f is None:                                             #    kept in a separate journal
        raise StoreUnavailable("fault journal unreadable")    # executor halts (see below)
    if f != "clear":
        if not released(f):
            return Pause("store fault: waiting for a named release", page=h.holder)
        faults.close(h.id, release=f.release.id)

    for s in scopes:                                          # 1. an open latch stays open
        state = latches.read(s)                               # "clear", a Latch, or None
        if state is None:                                     # record the fault outside the
            faults.open(h.id, reason="latch store unreadable", at=now)  # failed store; raises
            return Pause("latch store unreadable", page=h.holder)       # if it cannot persist
        if state != "clear":
            if not released(state):
                return Pause("paused: waiting for a named release", page=h.holder)
            latches.close(s, release=state.release.id)        # released; re-check everything below

    if not directory.is_active(h.holder):                     # 2. system-level stops latch
        return stop("holder inactive", page=directory.manager_of(h.holder))
    rec = approvals.get(h.approval_ref)                       # standing approval
    if rec is None:
        return stop("approval record missing", page=h.holder)
    if now.date() > h.expires:
        return stop("approval expired", page=h.holder)
    if not monitor.healthy():
        return stop("monitor down", page=h.holder)            # unobserved == not held

    if h.authority == "approver":                             # 3. per-case: holds this case only
        if (dec is None or dec.action != action.name
                or dec.case_id != action.case_id
                or not directory.is_active(dec.reviewer)):
            return Pause("no named decision for this case", page=h.holder)

    if h.budget is not None:                                  # live budget verdict
        v = budget.verdict(action.name, reviewer=dec.reviewer if dec else None, at=now)
        if v is None:
            return stop("budget verdict unavailable", page=h.holder)
        if v.system_exceeded:
            return stop("system budget exceeded", page=h.holder)
        if v.reviewer_exceeded:
            return stop("reviewer budget exceeded", page=h.holder, scope=(h.id, dec.reviewer))

    if h.out_of_band:                                         # bound to this transaction
        c = confirmations.get(action.id)
        if (c is None or c.contact_ref != directory.contact_on_file(action.payee)
                or c.payee != action.payee or c.amount != action.amount):
            return Deny("out-of-band confirmation missing or not for this action")

    return Allow(receipt=Receipt(action_id=action.id, hold_ref=h.id, approval_ref=rec.id,
                                 decision_ref=dec.id if dec else None, answer="allow",
                                 hold_version=hold.version_hash()))

A release is honoured only if it was recorded after the latch opened, by a person on release_by who is still active. Closing the latch does not by itself allow anything: the evaluation carries on through every check below it, so a release recorded while the cause is still present latches again at once.

An unreadable latch store is itself a system-level stop, so it needs a record that survives the store coming back. Refusing the first evaluation is not enough: if the store later reads "clear", nothing in it shows that a stop ever happened, and the next evaluation would allow. So the gate writes the fault to a separate fault journal (a different store, not a second table in the failed one) and reads that journal before anything else. The fault clears only by the same named release as a latch.

That moves the question one step back: what if the journal fails too? Any store can fail, so the last step cannot be another store inside the gate. It is an executor interlock:

def execute(action, gate_inputs, executor):
    if executor.halted(action.name):                  # a halt ends only by a person's restart
        return Pause("executor halted: waiting for a named restart")
    try:
        answer = held(action, **gate_inputs)
    except Exception as err:                          # unreadable journal, failed latch write, ...
        executor.halt(action.name, reason=repr(err))  # no automatic restart
        return Pause("gate error: executor halted")
    if isinstance(answer, Allow):
        return executor.run(action, receipt=answer.receipt)
    return answer                                     # Constrain handling is not shown here

The executor's halt is the place where "a dependency came back" must not turn into "go". Its restart is a person's act, checked the way a release is: by someone on release_by, active, recorded after the halt. How the executor keeps its halted state through its own crash or redeploy is a property of the real executor, and this sketch does not show it. If it cannot keep that state, it must start halted. The sketch does not prove that it does.

Counterexamples a test must include, each removing or changing exactly one input:

Input removed or changedExpected result
Standing approval current, no per-case decision for send_rejection_letterPause("no named decision for this case"); no latch
Decision exists but for a different case or actionPause(...): a decision is not reusable
Monitor healthy, budget.verdict reports reviewer or system exceededPause(... "budget exceeded"), latched
budget.verdict returns None (count unavailable)Pause("budget verdict unavailable"), latched
Confirmation present but for a different amount or payeeDeny(...)
Holder inactive, approval missing, approval expired, monitor downPause(...) as listed, latched
Latch store unreadablePause("latch store unreadable"); fault written to the journal
Fault journal unreadableStoreUnavailable raised; execute halts the executor and pauses

And one sequence, the restart test, run on the same action and case:

StepInputsExpected result
1Budget verdict NonePause("budget verdict unavailable"); latch opens
2Verdict readable and under budget; no release recordedPause("paused: waiting for a named release")
3Release recorded before the latch opened, or by someone not on release_byStill paused
4Release recorded after the latch opened by Dana, who is activeLatch closes; checks re-run; Allow

The same sequence with monitor.healthy() going false, then true, must give the same results. Without the latch, step 2 would allow: the dependency recovered, but no person said yes again.

Two more sequences cover the stores themselves, starting with no latch open:

StepInputsExpected result
1latches.read returns NonePause("latch store unreadable"); fault opens in the journal
2Latch store readable again and "clear"; no release recordedPause("store fault: waiting for a named release")
3Release recorded after the fault opened, by Dana, who is activeFault closes; checks re-run; Allow
1′faults.read returns Noneexecute halts the executor; Pause("gate error: executor halted")
2′Journal and latch store both readable and clear; no restart recordedPause("executor halted: waiting for a named restart")
3′Restart recorded after the halt, by Dana, who is activeHalt ends; checks re-run; Allow

Step 2 is the case the earlier sketch got wrong: it paused once while the store was unreadable, then allowed as soon as the store read "clear", with no release.

monitor.healthy() reports availability; it does not enforce a volume limit. A healthy monitor observing a breach must still change the result, which is why the budget verdict is a separate input. The out-of-band confirmation is a record looked up from the call-back channel and bound to the action's id, payee and amount, not a boolean the requester can set.

The fail-open bug to look for is any branch where an exception, timeout or None falls through to Allow (for example a try/except: pass around the approval lookup). Test each on: condition with a dry run that removes exactly that input and asserts the action does not execute; module 300.9 builds and proves that stop. This sketch does not establish that every exception path is caught.

Worked case (quiz idea): the approval was granted eight months ago by a holder who has since left. directory.is_active(holder) is false, so the predicate pauses and latches regardless of expires, and pages someone who can name a new holder. A system whose holder is inactive is not held. The holder cannot release this latch (they are inactive), so the only way back is a person with that authority naming a new holder and release_by in the hold itself, then recording a release.

Receipts and coverage

The Receipt carries references and hashes only: the action's id, the hold entry's id (HOLD-WC-03), the approval and decision record ids, the gate's answer, a hash of the hold version in force and an undo reference. It carries no names, directory keys, contact details, claim text, letter text, prompts or model output. The holder's and reviewer's identities stay in the hold and decision records, which are access-controlled work records that the references resolve to for people entitled to see them. A reference is not a licence to disclose what it points to.

This is the course's boundary, not a privacy-law ruling. Replacing dana.okafor with HOLD-WC-03 keeps the name out of the receipt; it does not make the receipt anonymous. Anyone who can open the hold can resolve the reference to Dana, and whether such a reference counts as personal data under a given law is not checked here. That question belongs to the organisation's privacy owner. If an organisation wanted holder identities inside its receipts, that would be an exception to the course rule, decided and recorded by a person, not a quiet change to the schema. This sketch also does not sign the receipt; signing and offline verification are not shown in this module.

Coverage, matched action by action

The hold also reports coverage. Count it by action identity, not by comparing two totals, and check every execution record, not just the first one for each id:

An earlier version of this sketch kept only the first execution of each id. Then Allow, run, Pause, run again on one id reported 1 / 1 covered and no violation: the second run, after the Pause, disappeared. Checking each execution against its own latest prior evaluation is what makes that second run visible.

Proposals the gate refused never ran, so they belong in neither the numerator nor the denominator. Count them separately, by answer.

"Latest" and "before" need an order the records can prove. Clock times cannot: two records can carry the same time, and then the answer would depend on which one the code happened to read first. The contract here is that the gate and the executor both write to one append-only log, and the log gives each record a seq number that only goes up. Timestamps stay for people to read; order comes from seq. Each record must also carry the action's id, issued when the action was proposed. The function checks both before counting, and says "cannot count yet" rather than guessing.

def coverage(evaluations, executions):
    evaluations, executions = list(evaluations), list(executions)
    seqs = []
    for r in evaluations + executions:             # validate before counting anything
        if getattr(r, "action_id", None) in (None, "") or getattr(r, "seq", None) is None:
            return "cannot count yet: a record has no action id or no log position"
        seqs.append(r.seq)
    if len(seqs) != len(set(seqs)):
        return "cannot count yet: two records share a log position, so their order is unknown"
    if not executions:
        return "nothing executed in this bucket"   # not 100%, and not 0%
    covered, contrary, unchecked, runs = 0, [], [], {}
    for e in sorted(executions, key=lambda r: r.seq):  # every execution, not just the first
        runs[e.action_id] = runs.get(e.action_id, 0) + 1
        prior = [g for g in evaluations if g.action_id == e.action_id and g.seq < e.seq]
        if not prior:
            unchecked.append((e.action_id, e.seq))
        elif max(prior, key=lambda g: g.seq).answer in ("Allow", "Constrain"):
            covered += 1
        else:                                      # Pause, Deny, or anything not a yes
            contrary.append((e.action_id, e.seq))
    return {"covered": covered, "executed": len(executions),
            "contrary_to_gate": contrary, "no_evaluation": unchecked,
            "ran_more_than_once": sorted(a for a, n in runs.items() if n > 1)}
Synthetic caseResult
Gate evaluates A (denied) and B (paused); executor runs C and D0 / 2 covered; no_evaluation = C and D, each with its seq. Comparing totals would have said 2 ÷ 2 = 100%.
Gate evaluates A (allowed), B and E (refused); executor runs A1 / 1 covered. B and E are refused proposals, reported apart.
Executor runs B after the gate paused it0 / 1 covered; contrary_to_gate = B, with its seq
A: Allow (seq 1), run (2), Pause (3), run again (4); read in either input order1 / 2 covered; contrary_to_gate = A at seq 4; ran_more_than_once = A
A: Allow (1), run (2), run (3), with no evaluation between2 / 2 covered; ran_more_than_once = A, for a person to find out whether it ran twice or was logged twice
Nothing executed"nothing executed in this bucket"
Executor does not log action ids, or logs None for some"cannot count yet"; records with no id are never joined as one action
A record has no seq"cannot count yet"
A: Allow and Pause at the same clock time, then executed; read in either input orderSame answer both ways, decided by seq: if the Pause has the higher seq, 0 / 1 and contrary_to_gate = A
Two records with the same seq"cannot count yet": order unknown

The tied-time row must be run twice, with the two evaluations swapped in the input, and give the same result both times; so must the run-Pause-run row, with each list reversed. The checks above catch missing ids and repeated positions. The ran_more_than_once list does not decide whether a repeated id means two real effects; it makes sure a person is asked. They cannot show that an id was issued honestly or that the log's order matches what happened; those are properties of the real log and executor, not shown here.

For Constrain, checking that the executed action stayed inside the constraint is a further check, not shown here. Coverage can only see executions the executor logs: an action run by a path that writes no log is invisible to it. An empty breach list over an uninstrumented period is not evidence of coverage.

Out-of-band confirmation

An out-of-band check moves identity verification to a channel the requester does not control. The reference failure: the Hong Kong Government's 26 June 2024 Legislative Council reply describes a case received by the Police in end-January 2024 in which deepfake technology was used to "fabricate a pre-recorded video conference"; "the informant eventually gave authorisation to transfer funds to five local bank accounts" and "suffered a loss of about HK$200 million".1 "Since the meeting was pre-recorded, there was actually no interaction between the informant and the fraudster."1 The record does not name the organisation, and neither does this course. Separately, as general advice, the Police recommend "verifying with a phone call when a relative or a friend makes a request for remittance".1 The course generalises that narrower advice into its own rule: in the hold, out_of_band: true means a confirmation record from a call-back to a contact already on file, bound to this transaction, and recorded by reference in the receipt.

Relation to Article 14 of the EU AI Act

For high-risk systems, Article 14(1) requires design such that they "can be effectively overseen by natural persons during the period in which they are in use", and Article 14(4)(e) requires that the persons assigned oversight be enabled, as appropriate and proportionate, "to intervene in the operation of the high-risk AI system or interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state."2 The hold's holder and stop fields are the course's operational form of that idea; whether Article 14 binds a given system, and from what date, is 300.7's analysis. Implementing this gate is not a claim of compliance with the Act.

2. Aggregation and budgets

Aggregation

Let each action carry a small individual risk. The risk of the system is a function of volume and concentration, not only of the per-action risk: n low-stakes acts by one principal in one window can constitute a high-stakes act (for example, a pattern of denials with no individual review). A single action can of course already be serious on its own. The course calls the volume effect aggregation.

The warning case: ProPublica reported on 25 March 2023, from "corporate documents and interviews with former Cigna officials", that "Over a period of two months last year, Cigna doctors denied over 300,000 requests for payments using this method", "spending an average of 1.2 seconds on each case".3 Asked whether PXDX lets Cigna doctors reject claims without examining them, Cigna told ProPublica that description was "incorrect"; it also said that "Reviews occur after the service has been provided to the patient and does not result in any denials of care".3 Cigna's "incorrect" is addressed to that description of the review process, not to the totals. The class-action complaint in Kisting-Leung v. Cigna Corp. (E.D. Cal., filed 24 July 2023) alleges the same totals at ¶2 and cites ProPublica for them (footnote 3).4 The two are therefore one count, reported then pleaded, not two independent measurements, and neither is a finding. The course treats them as alleged totals only. It does not derive a rate from them and does not use them as inputs below.

Budget specification

A budget is a set of constraints over a bucketed count c(p, t) of counted actions by principal p in bucket t (here, one hour). In the example the counted action is a reviewer's decision taking effect.

Both are evaluated on each bucket (the maximum), never on the mean over the window.

Detection and enforcement are different controls. Detection is retrospective: compute c from the log and list breaches. Enforcement is prospective: the gate consults a live count before each counted action (the budget.verdict call in section 1) and returns Pause for the action that would make c(p, t) exceed B<sub>p</sub>, or the system sum exceed B<sub>sys</sub>. The stop must act on the counted action itself. Withholding new arrivals from p is a different, weaker control (an arrivals budget): p could still clear an existing queue past B<sub>p</sub>. On a per-principal breach:

  1. further decisions by p do not take effect (no letter is sent on them), and the latch keeps it that way after the bucket ends, until a named release;
  2. cases already queued to p stay queued; none is auto-approved or auto-rejected;
  3. the holder is paged and decides, on the record, whether to reassign queued cases to principals under budget or let them wait;
  4. new arrivals to p stop until the holder releases the pause.

A system-wide breach applies the same to all principals. A new hour starting is not a release: the count resets, the latch does not. This sketch does not solve concurrent decisions racing the counter; a real implementation needs an atomic reserve-then-commit or equivalent, and that is not specified here.

Worked example on the synthetic log

Course-owned synthetic log (hold-prompts.md); the numbers are invented. Each cell counts decisions that took effect, recorded with no live enforcement running, so this is a detection exercise. B<sub>p</sub> = 50, B<sub>sys</sub> = 120.

HourR1R2R3System total
09:0012101133
10:001491336
11:0071121093
12:00811928
log = {"09:00": {"R1": 12, "R2": 10, "R3": 11},
       "10:00": {"R1": 14, "R2": 9,  "R3": 13},
       "11:00": {"R1": 71, "R2": 12, "R3": 10},
       "12:00": {"R1": 8,  "R2": 11, "R3": 9}}
B_P, B_SYS = 50, 120
breaches = [(t, p, n) for t, row in log.items() for p, n in row.items() if n > B_P]
sys_breaches = [(t, sum(row.values())) for t, row in log.items() if sum(row.values()) > B_SYS]
# breaches     == [("11:00", "R1", 71)]
# sys_breaches == []          # peak system hour is 93

These comprehensions detect; they do not stop anything. An empty breaches list over an empty or partial log means no breach was observed, not that the system was within budget.

Design notes: choose the bucket to match the harm (an hour for reviewer overload; a day may hide it); use fixed buckets for reporting and a sliding window for enforcement so a spike across a bucket boundary (e.g. 10:30 to 11:30) is not split (the fixed-hour exercise above does not implement this); report every count with its denominator and bucket.

3. Rubber-stamp review

Definition

A review is a rubber stamp when the recorded decision is attributed to a person but the person did not form an independent judgment: the output of the system passes through unchanged and unexamined. Formally, oversight requires that the reviewer had (i) access to the case evidence, (ii) time to use it, (iii) authority to depart without penalty, and (iv) that departures are possible and recorded. A rubber stamp lacks at least one in practice.

The Cigna complaint alleges the pattern: "Cigna's doctors instantly reject claims on medical grounds without ever opening patient files" (¶2), and Cigna "allowed its doctors to sign off on the denials in batches" (¶5).4 ProPublica quotes a former Cigna doctor: "We literally click and submit" and "It takes all of 10 seconds to do 50 at a time."3 Cigna disputes the description.3 The Lokken complaint alleges a constraint on condition (iii): "targets to keep stays at skilled nursing facilities within 1% of the days projected by the AI Model", and that "Employees who deviate from the nH Predict AI Model projections are disciplined and terminated".5 All of these are allegations.

Why it is not oversight

Article 14(4)(b) of the AI Act requires that persons assigned oversight of a high-risk system be enabled "to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)", in particular for systems "used to provide information or recommendations for decisions to be taken by natural persons"; 14(4)(d) requires that they be able "to decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output".2 A reviewer without the evidence, the time or the freedom to override cannot exercise (d), so the signature records attribution, not oversight.

Detection from logs

Given a review log with events evidence_view(case, reviewer, ts, channel) (a file open, or any other route by which the case evidence is presented) and decision(case, reviewer, ts, outcome, model_outcome), compute, each with its denominator:

MetricDefinitionRubber-stamp signal
View ratecases with ≥ 1 evidence_view before decision ÷ decided casesnear 0
Dwell timedecision.ts − first evidence_view.ts per case; report the distributionlow median, low variance
Override ratecases where outcome ≠ model_outcome ÷ decided cases0 over a large n
Batch sharedecisions sharing one submit event ÷ decided caseshigh
Loaddecisions per reviewer per bucket (section 2)over budget

None is proof alone, and together they are evidence for a person to read, not a verdict. A correct model can have a low override rate; there is no target override rate, and a quota would itself corrupt review. A view event shows the evidence was presented, not that it was understood. If no evidence_view is logged, the view rate and dwell time are "cannot count yet", and the hold must say so rather than infer review or its absence. A fallback such as time since the reviewer's previous decision is an inter-decision interval, a proxy; report it under that name and never as measured reading time.

4. Claim types: allegation, statement, gate

The record

Claim typeSourceContent
AllegationEstate of Gene B. Lokken v. UnitedHealth Group, Inc., No. 0:23-cv-03514 (D. Minn.), complaint filed 14 Nov 20235¶1: "illegal deployment of artificial intelligence (AI) in place of real medical professionals to wrongfully deny elderly patients care owed to them under Medicare Advantage Plans by overriding their treating physicians' determinations as to medically necessary care based on an AI model that Defendants know has a 90% error rate." ¶6: coverage determinations "not based on individual patient's needs, but based on the outputs of the nH Predict AI Model".
Company statementUnitedHealth spokesperson, emailed to STAT, 14 Nov 20236The naviHealth predict tool is not used to make coverage determinations; "The tool is used as a guide to help us inform providers, families and other caregivers"; "Coverage decisions are based on CMS coverage criteria and the terms of the member's plan"; "lawsuit has no merit and we will defend ourselves vigorously".
Company assertion, reported by STATSame STAT article, separate from the emailed statement6"UnitedHealth has asserted in response to STAT’s reporting that its physician medical reviewers provide a check" "by making a final determination about whether a patient is issued a payment denial". STAT adds, in its own voice: "But those physicians are reviewing denial recommendations sent to them by clinical case managers" "who are subject to the company’s 1% performance target".
GateThe course (Practice 300 Hold, hold-prompts.md)A model may inform but not replace clinical judgment in a coverage decision.

The 90% figure is the complaint's allegation; its basis is not checked here. The outcome of the case is not checked and is not stated.

Treat claim type as data. In notes and incident reports (work records, not receipts), every case fact carries its type and source, and nothing typed allegation or statement is ever restated as finding.

Invariance of the gate

Let H<sub>C</sub> be "the model's output replaced clinical judgment" (the complaint) and H<sub>U</sub> be "the tool is a guide; coverage decisions rest on CMS coverage criteria and plan terms" (the company's statement). The gate G requires: for each case where the model's predicted stay m and the treating clinician's determination d differ, a qualified person reviews that case and decides, and the record names who and why. G does not prescribe that d prevails; it prescribes who decides.

G's requirement is the same under both, so the builder need not resolve the dispute to implement it.

Against "accurate on average": a model's aggregate error rate is a population statistic. It can inform an individual case but does not decide it, because the gate is evaluated per case. This is section 2 inverted: aggregation hides individual harms just as averages hide peaks.

Big Widget implementation

- id: HOLD-WC-05
  action: approve_repair_hours
  authority: approver
  holder: dana.okafor            # in the controlled hold; receipts carry HOLD-WC-05, not the name
  release_by: [dana.okafor]
  expires: 2027-05-31
  decision: per_case
  judgment: {informs: service_planner_estimate, decides: examining_engineer}
  on_disagreement: engineer_figure_stands; reason_code_required   # Big Widget's own repair rule
  stop: {on: [no_engineer_decision_on_record], action: hold_job_open}
  report: [override_rate_with_denominator, view_rate_or_cannot_count_yet, coverage_or_cannot_count_yet]

An override rate of 0 out of 2,000 jobs is a signal to investigate (section 3), not a success metric and not, on its own, proof of a rubber stamp.

Further learning (free university courses)

Optional background, not sources: University of Helsinki, Ethics of AI (https://ethics-of-ai.mooc.fi/), on accountability for automated decisions; MIT OpenCourseWare, 6.S897 Machine Learning for Healthcare (Spring 2019) (https://ocw.mit.edu/courses/6-s897-machine-learning-for-healthcare-spring-2019/), on clinical models.

Sources

  1. s5 Government of the Hong Kong Special Administrative Region (written reply by the Secretary for Security to a Legislative Council question) (2024). LCQ9: Combating frauds involving deepfake. Record. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, fresh session separate from the writer, reviser and earlier checkers), round 3 re-check, 19 Nov 2026; live page, info.gov.hk, full reply. ↩ ↩ ↩
  2. s6 European Parliament and Council of the European Union (text as shown by the European Commission's AI Act Service Desk) (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 14 Human oversight. Standard. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, fresh session separate from the writer, reviser and earlier checkers), round 3 re-check, 19 Nov 2026; AI Act Service Desk Article 14 page (consolidated text as at 27 Jul 2026) and the Official Journal text (OJ L, 12.7.2024) via Publications Office Cellar DOC_1; every quoted passage matches in both. ↩ ↩
  3. s2 Patrick Rucker (The Capitol Forum), Maya Miller and David Armstrong (ProPublica) (2023). How Cigna Saves Millions by Having Its Doctors Reject Claims Without Reading Them. Record. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, fresh session separate from the writer, reviser and earlier checkers), round 3 re-check, 19 Nov 2026; live page, whole page saved. ↩ ↩ ↩ ↩
  4. s1 Plaintiffs Suzanne Kisting-Leung and Ayesha Smiley, by Clarkson Law Firm, P.C. (pleading filed in the United States District Court for the Eastern District of California, via CourtListener) (2023). Kisting-Leung v. Cigna Corp., No. 2:23-cv-01477 (E.D. Cal.), class action complaint (docket entry 1). Record. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, fresh session separate from the writer, reviser and earlier checkers), round 3 re-check, 19 Nov 2026; RECAP PDF Doc. 1, full text. ↩ ↩
  5. s3 Plaintiffs the Estate of Gene B. Lokken and the Estate of Dale Henry Tetzloff (pleading filed in the United States District Court for the District of Minnesota, via CourtListener) (2023). Estate of Gene B. Lokken v. UnitedHealth Group, Inc., No. 0:23-cv-03514 (D. Minn.), class action complaint (docket entry 1). Record. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, fresh session separate from the writer, reviser and earlier checkers), round 3 re-check, 19 Nov 2026; RECAP PDF Doc. 1, full text. ↩ ↩
  6. s4 Casey Ross and Bob Herman (STAT) (2023). UnitedHealth faces class action lawsuit over algorithmic care denials in Medicare Advantage plans. Record. Checked 2026-11-20 by source-checker agent (Claude Opus 5.5, fresh session separate from the writer, reviser and earlier checkers), round 4 re-check, 20 Nov 2026; live page (HTTP 200), full article body, all nine uses including the revised plain.md wording. ↩ ↩ ↩

Next: the quiz