
Research and governance roles (role tracks) · Technical gear
Reviewed and published, version 1.0.0. Big Widget Company and packet Cards A–G are fictional, course-owned teaching examples, not real records or conformance tests. The handoff tuple, the role tracks, the notation and the prep-rail mapping are this course's own design, not a standard, a conformance test or an exam syllabus.
Scope
Figure data as text
Keep the evidence, not the secrets
| Part | Wording | Sources |
|---|---|---|
| A receipt carries | hashes, decision, tier, policy hash, key id | [s7] |
| A receipt never carries | prompts, outputs, payloads, personal data | [s7] |
The receipt's allowed metadata versus the private content it never carries; packet details belong in a separately controlled work record, and the handoff lists them by hash (s7).
This gear treats an expert's output as an object that must survive transfer to an independent party. It defines an assurance handoff formally, states the conditions under which it is checkable (500.R-a), and types evidence relative to the claim it is offered for, so that preparation evidence about a person cannot be used as control evidence about a system (500.R-b).
| Objective | Formal object | Primary evidence used |
|---|---|---|
| 500.R-a | Handoff tuple $H$; bounded confidence statement $\kappa$; independence and authority conditions on the next owner | NIST AI RMF 1.0, MEASURE 1.3, 2.1, GOVERN 2.1–2.3, Fig. 3 note1; Raji et al., SMACTR and §52; Pineau et al., NeurIPS 2019 program3; Playbook, for versioning and voluntary use4 |
| 500.R-b | Mapping relation $M$ from handoff parts to rail practices and objectives; claim-relative evidence typing $\sigma(e, c)$ | Rail structure: this course's Level 500 map (levels/500/objectives.yml), not an external source. Person/system split: NIST AI RMF 1.0, GOVERN 2.2 versus MEASURE 1.3/2.1 and §3 characteristics1 |
Misconceptions targeted: that expert work is complete when an expert names a question or states confidence (500.R-a), and that completing a prep rail certifies a person or a system (500.R-b).
1. Why transfer, and why to an independent party
Three primary sources motivate the object.
- Independent review as a measurement property. NIST AI RMF 1.0 §5.3 states that processes for independent review "can improve the effectiveness of testing and can mitigate internal biases and potential conflicts of interest" (p. 28).1 MEASURE 1.3 asks that internal experts who were not front-line developers, and/or independent assessors, take part in regular assessments (p. 29).1 The Fig. 3 note records the best practice of separating those building and using models from those verifying and validating them (p. 11).1
- Audit as an artifact chain. Raji et al.'s SMACTR framework (Scoping, Mapping, Artifact Collection, Testing, Reflection) makes each stage yield documents that together form the audit report (pp. 5, 10).2 The Reflection stage's algorithmic design history file should support an audit trail "enabling the reconstruction of key decisions and events", and the summary report aggregates the artifacts "in one accessible location for review" (p. 10).2 Section 5 states that internal auditors "necessarily share an organizational interest with the target of the audit" and that maintaining independence is challenging (p. 10).2 The paper positions internal audits as complementing external accountability by generating artifacts third parties can use (p. 3).2
- Reproduction as secondary analysis. Pineau et al. define reproducibility as obtaining similar results using the same code and data when available, "a necessary step" to verify reliability (p. 1).3 The NeurIPS 2019 program had three components: a code submission policy, a community-wide reproducibility challenge and the ML reproducibility checklist (pp. 1–2).3 The challenge aimed to provide independent verification of empirical claims and leave a public trace; 173 papers were claimed (p. 9).3 The authors report that reproducibility outcomes rarely reduce to a concise yes/no, and they had not found that most studies' findings lend themselves to a coarse score (p. 10).3
The shared requirement: the verifier is not the producer, the record supports reconstruction, and the result is reported with its structure intact.
Lifecycle placement (course application). The Foundation's three movements are Get to Yes, Stay at Yes and Recover to Yes, and its gate answers "go ahead · only this far · wait for a person · no".5 In those terms, $H$ is an input to a gate, never the gate's answer: $\kappa$ and $N$ bound what a Get to Yes decision may rely on; $F$ and the pinned $v$ keep Stay at Yes honest across versions; and Recover to Yes needs a fix, a retest by $x \notin A(W)$ and a person's ruling, not an announcement.
2. The handoff as a tuple
Let $W$ be a piece of expert work produced by a set of people $A(W)$. A handoff is
H = (P,\ \Gamma,\ \bar\Gamma,\ \kappa,\ F,\ K,\ o)
where
- $P$ is the evidence packet, pinned by a manifest $\mu(P) = \{(\text{path}_i, h(\text{bytes}_i))\}$ of file hashes or immutable revision ids (for this lesson's cards: module, version and card letter);
- $\Gamma$ is the set of checks performed and $\bar\Gamma$ the declared set of checks not performed, including every input in $P$ that a reader cannot re-execute;
- $\kappa$ is a bounded confidence statement (§3);
- $F \neq \varnothing$ is a set of findings with status $\text{open}$;
- $K$ is a set of check instructions: for each key check, pinned inputs $\subseteq \mu(P)$, an environment specification, a procedure and an expected output;
- $o$ is a named next owner with a decision $d$ and a deadline.
Checkability conditions. $H$ is checkable iff:
- Pinning: every input referenced by $\kappa$ and $K$ resolves in $\mu(P)$.
- Declared coverage: $\Gamma \cup \bar\Gamma$ covers the claim's scope at the stated granularity, so that silence is never read as coverage.
- Rerunnable: each $k \in K$ can be executed by a party $x \notin A(W)$ with only the listed inputs and environment. What cannot be executed that way belongs in $\bar\Gamma$, not $K$.
- Open findings persist: no $f \in F$ changes status except by a ruling of $o$ (or a later owner $o$ designates under the same independence rule).
- Independence (course rule): $o \notin A(W)$, $o$ is not the author of $H$, and $o$ is not a direct collaborator on $W$. This is the course's design, borrowing the separation of building from verifying (p. 11).1 It removes the producer's own interest in $W$. It does not remove the shared organisational interest that Raji et al.'s §5 limitation names (p. 10).2 Where that shared interest bears on $d$, $H$ should say so and, if possible, choose an external $o$; otherwise record the conflict as an element of $F$.
- Authority: $o$ can actually decide $d$. Raji et al. stress identifying decision makers who can drive responses to audit outcomes (p. 3); the AI RMF asks that executive leadership take responsibility for AI risk decisions (GOVERN 2.3, p. 23) and that responsibilities be assigned to supersede, disengage or deactivate systems inconsistent with intended use (MANAGE 2.4, p. 32).21 $o$ is a person; an agent may draft $H$ but is never $o$ and never signs (TRUTH-STANDARD.md, rule 5).
Remediation versus ruling. Assigning a fix for $f$ to a member of $A(W)$ is compatible with condition 5. The ruling that the fix closes $f$ is not: it belongs to $o$.
GOVERN 2.1 is the organisational precondition: roles, responsibilities and lines of communication for AI risk work are documented and clear to individuals and teams throughout the organisation (p. 23).1 The named-person owner field is this course's rule, not a NIST requirement: a handoff whose owner field is a team name fails condition 6. Reading that as falling short of GOVERN 2.1's aim at organisation scale is the course's inference, not a statement in the framework.
Privacy of $P$. If $H$ references a signed receipt, that receipt holds no private content.5 The Foundation's receipt carries "hashes, decision, tier, policy hash, key id" and never "prompts, outputs, payloads, personal data".6 Transcripts and personal data in $P$ stay in a separately controlled work record and enter $\mu(P)$ only as hashes. A hash is not a licence to disclose its preimage.
3. Bounded confidence statements
Define
\kappa = (c,\ v,\ S,\ E,\ N,\ (k^\*, r^\*))
with claim $c$, artifact version $v$, scope $S$, supporting evidence $E$, explicit non-implications $N$ (claims a reader might infer but that $E$ does not support), and a falsifier: a check $k^\$ whose outcome $r^\$ would cause $c$ itself to be withdrawn or narrowed.
Falsifier target. $k^\*$ must test $c$ under $S$. A check whose outcome bears only on some $f \in F$ (an explanation of a discrepancy, say) is a proposed resolution path for $f$, recorded with $f$, not a falsifier for $c$.
Relevance. An item $e$ belongs in $E$ only if its outcome bears on $c$, informally $\Pr(c \mid e) \neq \Pr(c \mid \neg e)$, equivalently a likelihood ratio different from 1. Evidence that would leave your credence unchanged whichever way it came out is decoration.
Non-transfer. Strong evidence for $c$ does not resolve a distinct finding $f \in F$. Formally, $E$ may change $\Pr(c)$ while $f$'s status remains $\text{open}$; the handoff keeps both. "I don't know yet" is a legitimate value for $f$; the Foundation's line is that it "always pairs you with someone",6 which here is $o$.
Uncertainty. The AI RMF says measurement processes should include methods "with associated measures of uncertainty" and "formalized reporting and documentation of results" (p. 28); MEASURE 2.1 asks that test sets, metrics and tool details used in TEVV be documented (p. 29).1 For quantitative claims, report the estimate with its interval, the sample it was computed on, and what variation the interval describes; for qualitative claims, state the scope boundary in words.
Worked instance (Card A, fictional). The paper reports 91.0% on its released test set, ±1.0 across its own seeds. Dana logged three seeds of the released code at one pinned commit on the released set (90.6, 91.2, 90.9; mean 90.9) and on a frozen 200-image Big Widget sample (83.5, 84.0, 83.0; mean 83.5). Then:
- $c$ = "the released code reproduces the reported metric on the released data"; $v$ = pinned commit; $S$ = released test set, seeds 1–3; $E$ = runs 1–3;
- $N$ = {"meets the reported metric on Big Widget images"} (runs 4–6 bear on $N$, not $E$);
- $k^\$ = a rerun by $x \notin A(W)$ at $v$ on the released set with seeds 1–3; $r^\$ = mean outside $[90.0, 92.0]$, under which $c$ is withdrawn;
- $F$ = {unexplained 7.4-point drop on the Big Widget sample}.
What the intervals measure. The paper's ±1.0 is seed-to-seed spread on the released set; it calibrates $k^\*$. It is not an interval on a between-set difference, and no sampling interval for the 200-image estimate is given, so the precision of 83.5 is in $\bar\Gamma$. The drop exceeds the reported seed spread by a wide margin, which makes seed variation of the kind the paper measured an unlikely explanation on its own, and says nothing more. No cause is attributed in $H$.
Resolution path for $f$ (recorded with $f$, not in $\kappa$). A controlled relabelling check: a second labeller relabels the same 200 images while code, seeds and images are held fixed. If the gap closed, label error would be supported as a contributor on this sample; it would not establish labels as the sole cause beyond it. A camera-controlled comparison is an alternative path. $o$ = Priya, a data-quality lead not in $A(W)$ with authority to commission checks, decides which path runs first; the purchasing decision holds until she rules.
$K$ for this lesson. Recompute both means from Card A; expected 90.9 and 83.5. A full rerun needs the code, environment and images, which are not supplied: those are in $\bar\Gamma$.
4. Role tracks as instances of $H$
Each track instantiates $H$ on one packet card from the standard gear (Cards A–G). As of 19 November 2026 the runnable synthetic packets and verifier for 500.6 and 500.7 are not in this course's repository, so no track depends on them; a learner may substitute an earlier module's synthetic packet once it is published and checked. On every card, results the reader cannot re-execute (Card B's signature verification, Card D's prompts against the version-4 build) go in $\bar\Gamma$.
| Track | Card | Artifact | Typical $c$ | Dominant open $f$ | Typical $o$ |
|---|---|---|---|---|---|
| Researcher | A | Replication discrepancy | Reproduces on released data at $v$ | Cause of divergence on new data | Data-quality or methods lead outside $A(W)$ |
| Auditor or assessor | B | Missing-evidence register | Refunds 1–2 carry both required records; 3–4 do not | Control execution on unregistered cases; effectiveness | External assessor or control owner's manager |
| Regulator or policy staff | C | Scope-exception memo | Duty's application turns on one missing role fact | The role fact; any legal conclusion | Counsel or named decision-maker |
| Red-team lead | D | Ruling and retest handoff | F-1 accepted; listed prompts logged as refused on version 4 | Coverage beyond the list; retest by non-tester | Retester other than the tester |
| Standards editor | E | Versioned ambiguity / test-vector note | Clause 5 admits two readings at version 0.3; V1 separates them | Adopted reading | Editors' group, named chair |
| Governance lead | F | Escalation record | Gate answered hold for the use; required items 1–2 absent | Go ahead, only this far, wait, or no | Accountable executive |
| Reviewer | G | Evidence/revision disposition | Claim 1 supported by Card A; claims 2–3 not | Author's revision | Author for the fix; second reviewer rules on the re-read |
Track notes, with primary support:
- Reviewer. In the NeurIPS 2019 data, 34% of reviewers asked whether checklist answers were useful for evaluation said yes (p. 12), and the authors state they lacked concluding evidence that the program's processes affected the quality of the work (p. 12).3 A reviewer's $\kappa$ should be claim-level, not paper-level.
- Policy staff. MAP 1.1 asks that intended purposes, context-specific laws and prospective deployment settings be understood and documented (p. 26); GOVERN 6.1 covers risks from third-party entities (p. 24).1 Appendix A defines third-party entities as external to the acquiring organisation's design, development or deployment team and notes their risk tolerances may not align with the deploying organisation's (p. 37).1 A scope-exception memo sets $N$ to include any legal conclusion.
- Governance lead. The record states which of the four gate answers was given and who can change it.5 A hold is not a standing permission. The Foundation's checklist asks for the human who releases a hold and how long a hold lives before it "decays to no";5 the record carries both. By this course's rule the releaser is $o$, never the drafting agent or a process name. MANAGE 4.1 asks for post-deployment monitoring plans including appeal and override, incident response, recovery and change management (p. 33); MEASURE 3.3 asks for feedback processes for end users and impacted communities to report problems and appeal outcomes (p. 31); MANAGE 1.4 asks that negative residual risks to downstream acquirers and end users be documented (p. 32).1
- Standards editor. Guidance documents are versioned in practice: the Playbook landing page shows "Created July 8, 2022, Updated June 10, 2026" and a first complete version announced 30 March 2023.4 Pin $v$ in every ambiguity note.
- Red-team lead. This course's process (TRUTH-STANDARD.md rule 3) requires a person other than the tester to rule on findings; condition 5 generalises that rule to the next owner of any handoff. A retest log is a record of one run. For $c$ = "the listed prompts were logged as refused in Sam's run on version 4", a later run with a different outcome is not $r^\*$: it bears on repeatability or coverage, a distinct claim. Valid falsifiers are either a review of the original log (completeness, genuineness, prompt-to-row match), or, for $c'$ = "version 4 refuses the listed prompts repeatably in a pinned setup", a rerun against a repeatability criterion fixed before the rerun.
Against checklist use. The AI RMF states its actions "do not constitute a checklist" and are not necessarily ordered (p. 20), and that it does not prescribe risk tolerance (p. 7).1 The Playbook is for voluntary use; organisations may borrow "as many – or as few – suggestions as apply".4 Meeting the checkability conditions is necessary for a usable handoff, not sufficient for good underlying work.
5. Failure modes as violated conditions
| Failure | Violated condition |
|---|---|
| "High confidence" with no evidence, scope or falsifier | $\kappa$ incomplete: $E$, $S$ or $(k^\, r^\)$ empty |
| Falsifier tests an explanation of $f$, not $c$ | Falsifier target (§3) |
| Author marks own finding resolved | Condition 4 |
| Next owner is the author, a collaborator or a team | Conditions 5 and 6 |
| Check needs the author's machine or private access | Conditions 1 and 3 |
| Lists only what was checked | Condition 2 |
6. Mapping to the unnamed prep rail
The course's Level 500 rail (500.rail-ai-assurance, defined in levels/500/objectives.yml) is a practice route toward an AI governance or management-system audit certification. Its structure is the course map's, not any cited source's. Under Bob's decision of 18 November 2026, its vendor and exam remain unnamed until he clears each name. No current exam alignment is claimed; the course supplies no vendor-proctored exam and no endorsement.
The rail draws on $O_R = \{$500.6-a, 500.7-b, 500.8-b, 500.9-d, 500.R-a, 500.R-b$\}$ and three practices: $\Pi = \{\pi_1$ produce a role-specific handoff, $\pi_2$ inspect a missing-evidence packet, $\pi_3$ route an unresolved finding to an independent owner$\}$. A learner's mapping is a relation
M \subseteq \text{Parts}(H) \times \Pi \times O_R
that must be total on $\Pi$ (every practice evidenced by some part) and must cite only objectives in $O_R$.
| Part of $H$ | $\pi$ | Objective |
|---|---|---|
| $\mu(P)$, $\kappa$ | $\pi_1$ | 500.R-a |
| $\Gamma$, $\bar\Gamma$, missing-proof list | $\pi_2$ | 500.6-a; 500.7-b |
| Clause → gate → evidence → owner rows | $\pi_2$ | 500.8-b |
| Tested/untested coverage; ruling by non-tester | $\pi_3$ | 500.9-d |
| $F$, $o$ | $\pi_3$ | 500.R-a |
| $M$ itself, with $\sigma$ respected (§7) | $\pi_1, \pi_2, \pi_3$ | 500.R-b |
7. Typing evidence relative to a claim
Subject is a property of an evidence item as used for a claim, not of the item alone. Define $\sigma(e, c) \in \{\text{person}, \text{system}\}$ by the subject of $c$. One item can carry both types: Morgan's ruling on F-1 (Card D) is person evidence for "Morgan can rule on findings" and system evidence for "F-1 was found and accepted on version 4".
- Claims about a person draw on completed handoffs, mapping tables, reviewer feedback on practice and a later exam result.
- Claims about a system's controls draw on test sets, metrics and results; signed records; ruled red-team findings and retests; monitoring, incident and appeal records.
Typing rule. A claim about a system's controls $c_{\text{sys}}$ may cite only $e$ with $\sigma(e, c_{\text{sys}}) = \text{system}$. Person evidence can raise confidence that an assessor will perform a check well; it cannot stand in for the check's result. An exam result is never typed $\text{system}$ for any $c_{\text{sys}}$.
Scope of system evidence. Typing is necessary, not sufficient. Each system item supports only a narrow inference, under stated assumptions:
| $e$ | Supports, under assumptions | Does not support |
|---|---|---|
| Signed record that verifies | Integrity and origin: bytes unchanged since signing by a holder of the key, given an uncompromised key and a correct verifier | That the described event occurred; control effectiveness |
| Retest log on $v$ | Listed inputs logged as passing on $v$, given a complete and genuine log | Unlisted inputs; other versions; independence of the retester |
| Ruled red-team finding | A weakness existed and a person ruled on it | Absence of other weaknesses; overall readiness |
| Appeal and complaint records | An appeal route existed and was used | Quality of appeal outcomes; reachability for all users |
| Missing-evidence register | Which required records are present or absent | Control execution on unregistered cases |
The AI RMF's structure respects the person/system split. GOVERN 2.2 concerns personnel and partners receiving AI risk management training to perform their duties (p. 23); MEASURE 1.3 and MEASURE 2.1 concern who assesses the system and how its testing is documented (p. 29).1 The trustworthiness characteristics (valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; fair with harmful bias managed) are characteristics of AI systems (p. 12).1
Consequences:
- an exam result has $\sigma = \text{person}$ for every claim and supports no $c_{\text{sys}}$;
- an assessor's preparation does not change a system's control evidence;
- no single system item supports "the system is ready"; each states its inference and what is missing;
- completing the rail produces only person-typed evidence and grants no certification of the learner or any system.
8. Exercises
- Write $\kappa$ for the reviewer disposition on Card G. Identify one item a careless reviewer would put in $E$ that fails the relevance test, and one check that is a resolution path for $f$ rather than a falsifier for $c$.
- On Card D, the red-team handoff names the original tester as $o$ for the retest. State which conditions fail and rewrite $o$.
- A mapping table cites a passed practice quiz as evidence that Big Widget's hiring assistant has working human review. Show where the typing rule is broken.
Further learning (not a citation)
Harvard CS 2881R (AI Safety, Fall 2025), Stanford CS 181/181W (Computers, Ethics, and Public Policy, Spring 2024) and Stanford CS25 (Transformers United V6) go deeper. They support no claim here and none of their material is copied. Links and caveats are in levels/500/universities.md.