
Research and governance roles (role tracks) · Standard gear
Reviewed and published, version 1.0.0. Big Widget Company, its people and the seven packet cards are made up. The cards are course-owned teaching examples, not real records and not conformance tests. The six-part handoff, the seven role tracks and the prep-rail mapping are this course's own practice, not a law, a standard or an exam syllabus. Where we do not know something, we say so.
What this module does
Figure data as text
Keep the evidence, not the secrets
| Part | Wording | Sources |
|---|---|---|
| A receipt carries | hashes, decision, tier, policy hash, key id | [s7] |
| A receipt never carries | prompts, outputs, payloads, personal data | [s7] |
The receipt's allowed metadata versus the private content it never carries; packet details belong in a separately controlled work record, and the handoff lists them by hash (s7).
Modules 500.1 to 500.9 each ended with a piece of expert work: a critique of a paper, an evaluation reading, a receipt design, a governance table, a red-team finding. This module asks what happens next. Expert work helps an organisation only when someone else can pick it up, check it and act on it. The record that makes that possible is a handoff.
You take one of seven role tracks: researcher, auditor or assessor, regulator or policy staff, red-team lead, standards editor, governance lead, or reviewer. You work from a small made-up packet card in this lesson (Cards A to G, below). You do not start a new literature review or a new regime design.
By the end you will be able to:
- produce a role-specific assurance handoff from a supplied evidence packet, with a bounded confidence statement, an unresolved finding, check instructions and a named independent next owner (500.R-a);
- map your handoff practice to this course's unnamed assurance prep rail and its objectives, and tell evidence about a person's preparation apart from evidence about a system's controls (500.R-b).
Two beliefs this module corrects:
- "Expert work is done when an expert names the question or says they are confident." A confidence statement that nobody else can check is not yet assurance. The NIST AI Risk Management Framework asks that roles and responsibilities for AI risk work be documented and clear to individuals and teams throughout the organisation, and that test sets, metrics and tool details used in testing be documented.1
- "Finishing a prep rail certifies me, or certifies the system." It does neither. This course does not issue a certification, and preparation evidence is about a person, not about a system's controls.
Why a handoff, and why to someone else
Three of this module's sources, from different fields, point the same way.
Independent checking is a design choice, not an afterthought. In the NIST AI RMF, the Measure function says processes for independent review "can improve the effectiveness of testing and can mitigate internal biases and potential conflicts of interest".1 Subcategory MEASURE 1.3 asks that internal experts who did not serve as front-line developers for the system, and/or independent assessors, are involved in regular assessments and updates.1 The framework's figure of AI actors notes that, as a best practice, those building and using models are separated from those verifying and validating them.1
Auditors who share the organisation's interest need a record others can scrutinise. Raji and colleagues propose SMACTR, an internal audit framework whose stages each produce documents.2 Their Reflection stage ends in a remediation and risk mitigation plan, an algorithmic design history file that should assist an audit trail by "enabling the reconstruction of key decisions and events", and an audit summary report put "in one accessible location for review".2 They also say that because audit results can be ambiguous, "it is critical to identify key stakeholders and decision makers who can drive appropriate responses to audit outcomes".2 And they name the limit plainly: internal auditors "necessarily share an organizational interest with the target of the audit", and keeping an independent viewpoint is challenging.2 They describe internal audits as complementing external accountability, producing artifacts third parties can use.2
Research claims get stronger when someone else reruns them, and the answer is rarely one word. Pineau and colleagues define reproducibility as obtaining similar results to those presented in a paper or talk, using the same code and data when available, and call it "a necessary step" to verify the reliability of findings.3 The NeurIPS 2019 reproducibility challenge aimed to provide independent verification of the empirical claims in accepted papers and to leave a public trace of the findings.3 Participants claimed 173 papers for reproduction.3 The authors report that a reproducibility study rarely gives a concise answer to "Is this paper reproducible?", and that they had not found that the findings of most reproducibility studies lend themselves to a coarse "reproducibility score".3
Put together: a check is stronger when the person checking did not do the work, when the work leaves a record that supports the reconstruction of what was decided, and when the result keeps its nuance instead of being squeezed into a verdict. The handoff is the course's name for that record.
Where the handoff sits in the three yeses. The Foundation names three movements: Get to Yes, Stay at Yes, Recover to Yes.4 Applied to a handoff (the course's application, not a quotation):
- Get to Yes. Before a decision leans on expert work, the handoff says what the evidence supports, what it does not, and which person decides. A handoff is not itself a yes.
- Stay at Yes. The open finding stays open with a named owner, and the handoff pins a version, so a new version is checked again rather than inheriting the old result.
- Recover to Yes. When a finding is accepted, the fix, a retest by someone other than the tester, and a person's ruling are the evidence of recovery. An announced fix is not a proved recovery.
The six parts of an assurance handoff
This is the course's own template. Every track uses it.
| Part | What it says | Fails if |
|---|---|---|
| 1. Packet and version | Exactly which evidence you received, pinned by a version id or file-hash list | Someone else cannot tell which version you looked at |
| 2. What you checked | The steps you actually took, and the steps you did not | It lists only what was checked, so coverage looks complete |
| 3. Bounded confidence statement | A claim, its scope, the evidence that moved your confidence, what it does not show, and what result would change your mind | It states confidence with no evidence, no scope or no falsifier |
| 4. Unresolved finding | At least one open question or finding, described so it can be ruled on, and marked open | It is quietly closed by the author, or vanishes from the handoff |
| 5. Check instructions | Pinned inputs, steps and the expected result, so a stranger can rerun your key check | The check needs the author present, or private access the next owner lacks |
| 6. Named independent next owner | A named person who did not produce the work being checked and who has the authority to decide, with what they must decide and by when | The owner is the author, a team name, "the governance process", or someone with no power to act |
A few rules make the template hold.
The confidence statement is bounded on four sides. A useful form is: "On version V, within scope S, evidence E supports claim C. It does not show N. If check K returned R, I would withdraw or narrow C." Each blank is checkable. "We are confident the model is fair" is not, because it has no scope, no evidence and no falsifier. The NIST AI RMF says measurement processes should include testing and assessment methods "with associated measures of uncertainty" and "formalized reporting and documentation of results".1
The falsifier tests the claim you made, not a different one. K must be a check whose result bears on C itself. A check that would help explain an open finding belongs with that finding, not in the confidence statement.
Evidence must change confidence, and open questions must stay open. If a piece of evidence would not change your view whichever way it came out, it does not belong in the confidence statement. If a question is still open, write it as open. Do not let a strong piece of evidence on one question close a different question. "I don't know yet" is a fair answer in a handoff. The Foundation pairs it with a person: it "always pairs you with someone".5 Here, that someone is the named next owner.
Independence is this course's rule, with limits you should name. In this course, "independent" means the next owner did not build, test or write the thing being handed over. That rule borrows NIST's best practice of separating building from verifying.1 It is the course's design, not a requirement any source states for handoffs. It removes the author's own interest in the result. It does not remove the interest a colleague shares with the whole organisation: Raji and colleagues' limit on internal auditors still applies.2 They warn against audits "becoming simply acts of reputation management for an organization".2 The course's reasoning, not theirs, is that letting the author quietly close their own finding is one easy route to that. When the shared interest could sway the ruling (a purchase the organisation wants, a launch date it has announced), say so in the handoff, and name an outside assessor if one is available. If none is, record the conflict as unresolved.
A fix and a ruling are different jobs. The author may be the right person to make a fix. The ruling on whether the fix closes the finding goes to someone else.
The owner is a person who can act. Raji and colleagues ask for identified decision makers who can drive a response.2 The NIST AI RMF asks that executive leadership take responsibility for decisions about AI risks, and that responsibilities be assigned and understood for superseding, disengaging or deactivating systems that perform inconsistently with intended use.1 An agent may draft a handoff. Only a person rules on a finding, owns the next decision or signs: this course's own rule is that agents never count as reviewers and only people sign (TRUTH-STANDARD.md, rule 5).
Keep the evidence, not the secrets. If your handoff points to a signed receipt, the receipt holds no private content.4 The Foundation's receipt carries "hashes, decision, tier, policy hash, key id". It never carries "prompts, outputs, payloads, personal data".5 Prompts, transcripts and customer details in a packet stay in a separately controlled work record, if your workplace allows you to keep them. The handoff lists them by hash. A hash does not give you permission to share what it points to.
The packet cards (made up)
The earlier Level 500 modules plan their own course-owned synthetic packets. As of 19 November 2026, the runnable packets and verifier for 500.6 and 500.7 are not yet in this course's repository, so this lesson does not depend on them. Each track below has a small packet card instead. Pin your packet as "500.R, version 0.1.0, Card X". Every key check on a card can be done on paper, by anyone, with only the card. Where a card gives a result that a reader cannot rerun (a signature check, a chatbot retest), the card says so, and your handoff must list it under "not checked".
Card A · Researcher. Big Widget's research group must say whether a published defect-detection result can support a purchase. The paper reports 91.0% accuracy on its released test set, with a spread of ±1.0 points across its own training seeds. Dana, a research scientist, logged six runs of the released code at one pinned commit:
| Run | Data | Seed | Accuracy |
|---|---|---|---|
| 1 | Released test set | 1 | 90.6% |
| 2 | Released test set | 2 | 91.2% |
| 3 | Released test set | 3 | 90.9% |
| 4 | Big Widget frozen sample (200 images) | 1 | 83.5% |
| 5 | Big Widget frozen sample (200 images) | 2 | 84.0% |
| 6 | Big Widget frozen sample (200 images) | 3 | 83.0% |
Key check: average runs 1–3 and runs 4–6. Expected: 90.9% and 83.5%. Cannot be rerun here: the code and images themselves are not supplied.
Card B · Auditor or assessor. Control claim: "Every refund above $500 waits for a person's approval." Each such refund needs a signed decision record and an approval record naming the approver. The pilot-week register lists four refunds above $500:
| Refund | Signed decision record | Verifier result (as given) | Approval record |
|---|---|---|---|
| 1 | Present | Valid under key K1 | Present |
| 2 | Present | Valid under key K1 | Present |
| 3 | Present | Valid under key K1 | Absent |
| 4 | Absent | None | Present |
Key check: count the refunds with both records. Expected: 2 of 4. Cannot be rerun here: the signature verification (the card gives only its result), and whether other refunds above $500 happened but were never registered.
Card C · Regulator or policy staff. Made-up "Widgetland AI Rule 4": "A provider of a hiring AI system must keep a test log. A deployer that substantially modifies a hiring AI system is treated as its provider." Facts: Big Widget bought a hiring assistant from a vendor and uses it. Whether Big Widget changed the assistant's scoring settings: not known. Key check: apply the rule's two sentences to the facts. Expected: whether Rule 4's test-log duty falls on Big Widget turns on the missing fact.
Card D · Red-team lead. Finding F-1: Sam, a tester, got version 4 of Big Widget's support chatbot to reveal another customer's order status. Morgan, the red-team lead, ruled F-1 accepted. A fix was made. Sam retested five listed prompts on version 4, and the log records a refusal for all five. Key check: match the five listed prompts to the five log rows. Expected: five of five recorded as refusals. Cannot be rerun here: the prompts against the version-4 build. Note: the retest was run by the original tester.
Card E · Standards editor. Made-up "Widget Record Format, version 0.3, clause 5": "The record carries the decision time." Test vector V1: a gate decided at 10:00:00, the record was signed at 10:05:00, and the record's time field says 10:05:00. Key check: apply each reading to V1. Reading 1 ("the time the gate decided"): V1 fails. Reading 2 ("the time the record was made"): V1 passes.
Card F · Governance lead. Proposed use: the chatbot approves refunds above $500 with no person in the loop. The gate's answer was hold ("wait for a person"). The required evidence was: (1) a retest of F-1 by someone other than the tester, (2) a working appeal route for customers, (3) a named owner who can switch the feature off. Present: item 3 only. Key check: compare the required list with what is present. Expected: items 1 and 2 missing.
Card G · Reviewer. A draft summary of Card A makes three claims: (1) "The released code reproduces the paper's figure on the released test set." (2) "The model will reach 91% on Big Widget images." (3) "The drop is caused by label noise." Key check: compare each claim with Card A. Expected: claim 1 supported; claim 2 contradicted by runs 4–6; claim 3 not shown by anything on the card.
A worked handoff (researcher track, Card A)
A weak handoff: "I reproduced the paper. I'm fairly confident it works. There may be some issues with our data. Over to the team."
It names a question but cannot be checked. It does not say which version, which data or which steps. "Works" has no scope. The open issue is vague. "The team" is not an owner.
A strong handoff:
- Packet and version. 500.R, version 0.1.0, Card A: the paper version, one pinned code commit, the released test set and the 200-image frozen Big Widget sample, as logged.
- What I checked. The six logged runs: three seeds on each data set. Not checked: label quality on our sample, other camera types, the paper's ablations, and the precision of the 83.5% figure (no sampling interval was computed for the 200 images).
- Bounded confidence statement. On the pinned commit and the released test set, runs 1–3 average 90.9%, inside the paper's 90.0–92.0 band. This supports C: "the released code produces the reported result on the released data". It does not show that the model will reach that figure on Big Widget images; runs 4–6 average 83.5%. Falsifier for C: if a rerun by someone else, at the same commit on the released test set with the same three seeds, averaged outside 90.0–92.0, I would withdraw C.
- Unresolved finding (open). The 7.4-point drop on our sample is unexplained. What the numbers do and do not say: the paper's ±1.0 is run-to-run spread across seeds on the released set. It tells you how much a rerun there should wobble. It is not a test of a difference between two data sets. The drop is far larger than that seed spread, so seed-to-seed wobble of the kind the paper measured is an unlikely explanation on its own. Nothing on the card says why it happened. Candidates: label errors in our sample, a different camera, or the model itself. Not attributed, not ruled on.
- Check instructions. With only Card A: average runs 1–3 and runs 4–6; expect 90.9% and 83.5%. No access to Dana's machine is needed. A full rerun would also need the code, the environment file and the images, which this lesson does not supply: listed as not checked.
- Next owner. Priya, a data-quality lead who did not run either experiment and who can commission checks, decides by the purchasing review which check to commission first. One option is a controlled relabelling check: a second labeller relabels the same 200 images, while the code, seeds and images stay unchanged. If the gap closed, that would point to label errors on this sample. It would not show label errors are the only cause elsewhere. A camera check is another option. The purchase waits for a person: it holds until Priya rules.
The evidence changed confidence (the reruns moved "reported" to "reproduced on released data"), the falsifier tests that claim and nothing else, the open question stayed open, and the owner did not produce the work. That is the shape every track aims for.
The seven tracks
Pick the track nearest your job. When an earlier Level 500 module's synthetic packet is published and checked, you may use it instead of the card. Until then, use the card.
| Track | Card | What you hand off | A typical bounded claim | What usually stays open | Typical independent next owner |
|---|---|---|---|---|---|
| Researcher | A | A replication discrepancy | "Reproduces on the released data; not shown on ours" | Why the result moved | A data-quality or methods lead who did not run it |
| Auditor or assessor | B | A missing-evidence register | "Refunds 1 and 2 have both required records; refunds 3 and 4 do not" | Whether the control ran on every refund, and worked | The control owner's manager, or an external assessor |
| Regulator or policy staff | C | A scope-exception memo | "On these stated facts, whether the duty applies turns on one missing fact" | The missing role fact; any legal conclusion | Counsel or the decision-maker named in the rule |
| Red-team lead | D | A ruling and retest handoff | "F-1 was accepted; a retest on version 4 recorded refusals for these five prompts" | Coverage beyond five prompts; a retest by someone other than the tester | A retester other than the original tester |
| Standards editor | E | A versioned ambiguity and test-vector note | "Clause 5 in version 0.3 can be read two ways; V1 separates them" | Which reading the editors adopt | The editors' group, with a named chair |
| Governance lead | F | An escalation record | "The gate held this use because items 1 and 2 were missing" | Go ahead, only this far, wait, or no | The executive named as accountable |
| Reviewer | G | An evidence and revision disposition | "Claim 1 is supported by Card A; claims 2 and 3 are not" | The author's revision | The author makes the fix; a second reviewer rules on the re-read |
Notes on the tracks, tied to the sources:
- Researchers and reviewers. Pineau and colleagues found that 34% of reviewers who were asked said the reproducibility checklist answers were useful for evaluating the submission.3 They also write that they did not yet have concluding evidence that the reproducibility program's processes improved the quality of the work.3 A reviewer's disposition should be just as careful: say which claims the evidence supports, not that the paper "is good".
- Auditors and assessors. MEASURE 2.1 asks that test sets, metrics and tool details be documented.1 A missing-evidence register lists what was required and absent. It never turns a missing record into a pass.
- Policy staff. MAP 1.1 asks that intended purposes, context-specific laws and the settings of deployment be understood and documented.1 GOVERN 6.1 asks for policies covering risks from third-party entities.1 A scope-exception memo records which facts the scope depends on and which are missing. It is not a legal opinion.
- Red-team leads. This course's own red-team rule is that a person, not the tester, rules on each finding, and accepted findings close only with a retest (TRUTH-STANDARD.md, rule 3). The handoff records the ruling, the retest, who ran it, and what was not tested. A retest log records what happened in that run. A later run that gives a different answer does not make the log wrong; it challenges whether the result repeats, or how far it covers. So a red-team falsifier is either a review of the original log, or a claim about repeatable behaviour on a pinned version with the repeat rule written down first. Retest transcripts hold prompts and outputs, so they stay in the controlled work record; the handoff lists them by hash.
- Standards editors. Living documents move. NIST's Playbook page, for example, was created on 8 July 2022 and shows an update on 10 June 2026.6 An ambiguity note pins the version it is about.
- Governance leads. A gate decides "go ahead · only this far · wait for a person · no".4 The escalation record says which answer the gate gave, why, and who must decide next. MANAGE 4.1 asks for post-deployment monitoring plans that include appeal and override, incident response, recovery and change management.1 MEASURE 3.3 asks for ways for end users and impacted communities to report problems and appeal outcomes.1 A hold is not a standing permission and not a quiet yes. The Foundation's checklist asks you to name the human who releases a hold, and how long a hold lives before it "decays to no".4 The escalation record carries both. By this course's rule, "the governance process" is not a person, and neither is the agent that drafted the record.
The NIST material also warns against using any of this as a box-ticking list. The AI RMF says its actions "do not constitute a checklist", and it does not prescribe risk tolerance.1 The Playbook is for voluntary use, and organisations may borrow "as many – or as few – suggestions as apply" to their use case.6 The handoff template is a minimum shape for a record, not a measure of whether the work underneath was good.
Four common failures
- Confidence without a handle. "High confidence" with no evidence, scope or falsifier. Fix: fill all four sides of the confidence statement, with a falsifier that tests the claim itself.
- The quiet close. The author marks their own open finding resolved. Fix: open findings stay open until the named next owner rules.
- Handing back to yourself. The next owner is the author, the author's direct collaborator on the same work, or a team. Fix: name a person who did not produce the work and who can decide.
- Instructions only the author can run. The check needs the author's machine, memory or private access. Fix: pin inputs, list steps and expected outputs, and test that a colleague can follow them. Whatever a stranger cannot rerun goes under "not checked".
Mapping your handoff to the prep rail
Level 500 includes one practice rail toward an AI governance or management-system audit certification. It is this course's own practice, not an exam. Its structure comes from this course's Level 500 map (levels/500/objectives.yml), not from any source cited here. On Bob's decision of 18 November 2026, the vendor and exam stay unnamed until Bob clears each name. No current exam alignment is claimed, and this course supplies no vendor-proctored exam and no endorsement.
The rail draws on six course objectives: 500.6-a, 500.7-b, 500.8-b, 500.9-d, 500.R-a and 500.R-b. Its practice is to produce a role-specific handoff, inspect a missing-evidence packet, and route an unresolved finding to an independent owner. You map each practice record to the objectives it exercises:
| Handoff part | Rail practice it exercises | Course objective it draws on |
|---|---|---|
| Packet and version; bounded confidence statement | Produce a role-specific handoff | 500.R-a |
| What was checked; missing-proof list | Inspect a missing-evidence packet | 500.6-a (integrity and origin versus attestation and effectiveness); 500.7-b (missing evidence for a control claim) |
| Clause, gate, evidence and owner rows | Inspect a missing-evidence packet | 500.8-b |
| Tested and untested coverage; ruling by someone other than the tester | Route an unresolved finding to an independent owner | 500.9-d |
| Unresolved finding; named independent next owner | Route an unresolved finding to an independent owner | 500.R-a |
| This mapping, with each item's evidence typed against its claim | All three | 500.R-b |
Evidence about a person is not evidence about a system
Sort evidence by the claim you want it to support, not by the item alone. The same item can bear on two claims. Morgan's ruling on F-1 (Card D) shows something about Morgan's practice as a red-team lead. It also records a claim about the chatbot. Ask of each item: which claim is this evidence for, and what does it fail to show?
- Claims about a person's preparation draw on your completed handoffs, your mapping table, a reviewer's comments on your practice and, later, if you sit one, an exam result. These say something about what you can do.
- Claims about a system's controls draw on items such as these, each of which supports only a narrow inference:
| Item | What it can support, under its assumptions | What it does not show |
|---|---|---|
| Retest log (Card D) | The listed prompts were recorded as refused on version 4, if the log is complete and genuine | Other prompts, other versions, or a retest by someone other than the tester |
| Signed record that verifies | The record is unchanged since someone holding the key signed it, if the key was not misused and the verifier is correct | That the event it describes happened, or that the control worked |
| Ruled red-team finding | A weakness was found and a person ruled on it | That no other weaknesses exist, or that the system is ready overall |
| Complaint and appeal records | An appeal route exists and was used | That appeals were decided well, or that everyone could reach the route |
| Register with gaps (Card B) | Which required records are present and which are absent | Whether the control ran on the cases that were never registered |
One rule does not bend: a person's exam result is never evidence about a system's controls.
The NIST AI RMF keeps people and systems apart too. GOVERN 2.2 is about people: personnel and partners receive AI risk management training to perform their duties.1 MEASURE 1.3 and MEASURE 2.1 are about the system: who assesses it, and documenting the test sets, metrics and tools used.1 The framework's trustworthiness characteristics (valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; fair with harmful bias managed) are characteristics of AI systems, not of the people who assess them.1
So:
- a passed exam concerns a person, never a system;
- a well-prepared assessor does not make the system they assess compliant;
- no single system artifact shows a system is "ready"; say what each one supports and what is missing;
- finishing this rail grants no certification, to you or to any system.
Check yourself
- Your handoff says "The vendor's model is fair, high confidence." Name the three things missing from its confidence statement.
- You are the red-team lead on Card D. The tester who found F-1 also ran the retest. Who must be the next owner of the retest result, and why?
- A colleague says the rail mapping table "proves our hiring assistant is audit-ready". Which claim is that table evidence for, and what is wrong with the colleague's claim?
Further learning (not a citation)
These free university courses go deeper. They are not sources for any claim in this module, and their materials are not copied here: Harvard CS 2881R, AI Safety (Fall 2025); Stanford CS 181/181W, Computers, Ethics, and Public Policy (Spring 2024); Stanford CS25, Transformers United V6. See levels/500/universities.md for links, and note its warnings about what each course does not teach.
Glossary
- Assurance handoff: this course's six-part record that lets an independent person check expert work and act on what is still open.
- Bounded confidence statement: a claim with its version, scope, supporting evidence, what it does not show, and the result that would change it.
- Falsifier: a check whose result would make you withdraw or narrow the claim you stated.
- Independent next owner: a named person who did not produce the work being checked and who can decide on the open finding. Independence from the author is not independence from the organisation.
- Packet card: a small made-up evidence packet in this lesson, with a key check anyone can do on paper.
- Reproducibility: obtaining similar results to those presented in a paper or talk, using the same code and data when available.3
- Prep rail: this course's practice route toward an unnamed exam; it certifies nothing.