Level 500: Governance, safety and research · Module 500.R

Try each exercise first, then open its model answer.

Research and governance roles (role tracks) · Workbook

Reviewed and published, version 1.0.0. As of 20 November 2026. Big Widget Company, its people and the seven packet cards (A to G) in the lesson are made up. The six-part handoff, the seven tracks and the prep-rail mapping are this course's own practice, not a law, a standard or an exam syllabus. The model answers are examples of what a good answer contains, not the only right answers. A person reviews your work. An AI does not.

What not to write here. Use only the packet cards in this lesson. Do not paste real company records, personal data, chat logs, private-repository content or real exam material into these exercises. If your handoff points to prompts or transcripts, list them by hash; do not copy them in.

Exercise 1 (500.R-a): write a six-part handoff from your track's packet card

Pick the track nearest your job: researcher (Card A), auditor or assessor (Card B), regulator or policy staff (Card C), red-team lead (Card D), standards editor (Card E), governance lead (Card F) or reviewer (Card G). Everything you need is on the card. Pin your packet as "500.R, version 0.1.0, Card X".

  1. Write a handoff with all six parts:
    1. Packet and version: the card you used, pinned as above.
    2. What you checked: the steps you took, and the steps you did not take. Anything the card gives only as a result (a signature check, a chatbot retest) goes under "not checked".
    3. Bounded confidence statement: fill every blank of "On version V, within scope S, evidence E supports claim C. It does not show N. If check K returned R, I would withdraw or narrow C." K must test C itself, not explain a different open question.
    4. Unresolved finding: at least one open question, described so someone can rule on it, and marked open. Do not say what caused it unless the card shows it.
    5. Check instructions: the card's key check, its steps and the expected result, so a stranger can do it with only the card. Say what a full rerun would need that the card does not supply.
    6. Named independent next owner: a named person who did not build, test or write the work, who has the power to decide, what they must decide, and by when. If a shared company interest could sway the ruling, say so.
  2. For each item of evidence in part 3, write one line: if this evidence had come out the other way, would my view change? Remove any item where the answer is no.
  3. Check your handoff against the four common failures: confidence without a handle, the quiet close, handing back to yourself, and instructions only you can run. For each, write "avoided" and point to the part of your handoff that avoids it.
Model answer

A model answer for the red-team lead track (Card D, made up). Other tracks follow the same shape with their own card, claim and owner.

  1. Packet and version. 500.R, version 0.1.0, Card D: finding F-1, Morgan's ruling that F-1 is accepted, and the log of Sam's retest of five listed prompts on version 4 of the support chatbot. The prompts and replies stay in the controlled work record and are listed here by hash.
  2. What I checked. Matched the five listed prompts to the five log rows: five of five recorded as refusals. Not checked: running the prompts against the version-4 build (the card gives only the log); other prompts, languages or longer conversations; any later version; a retest by anyone other than Sam, the original tester.
  3. Bounded confidence statement. On version 4, for the five listed prompts, the retest log supports C: "in Sam's retest after the fix, these five prompts were recorded as refused". It does not show that other prompts are refused, that later versions refuse them, that a repeat run would give the same result, or that someone other than the tester would. Falsifier for C: C is about what the original retest recorded, so the check is a review of that record. If a review of the original retest record (the prompts and replies held by hash in the controlled work record) showed that any of the five replies was not a refusal, or that the log rows do not match the five listed prompts, I would withdraw or narrow C. A different result in a later rerun would not make the log wrong; it would challenge whether the result repeats, which C does not claim. That question stays with part 4.
  4. Unresolved finding (open). Whether the fix closes F-1. The only retest was run by the tester who found it, coverage beyond the five prompts is untested, and no repeat rule was written down before the retest, so whether the refusals repeat is not known. Not ruled on.
  5. Check instructions. With only Card D: match each of the five listed prompts to its log row; expect five refusals. No access to Sam's machine is needed. A full retest would also need the version-4 build and the prompts from the controlled record, which this lesson does not supply: listed as not checked.
  6. Next owner. Lee (made up, not on the card), a retester who did not find, fix or retest F-1 and who has the authority to rule on retests, decides by Friday whether the fix closes F-1 or whether wider coverage is needed first. The card states no launch date or other shared pressure; if one existed, I would record it here and name an outside assessor if one is available.

Relevance check. "Sam is an experienced tester" is removed from part 3: whichever way it came out, it would not change whether the five prompts were refused. The retest log stays: a log row that was not a refusal would change the claim.

Failure check.

  • Confidence without a handle: avoided; part 3 names version, scope, evidence, what it does not show and a falsifier that tests C itself (a review of the original record, not a new run).
  • Quiet close: avoided; whether the fix closes F-1 is marked open and only Lee rules on it.
  • Handing back to yourself: avoided; Lee did not find, fix or retest F-1, is a named person rather than "the red-team group", and has been given the power to rule.
  • Instructions only you can run: avoided; part 5 uses only the card, gives the expected result, and lists what a stranger cannot rerun as not checked.

What a good answer from any track contains:

  • a pinned card and a "not checked" list, so coverage never looks complete by silence;
  • a confidence statement with every side filled, so it is not "high confidence" alone (the NIST AI RMF asks for testing with measures of uncertainty and documented reporting of results);
  • a falsifier about the claim made, with any check that would explain an open question kept with that question (on Card A, a relabelling check belongs with "why the drop", not with "it reproduces");
  • an open finding that stays open, not closed by strong evidence on a different question;
  • an owner who did not produce the work and who can decide, matching NIST's best practice of separating those who build and use models from those who verify and validate them. Independence from the author is not independence from the organisation.

Exercise 2 (500.R-b): map your handoff to the prep rail, and type the evidence by claim

  1. Make a three-column table for the handoff you wrote in Exercise 1: handoff part, rail practice it exercises (produce a role-specific handoff; inspect a missing-evidence packet; route an unresolved finding to an independent owner), and course objective it draws on. Use only the six rail objectives: 500.6-a, 500.7-b, 500.8-b, 500.9-d, 500.R-a and 500.R-b. Every one of the three practices must appear at least once.
  2. For each of these eight items, write which claim it is evidence for (about a person's preparation, about a system's controls, or both) and what it does not show:
    • your completed handoff;
    • your mapping table;
    • a reviewer's comments on your practice handoff;
    • an exam result you might earn later;
    • Card D's retest log;
    • Morgan's ruling on F-1 (Card D);
    • refund 1's signed record, "valid under key K1" (Card B);
    • Card B's register of refunds above $500.
  3. A colleague says: "Our assessor finished the rail and passed the exam, and the retest log is clean, so the chatbot is ready for audit." In two or three sentences, say what is wrong with the claim.
Model answer

1. Mapping table (red-team lead example).

Handoff partRail practiceCourse objective
Packet and version; bounded confidence statementProduce a role-specific handoff500.R-a
What I checked and did not checkInspect a missing-evidence packet500.7-b
Tested and untested coverage; ruling by Lee, not the testerRoute an unresolved finding to an independent owner500.9-d
Unresolved finding; Lee as named independent next ownerRoute an unresolved finding to an independent owner500.R-a
This mapping, with each item's evidence typed against its claimAll three500.R-b

All three practices appear, and every objective cited is in the rail's list. A track with clause, gate, evidence and owner rows (such as policy staff or governance lead) would also map those rows to "inspect a missing-evidence packet" and 500.8-b.

2. Evidence typed by claim.

ItemClaim it is evidence forWhat it does not show
Your completed handoffYour practice at writing handoffs (person)Anything about the chatbot's controls
Your mapping tableYour preparation on the rail (person)Anything about a system
Reviewer's comments on your practiceYour practice (person)Anything about a system
A later exam resultYour preparation (person), and only thatNever evidence about a system's controls
Card D's retest logThe five listed prompts were recorded as refused on version 4 in Sam's run, if the log is complete and labelled right (system)That the record is complete (a check against Sam's original prompts and replies), whether the refusals repeat, other prompts, later versions, or a retest by someone other than the tester
Morgan's ruling on F-1Both: Morgan's practice as a lead (person), and that a weakness was found and a person ruled on it (system)That no other weaknesses exist, or that the chatbot is ready
Refund 1's signed recordThe record is unchanged since someone holding K1 signed it, if the key was not misused and the verifier is correct (system record)That the refund happened as described, or that the control worked; the card gives only the verifier's result, so the check is not checked
Card B's registerWhich required records are present and absent: 2 of 4 refunds complete (system)Whether the control ran on refunds that were never registered

3. What is wrong. The rail and the exam are evidence about the assessor, not about the chatbot. The retest log is evidence about the chatbot, but only that five prompts were recorded as refused on version 4, in a retest run by the original tester; no single record shows a system is "ready". The NIST AI RMF keeps people and systems apart (GOVERN 2.2 is about training people; MEASURE 1.3 and 2.1 are about assessing the system and documenting its testing), finishing this rail grants no certification to anyone or any system, and the course claims no exam alignment and supplies no vendor-proctored exam or endorsement.

Next module: 500.P Capstone: publish work that survives review