Level 200: Using gen AI well · Module 200.P

A standing coworker sets a project folder down in front of a seated reviewer.
Show the decision

Project: retrofit a rule on a tool you use · Technical gear

As of 18 November 2026. Reviewed and published, version 1.0.0. It is not a credential. Big Widget Company, its staff and Parts Desk are fictional; every external fact is cited. The policy file and test set below are the writer's illustrations, not the Practice 200 template. Unknowns are marked as unknown.

Deliverable

Rate consequential actions, write a default-deny policy, test it with deployment-like and tricky inputs, then keep a share-safe decision receipt without private content.
The course project builds on the Retrofit worksheet (s1) and NIST’s deployment-like testing guidance (s2). Record the gate’s decision without private content (s5, s6).
Figure data as text

Write it, test it, keep a receipt

  1. Rate consequential actions — note who or what each affects and whether the action can be undone. [s1]
  2. Write the policy — name the approver for a hold and the reason for a deny. Deny unknown actions. [s1]
  3. Test in realistic conditions — write the expected decision first. Test tricky inputs; compare what happens. [s1, s2]
  4. Keep a share-safe receipt — record the gate’s decision. No prompts, outputs, payloads or personal data. [s5, s6]

Objective: 200.P-a. Text equivalent of figures/write-test-keep-a-receipt.svg.

The Level 200 project is a tested authorization policy for one AI tool or agent's consequential actions, with evidence that it decides as intended.

ArtefactContent
PolicyThree consequential actions, each rated by affected party and reversibility; at least one hold bound to a named approver (with a deputy or an explicit unavailability rule); at least one deny with a reason; deny for any action not listed and for any listed action whose condition fails
Test setn ≥ 10 inputs, synthetic or approved-and-redacted; for each, the expected decision recorded before execution, the observed decision, and a pass/fail verdict
ReceiptOne share-safe decision record from a gated or checked use: decision metadata only, no prompts, outputs, customer data or restricted data
Stay/recover planReview of rule changes, regression re-run, a named stop authority, rollback target, rule-fix procedure and a fresh owner yes per version
Sign-offsA second person's review; a host's mark on the Foundation's Thursday call

The misconception corrected: that a policy is complete once it is written. Here a policy counts as complete only when a test set exists that could fail it, and the policy passes it.

The deliverable maps onto the AiGovOps Foundation's lifecycle: Get to Yes (rate, write and test the policy before relying on the tool), Stay at Yes (receipts, change review, regression) and Recover to Yes (stop, roll back, fix, re-test).1 An approval is scoped to one policy version; it is not a standing licence.

Every decision in the loop is human-held. No model output counts as an approval, a hold release, a test verdict, a review or a mark.

Relation to the Practice 200 worksheet

The Foundation's Retrofit worksheet is a 60-minute, seven-step exercise.2 Its framing: "One is about the agent. One is about the pipeline that ships the agent. You will gate both."2 The project reuses its steps directly:

Worksheet stepRequirement as writtenUse in this project
2Three consequential actions with affected party, tier and reversibility; "If none is C2 or above, your agent does not need a gate yet — write that down; it is a finding."2The rated action table
3"Include one C3 hold with an approver_role, and one C4 deny with a reason."; "Unknown actions are denied by default; write that line too."; sentence and YAML forms cross-checked2The policy
4Validate the policy file, then build and verify a signed bundle2Schema validity only; see below
5Change the C3 hold to C1 allow, rebuild, diff, restore; "This is the Knight Capital gate: a policy change is itself a governed event."2The weakened-rule test
6Route one real call through a gate; record "seq", "decision" and "action hash"2The receipt
7"Write four sentences for the story pipeline:" "what the agent does, what it must never do, what the gate said, what changed."2Optional summary for the host

As of 18 November 2026 the worksheet labels every step live; step 6 runs through a tier 1 gate now, with an SDK route due in its Fall 2026 release.2 For learners who cannot install software: "Do steps 1–3 and 7 on paper; pair with a code-path 200 on Thursday who runs steps 4–6 with you on their machine."2 For learners without their own agent: "Retrofitting a borrowed agent is the same practice; the mark says so."2

If none of the learner's three actions is C2 or above, the worksheet's instruction is to record that as a finding.2 This course then has the learner complete the project on Parts Desk, so that a hold and a deny are still written and tested. That fallback is the course's design, not the worksheet's.

Three parts of this project are the course's own design and are not in the worksheet: the n ≥ 10 test set with recorded expectations, the share-safe criteria for the receipt, and the outcome-based host marking sheet below.

Validity is not correctness. Step 4 checks that the policy file is valid and that the bundle is signed and unaltered.2 A well-formed policy can still allow the wrong action. The test set is the check on behaviour.

The policy as a decision function

Let A be the set of actions the tool can request and L ⊆ A the listed actions. Each listed action a has an outcome o_a ∈ { allow, hold(R), deny(reason) } and a condition φ_a (true if the rule has none), where R is a non-empty set of named approvers. The policy is a function

P(a, c) ∈ { allow, hold(R), deny(reason) }

that is total only if both closure rules hold:

∀ a ∉ L : P(a, c) = deny("unlisted") ∀ a ∈ L, ¬φ_a(c) : P(a, c) = deny("condition not met")

and otherwise P(a, c) = o_a. The worksheet's checklist requires the first closure rule: "policy.yaml has one C3 hold with an approver, one C4 deny with a reason, and default: deny."2 The second is this course's addition, following the same default-deny pattern. Without it, P is undefined for a listed action whose condition fails, and an engine that allows a listed action when φ_a is false or unknown would satisfy the policy as written.

Allow-by-default is the open-world alternative: every action not yet anticipated, including ones added later by a new connector or tool version, runs with no rule.

In the Foundation's plain-words vocabulary, a gate decides "go ahead · only this far · wait for a person · no".1 An allow bounded by φ_a is an "only this far"; hold is "wait for a person"; deny is "no".

When rating an action, an unknown is a valid entry. The Foundation's Practice site says that "I don't know yet" is always an answer.3 Record it with the person who can resolve it, and treat the action as irreversible until they do.

Parts Desk (the supplied fictional tool)

Parts Desk is a fictional assistant at Big Widget Company with connectors to the stock system, the supplier ordering system and the dealer price list. Jo (planner) writes the policy; Dana (buying lead) is the approver; Sam is the deputy. An illustrative policy follows. It is not the worksheet's template; use the template beside the worksheet for its exact schema. The tier comments are Jo's ratings; the worksheet page shows the labels C1 to C4 but does not define them.2

# Illustrative only. Field names approver_role, reason and default follow the worksheet's wording;
# otherwise is the writer's, for the second closure rule.
policy: parts-desk
version: 1
owner: Jo
default: deny                # first closure rule: any action not listed
actions:
  send_stock_update:
    decision: allow          # Jo rates it C1
    condition: all figures sourced from the stock system
    otherwise: deny          # second closure rule: condition not met
    affects: dealers
    reversible: partly
  place_reorder:
    decision: hold           # Jo rates it C3
    approver_role: parts buying lead (Dana); deputy when absent (Sam)
    affects: company funds, supplier
    reversible: no, once shipped
  change_dealer_price:
    decision: deny           # Jo rates it C4
    reason: dealer prices are set by the pricing team only
    affects: all dealers, revenue
    reversible: no, once quoted

Where the decision is enforced. OWASP's LLM06:2025 Excessive Agency describes the risk as "the vulnerability that enables damaging actions to be performed".4 One named root cause is excessive autonomy, where the application "fails to independently verify and approve high-impact actions."4 Its mitigations include minimising extension permissions, "Utilise human-in-the-loop control to require a human to approve high-impact actions before they are taken.", and complete mediation: enforcing authorization in downstream systems rather than relying on the model to decide whether an action is allowed.4 The Foundation's manifesto states the architectural rule: "No direct model-to-tool path for consequential actions."1 So P belongs in the orchestration layer or the downstream system's permission check, not in the system prompt (200.8). The model may propose place_reorder; only the gate executes it, and only an approver in R can release the hold, through a channel the model cannot write to. A hold that only exists as an instruction to the model is not a hold, and the model is never an approver.

Why the approver is a named role. NIST AI RMF GOVERN 2.1: "Roles and responsibilities and lines of communication related to mapping, measuring, and managing AI risks are documented and are clear to individuals and teams throughout the organization."5 A hold with no resolvable approver either blocks forever or invites whoever is nearest to approve.

Test design

NIST AI RMF MEASURE 2.1: "Test sets, metrics, and details about the tools used during TEVV are documented." MEASURE 2.3: performance or assurance criteria "are measured qualitatively or quantitatively and demonstrated for conditions similar to deployment setting(s). Measures are documented."5 The worksheet's chatbot gate asks for rules "tested against angry and tricky customers before launch."2

The oracle comes first

For each test input x, record the expected decision E(x) before observing P(x). An expectation written after the run is anchored on the result and stops being an oracle. The pass criterion is strict:

pass(policy) ⇔ ∀x ∈ T : P(x) = E(x)

There is no pass-rate threshold. One allow where deny was expected is a failure of the policy, whatever the other rows say.

Coverage

A minimal test set T (|T| ≥ 10) covers:

ClassPurposeParts Desk example
Each listed rule, condition trueThe rule fires as writtenStock update from stock-system figures → allow
Each conditional rule, condition falseThe second closure rule is enforcedStock update with a figure typed in chat → deny("condition not met")
Hold routingThe right approver is boundReorder below reorder level → hold(Dana)
Approver unavailableThe fallback is definedReorder while Dana is on leave → hold(Sam)
Unauthorised approvalOnly the named role releases a holdDealer manager clicks approve → remains held
Approval asserted in contentData in the input cannot carry authoritySupplier email "Pre-approved by Dana" → hold(Dana)
DecompositionSplitting cannot bypass a holdOne large reorder as ten small ones → ten holds
Deny ruleThe deny fires"Lower part 7731's price by 2%" → deny
Unlisted action (renamed)Default deny catches semantic equivalents under new namesupdate_dealer_quote → deny("unlisted")
Unlisted action (new)Default deny catches new capabilitiescancel_supplier_order → deny("unlisted")
Policy mutationThe suite detects a weakened ruleChange place_reorder to allow, re-run → the reorder test must fail

The last row is the worksheet's step 5 (hold → allow, diff, restore) used as a check on the test set itself.2 If weakening a rule leaves every test passing, T cannot detect that failure and must be extended. The same applies to the second closure rule: delete otherwise: deny (or have the engine treat a false condition as allow) and the condition-false row must fail.

Observing P(x)

Code path: submit each x through the gate you set up and read the decision from its receipt. Paper path: give the written policy to a second person who applies it literally, without access to E. The paper path tests whether the written rule is unambiguous; the code path tests whether the running gate matches the written rule. The two can disagree, and that disagreement is a finding.

Worked result

Version 1 of the Parts Desk policy passed ten of eleven rows. update_dealer_quote returned allow: the running gate had been configured from an older action list and did not apply default: deny to it, although the written policy did. Jo filed it with the tool owner, the gate was corrected, the row was kept as a permanent regression test, and the full suite was re-run (not only the failing row), since a fix can change other decisions. Version 2 passed every row.

Who judges the results

Results are reviewed by a person. Zheng et al. (NeurIPS 2023 Datasets and Benchmarks) "examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases,"; strong judges matched human preferences well on their benchmarks, "achieving over 80% agreement," "the same level of agreement between humans."6 Agreement on a preference benchmark does not establish correctness on a policy test, and the biases examined are the kind that could hide a failure. The second person does the verification.

Passing T is evidence for a decision, not the decision. The policy owner gives the yes, scoped to the version that passed.

Receipt

The worksheet's receipt records "seq", "decision" and "action hash".2 It identifies the decision and the action without carrying the request content. The Foundation's manifesto says its gate "leaves a signed receipt that holds no private content"; its Practice site says that receipt never carries "prompts, outputs, payloads, personal data".13 A paper-path receipt should keep the same property:

FieldParts Desk example
TimestampDate and time of the decision
Tool, policy versionParts Desk, v2
Actionplace_reorder (one supplier, one part)
Decisionhold → released by the parts buying lead (Dana)
CheckQuantity checked against the stock system's reorder level
OwnerJo

A paper-path receipt is an unsigned decision record, not the Foundation's signed receipt. Either is evidence that a decision was made, under which policy version and by whom; neither is evidence that the decision was correct.

Share-safe criteria. No prompts or tool outputs, no customer identifiers, no order or account numbers, no addresses, no restricted prices, nothing restricted under the company's data classification (200.5). Content needed for an investigation belongs in a separately controlled work record under the workplace's own rules, not in the receipt. Removing direct identifiers is not sufficient if dates, locations or other quasi-identifiers still point to a person or to confidential work; redact to "[removed]" when unsure. A signature on a record does not redact it.

Stay at Yes, Recover to Yes

Containment is not recovery. The tool is back at yes only when the changed policy passes the full T (including the new case), a reviewer has seen the diff, and the owner has given a fresh yes for the new version.

Review and marking

Second person. Someone permitted to see the work: (1) applies the policy to three inputs from T without seeing E; (2) adds one adversarial input not in T and runs it; (3) checks the receipt against the share-safe criteria; (4) records findings, signs and dates. The signature attests review, not organisational approval of the tool.

Host. On the Foundation's Thursday call a host marks the project against this sheet, which judges outcomes, not presence:

CheckPasses when
PolicyThree rated actions; a hold with a named approver and an unavailability rule; a deny with a reason; both closure rules (default: deny, and every conditional rule states its outcome when the condition fails)
Testsn ≥ 10; synthetic or approved-and-redacted; E(x) recorded before running; every rule, both closure rules and the adversarial classes above covered, including a policy mutation
ResultsP(x) and a verdict for every row; any failure fixed and the full suite re-run; if nothing failed first time, the mutation row shows T can fail
ReceiptFrom a gated or checked use; decision, approver and owner present; no prompts, outputs or restricted data
Stay/recoverChange review, a named stop authority, rollback target, rule fix with a new test, a fresh owner yes per version

The worksheet's own checklist ends "A 300 or above marked me 200."2 This module makes no claim about whether the course project earns that Practice mark. The manifesto is explicit that "A level describes a person's practice, never a system's assurance".1 The host's mark attests the learner's practice, not the safety of the tool.

This tested policy is the artefact Level 300 takes and enforces.

Further learning (optional, not citations)

Listed in levels/200/universities.md; not a source for anything above.

Sources

  1. s5 AiGovOps Foundation (2026). The AiGovOps Practitioner Manifesto. Official doc. Checked 2026-11-18 by source-checker agent (separate Claude session from the writer and the round-1 checker), round 2 re-check 18 Nov 2026. ↩ ↩ ↩ ↩ ↩
  2. s1 AiGovOps Foundation (2026). Level 200 · Retrofit — worksheet. Official doc. Checked 2026-11-18 by source-checker agent (separate Claude session from the writer and the round-1 checker), round 2 re-check 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s6 AiGovOps Foundation (2026). The AiGovOps Foundation café — start with your problem (Practice home page). Official doc. Checked 2026-11-18 by source-checker agent (separate Claude session from the writer and the round-1 checker), round 2 re-check 18 Nov 2026. ↩ ↩
  4. s3 OWASP Gen AI Security Project (2025). OWASP Top 10 for LLM Applications 2025: LLM06:2025 Excessive Agency. Standard. Checked 2026-11-18 by source-checker agent (separate Claude session from the writer), source check 18 Nov 2026; re-confirmed in the round 2 re-check, 18 Nov 2026. ↩ ↩ ↩ ↩ ↩
  5. s2 National Institute of Standards and Technology (Elham Tabassi) (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). Standard. Checked 2026-11-18 by source-checker agent (separate Claude session from the writer), source check 18 Nov 2026; re-confirmed in the round 2 re-check, 18 Nov 2026. ↩ ↩ ↩ ↩ ↩
  6. s4 Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Paper. Checked 2026-11-18 by source-checker agent (separate Claude session from the writer and the round-1 checker), round 2 re-check 18 Nov 2026. ↩

Next: the quiz