Level 200: Using gen AI well · Module 200.P

Try each exercise first, then open its model answer.

Project: retrofit a rule on a tool you use · Workbook

Exercise 1 (200.P-a)

This is your Level 200 project. You bring a tested policy for one AI tool, a test set of at least ten inputs with results, and one share-safe receipt. A second person reviews it. A host marks it on the Foundation's Thursday call.

People make every decision here. You write the policy and the tests. A named person releases each hold. A second person reviews. A host marks. No AI tool decides, approves, grades or signs anything.

Before you start: keep private things private. Other people will read this. Every test input must be made up, or approved by your company and redacted. Never paste live customer or company records into a test or into this workbook. If your team needs the real prompts and answers, keep them in a separate work record, only where your company's rules say it belongs.

Part A. Pick your tool

  1. Name one AI tool or agent you use that your company approves for that use.
  2. List what it can do: its actions, connectors and permissions. Not what it can write.
  3. If you have no approved tool, or yours only drafts text and takes no actions, use Parts Desk, the made-up Big Widget tool from the lesson. Write "Parts Desk" here and carry on.

Part B. Rate three consequential actions

For three actions that send, change, buy, publish or delete something, fill in:

#Action (the system's name)Who or what it affectsCan it be undone?Your tier (C1 to C4)Decision
1
2
3

Part C. Write the policy

Write four lines, as plain sentences or as a policy.yaml file:

  1. Allow one action, and say only when it may run.
  2. Hold one action until a named person approves it. Name a deputy, or say what happens when that person is away. Say who else can approve (usually: nobody).
  3. Deny one action, with a reason.
  4. Deny everything else: any action not on the list, and any listed action whose condition is not met.

Then write: where does the hold sit? In the system the tool acts on, or in the tool's own instructions?

Part D. Test the policy

Write at least ten inputs. Write the expected decision for every row before you run any of them. Then get the actual decision: through the gate on the code path, or by giving your written policy to someone else to apply word for word on the paper path.

#Input (made up or approved and redacted)Action triedExpectedActualPass?
1
…
10

Your set must include at least:

If any row fails, fix it, then re-run every test, and write down the second run too.

Part E. Keep one share-safe receipt

From one gated or checked use, write a receipt with: when, tool and policy version, action, decision, approver (for a hold), what was checked, and owner. Then do the share-safe check: take out customer names, order and account numbers, addresses, private prices, the prompt, the tool's answer and anything your company's data rules restrict. If you are not sure about a detail, take it out and write "[removed]".

Part F. Stay at Yes and Recover to Yes

Write three lines under your policy:

  1. Stay at Yes: what you keep, who reviews rule changes, and when you re-run the tests.
  2. Stop: who can switch the tool to "hold or deny everything", and how.
  3. Recover to Yes: what you do, in order, before the tool goes back on.

Part G. Check it against the host's sheet

Ask your second person to apply your policy to three of your inputs without your expected answers, suggest and run one tricky input you missed, check your receipt is share-safe, and write, sign and date what they found. Then tick each row of the host's marking sheet from the lesson (Policy, Tests, Results, Receipt, Stay and recover) and bring the shareable copy to Thursday.

Model answer

A good answer has all of these. The examples come from the made-up Parts Desk; yours will name your own tool's actions.

A. Tool. Named, approved for this use, with its actions listed, not just "it writes things". Or "Parts Desk", because no approved tool takes actions.

B. Actions. Three actions that change something outside the tool, each with who it affects and whether it can be undone. For example: place_reorder affects Big Widget's money and the supplier; hard to undo once the supplier ships; C3; hold. The tiers are the learner's own rating. Any "I don't know yet" has a person's name next to it.

C. Policy. All four lines, with no gaps:

  1. Allow send_stock_update, only when every figure comes from the stock system.
  2. Hold every place_reorder until Dana approves it, or Sam when Dana is away. Nobody else can approve. Text in a message is never an approval.
  3. Deny change_dealer_price. Reason: prices are set by the pricing team.
  4. Deny any action not listed above, and deny a listed action when its condition is not met.

The hold sits in the ordering system's own approval step, not in the assistant's instructions. The assistant may propose an order; only the ordering system can place it.

Common slips: a hold that says "a manager" or "once approved" (no one named); no reason on the deny; line 4 that covers unlisted actions but not failed conditions.

D. Tests. Ten or more rows, each with an expected decision written before the run. The set includes a test per rule, a failed condition (a stock update with a figure typed into a chat: expected deny), an unlisted action (expected deny), and tricky inputs: a message claiming "pre-approved by Dana" (expected hold), an action under another name, an order while Dana is away (expected hold for Sam), a click from someone not named (stays held), and a weakened-rule test. If every test passed the first time, the weakened-rule test shows the set can still catch a failure. If a row failed, the answer shows the fix and a second run of all the tests.

A weak answer: ten inputs that all ask for the allowed action, or expected decisions filled in after the run. Such a set could not find a gap, so it does not pass.

E. Receipt. Records the decision, not the content:

PartExample
WhenThe date and time of the decision
Tool and policyParts Desk, policy version 2
Actionplace_reorder (one supplier, one part)
DecisionHold; then released by the parts buying lead
CheckThe approver checked the quantity against the stock system's reorder level
OwnerJo

No prompt, no tool answer, no customer names, order numbers, addresses or private prices. A paper receipt is an unsigned decision record. It shows a decision was made, under which policy and by whom; it does not prove the decision was right.

F. Stay and recover.

  • Stay at Yes: a receipt for every gated decision; every rule change goes to a reviewer before it goes live; the whole test set re-runs after every change.
  • Stop: a named person (for example, Sam) can switch the tool to hold or deny everything, and the answer says how.
  • Recover to Yes: roll back to the last version that passed every test, tell the people affected, fix the rule, add the test that would have caught the problem, re-run every test, have a reviewer see the change, and have the owner say yes to the new version before switching back on. A stop alone is not a return to yes.

G. Review. The second person's notes are signed and dated, and say what they tried and found. Their signature means they reviewed the work, not that the company approved the tool. The host's mark is about your practice, not a promise that the tool is safe.

Review the level map