Level 400: Engineering AI systems · Module 400.9

Try each exercise first, then open its model answer.

The audit gate: prove it to a stranger · Workbook

Three exercises, one for each thing this module teaches. Big Widget Company, its people, its formula and its bundles are made up. Do not use real names, customers, dealers or documents from your own work. Write short answers. "I don't know yet" is an honest answer if you also say who you would ask.

Your answers are a private learning record. Do not paste customer, dealer or personal details into them.

Exercise 1 (400.9-a)

Is the average the problem? Big Widget's warranty analytics lead, Dana, proposes this rule:

For each dealer, average its monthly warranty claims over the last twelve months. Any month more than 40 percent above that average is treated as overclaimed, and Big Widget sends the dealer a notice clawing back the excess.

Dana's manager says: "It's ten lines of code and an average. An average can't be wrong. Ship it."

  1. In one or two sentences, say what the difference is between the rule's arithmetic and its premise.
  2. Name two ways the premise could be wrong even if the code is perfect.
  3. Using the Robodebt Royal Commission, explain why the premise must be checked before the first notice, and why getting advice is not enough on its own.
  4. Who should check it, and what three kinds of evidence should they ask for?
Model answer
  1. The arithmetic computes an average and a 40 percent line correctly. The premise is a claim about the world and the law: that a month above the line means the dealer overclaimed, and that the dealer contracts allow Big Widget to demand money on that evidence alone.
  2. Any two of: widgets may fail more in winter, so a cold month looks high with nothing wrong; a small rural dealer's claims can jump from three to nine in a month with nothing wrong; the contracts may not allow a clawback on this evidence, or may require notice and an appeal.
  3. Robodebt raised debts using income averaging. The Royal Commission's report (7 July 2023, updated 11 July 2023) called it "a crude and cruel mechanism, neither fair nor legal". Volume 1 records that in late 2014, before the scheme began, the Department of Social Services had in-house legal advice raising concern that averaging might not be consistent with the legislative framework, and policy advice that averaging did not accord with the legislation; the scheme went ahead without the legislative change the department had said was needed. In mid-2018 a tribunal decision that income averaging was unlawful was referred between departments. So the lesson is not only "ask first". Adverse advice existed, and nothing made it binding. Keep three things apart: obtaining advice, an independent validation decision, and enforcing that decision so a failed or missing check blocks every notice. Waiting for complaints means the harm has already been done to the people who got notices.
  4. An independent validator: someone who did not write the rule (not Dana, and not Dana's manager, who wants the money back), and not Marcus, who runs the system. The evidence is a legal basis (the contracts allow it, and what notice and appeal dealers are owed; if that advice says no, the answer is no), a statistical validation (on months auditors already checked by hand, how often the rule flags a correct month) and an equity test by cohort (whether that error rate is much higher for small, rural or cold-climate dealers).

Exercise 2 (400.9-b)

Fix the gates. A colleague drafts this premise gate for the clawback rule:

- action: decision.rule.publish
  decision: allow
  approver_role: rule_author
- action: notice.issue
  decision: allow

And this protected-attribute gate:

- action: decision.rule.publish
  decision: deny
  unless:
    - no_column_named(nationality)
    - receipt_exists(proxy_test)

Marcus in warranty operations is the Level 300 holder: he runs the clawback system, takes its alerts and can stop it.

  1. Rewrite the premise gate so it works. Say who approves, what evidence is required, and what a receipt must record before a notice can go out.
  2. Explain in one or two sentences why Marcus should not be the approver.
  3. Rewrite the protected-attribute gate so it catches more than the one column, and so a failed proxy test cannot pass. Use the Dutch case to say why banning the column alone is not enough.
  4. Write each gate as one sentence a minister or board member could read.
Model answer
  1. For example (course design, made up for Big Widget):

     - action: decision.rule.publish
       decision: hold
       approver_role: independent_validator   # not a rule author, not the Level 300 holder
       evidence_required: [legal_basis, statistical_validation, equity_test_by_cohort]
     - action: notice.issue
       decision: deny
       unless: valid_receipt(decision.rule.publish, decision == allow, rule_version == current, all_checks == pass)

    The team that built the rule cannot approve it. The gate fails closed: no validated premise, no notice. A receipt that merely exists is not enough: it might record a hold or a deny, or be for an older version of the rule. A valid receipt records allow, names the exact rule version, records that every check passed, and is signed by the independent validator's key. Validating version 0.2 does not license version 0.3. A version that allows notices and logs a warning still lets notices out while the check is missing.

  2. Marcus's job is to keep the system running (Stay at Yes). The validator's job is to judge whether the rule should run at all (Get to Yes). If one person does both, the person who runs the system decides whether it is sound. A validator from outside warranty, such as Ines in model risk, does it instead; Marcus can still stop the system at any time.
  3. For example (course design):

     - action: decision.rule.publish
       decision: deny
       unless:
         - valid_receipt(feature_review, rule_version == current)   # every risk feature listed and checked
         - no_protected_attribute(features)                         # no protected column at all
         - valid_receipt(proxy_test, rule_version == current, result == pass)  # run by the independent validator
     - action: notice.issue
       decision: hold
       approver_role: human_reviewer         # a named person reviews every adverse decision

    In a decision dated 25 November 2021, the Dutch data protection authority (the Autoriteit Persoonsgegevens, AP) imposed fines totalling EUR 2.75 million on the tax administration for processing the (dual) nationality of childcare-benefit applicants in an unlawful and discriminatory way. One indicator in a system that selected risky applications was whether the applicant was Dutch or not Dutch. That was direct use, which a column ban would catch. But a feature that stands in for a protected attribute (a proxy, such as a postcode or a language preference) would slip past. The worksheet bars "protected attributes and their proxies" and asks for meaningful human review of every adverse decision. The course adds the receipt, and the draft's receipt_exists(proxy_test) is not enough: a test that found a proxy still leaves a receipt. The proxy test must have passed. A dealer's region may pass only if the test shows it is not standing in for something protected.

  4. Premise gate: "No clawback notice goes out until someone who did not write the rule, and does not run it, has checked this exact version and recorded that it is lawful, that it gets the answer right, and that it is not wrong more often for some dealers than others." Protected-attribute gate: "The rule may not use nationality, ethnicity or anything shown to stand in for them, and a named person reviews every notice before it is sent."

Exercise 3 (400.9-c)

Verify a stranger's bundle offline. The course sandbox, apps/sandbox400/, holds three practice bundles from one Big Widget log: apps/sandbox400/bundles/bundle-a, bundle-b and bundle-c. Each has its own one-page VERIFY.md. The checker, apps/sandbox400/verify_bundle.py, needs only Python 3.9 or later. Read apps/sandbox400/README.md first. The bundles and keys are made up; no real system made them, and the keys come from public labels, so anyone can sign with them.

Your trust material is apps/sandbox400/trust/bw-warranty-gate.trust.json. It stands for what you would receive separately from the bundle. Check it against the values on the lesson page:

In a real audit, the keys and checkpoint come by a route you can authenticate, never in the same download.

  1. Switch off the network. From the repository root, follow each bundle's VERIFY.md by hand, or run

     python3 apps/sandbox400/verify_bundle.py apps/sandbox400/bundles/bundle-a --trust apps/sandbox400/trust/bw-warranty-gate.trust.json

    then the same for bundle-b and bundle-c. For each bundle, write accept or reject, and the step that decides it. Only then compare with apps/sandbox400/prepared/4009-c-verify-bundle-a.txt, -b.txt and -c.txt.

  2. Make tampered copies of Bundle A and check where each fails:

     python3 apps/sandbox400/sandbox400.py tamper --bundle apps/sandbox400/bundles/bundle-a --trust apps/sandbox400/trust/bw-warranty-gate.trust.json --case key-swap --case bad-manifest-signature --case altered-record-repacked --case extra-file-repacked --case suffix-cut --case fork-resigned --case nested-payload --case payload-in-eval-report --case person-in-crosswalk

    Before you run it, predict the step each copy fails at. apps/sandbox400/tamper.py says what each case changes.

  3. Bundle B passes its signatures and its chain. Why is that not enough?
  4. Bundle C ends at sequence 214, the same number as the checkpoint. Why is the number alone not enough?
  5. A colleague says: "Just check the signatures against the key that came inside the bundle; it's right there." Which tampered copy shows what is wrong with that, and why?
  6. Did you need Beacon, a Big Widget account or a network? Say what you did need.
  7. Bundle A carries a Level 300 receipt that was not signed when it was made. Write your own one-sentence verification statement for Bundle A, with your name, the date and what you found. Then write one sentence that overclaims, and say what is wrong with it.
  8. In payload-in-eval-report, the producer re-signed the file list after adding a list of model outputs to the evaluation report, so the file list's signature checks out. What fails it? The checker accepts Bundle A: is the content check then finished?
  9. In nested-payload, the producer re-signed the changed receipt, so its signature verifies under the pinned key. Why does the bundle still fail?

If you could not run the checker offline, do not write a statement saying you verified a bundle. You can read the prepared results instead, and answer questions 3 to 6, 8 and 9 from them and the lesson. For question 7 write: "I have not verified a bundle." That is an honest answer.

Model answer
  1. What each bundle should give, and the step that decides it:
    BundleResultThe step that decides it
    AAcceptEvery step passes: inventory; both keys match the pinned fingerprints and the file list's signature and hashes check out; 214 receipt signatures valid and the chain continuous from seq 1; a signed checkpoint at 214 matches the head hash and policy 0.3; the policy hash matches; the closed formats hold and the screen found nothing; one carried Level 300 item, not authenticated at source. Your read-through of its text files is still to do.
    BRejectStep 3, checkpoint: 163 signatures valid and the chain continuous, but it stops at 163, so 51 records are missing from the end.
    CRejectStep 3, checkpoint: 214 signatures valid and the chain continuous to 214, but the hash of record 214 does not match the checkpoint.
  1. What the tamper run should show, each copy caught at its expected step:

    CopyFails at
    key-swapStep 1: the evidence key in the bundle is not the pinned key
    bad-manifest-signatureStep 1: the file list's signature does not verify under the pinned packaging key
    altered-record-repackedStep 2: the changed receipt's signature is invalid
    extra-file-repackedStep 0: a file outside the inventory, even though the file list was re-signed
    suffix-cutStep 3: truncated, 12 records missing
    fork-resignedStep 3: the head hash at 214 does not match
    nested-payloadStep 2: a field outside the receipt's closed format
    payload-in-eval-reportStep 5: a field outside the evaluation report's closed format
    person-in-crosswalkStep 5: the screen flags an email address

    For every bundle, note the carried Level 300 receipt as unsigned at source and not verified.

  2. A hash chain shows nothing in the middle was changed or removed. It cannot show nothing was cut off the end: the front part of a valid chain is still a valid chain. Only the checkpoint you received separately shows that B stops 51 records short.
  3. Someone holding the signing key could sign a different history of the same length. That is what C is: rewritten after seq 150 and re-signed with the evidence key. The hash of the last record ties the bundle to the one history Big Widget committed to, so the number, the hash and the policy version must all match.
  4. key-swap. Its records and file list were re-signed with impostor keys, and the bundle carries those keys, so every signature checks out against the keys inside it. That shows only that whoever made the bundle signed it. A signature check shows that the holder of the matching private key signed the message; it does not say whose key it is. The pinned fingerprints, received by another route, tell you the keys are Big Widget's, and the swapped keys do not match them. A second file emailed alongside the bundle is not a separate route.
  5. No. VERIFY.md is followed "without Beacon, without an account, without us", on a machine with no network ("airplane mode counts"). You needed the bundle, its VERIFY.md, Python, and the key fingerprints and checkpoint you got separately.
  6. Bounded, for example: "I, [your name], on [date], verified course practice Bundle A offline by its VERIFY.md against the sandbox trust file bw-warranty-gate.trust.json: its 214 receipt signatures match the pinned evidence key (fingerprint f0c5ef8d…) and its file list the pinned packaging key, its chain is continuous from sequence 1 to the expected checkpoint (214, head hash matching the one on the lesson page) under policy version 0.3, and I found no payload content, by the checker's inventory, format and screen checks and by reading its text files myself; I did not verify the carried Level 300 receipt, which was unsigned at source." The checker prints a template of the same shape when A passes. Kofi Mensah's statement in the lesson has the same shape.

    Overclaims, for example: "I verified the bundle, so the clawback system is lawful, fair and complete." A verified bundle shows the record is intact, signed by the expected key and whole up to the checkpoint. It cannot show the decisions in it were right; that is the premise gate's job. Even a matching checkpoint does not show every real-world action was recorded. Signing the bundle protects the Level 300 receipt only from packaging onward. And the statement should claim no assurance level: the worksheet's own statement says "I have not claimed any level for any system", and interpreting levels is left to module 500.6. Here there is a further limit: these are practice bundles, so a good run shows you can do the check, and nothing about any real system. You sign your own statement; no tool or agent signs it for you.

  7. The content step fails it: the worksheet's checklist says any payload content inside fails the bundle, and model outputs are payload even in a file the bundle is allowed to hold, and even when the producer signed it. A signature shows who packed a file, not that the file is harmless. And no, an ACCEPT does not finish the content check. The checker catches extra files, fields outside the fixed formats and lines shaped like a prompt, an output or a person's details, but, as the sandbox README says, it cannot prove that free text holds nothing sensitive. You finish the check by reading VERIFY.md, crosswalk.yaml, known-issues.md and the policy file yourself.
  8. Because a valid signature is not the only test a receipt must pass. The added field is outside the receipt's closed format, so step 2 refuses the receipt however it was signed. The producer holds the signing key and can sign anything, payload included. The format check, the checkpoint and your own read-through do jobs a signature cannot.

Next module: 400.R Engineering AI in your role (role tracks)