Level 500: Governance, safety and research · Module 500.7

Try each exercise first, then open its model answer.

OVERT and the SAIL crosswalk · Workbook

Three exercises, one for each objective. They use the course's synthetic bundle in this module's bundle/ folder: Card R7, Card T, Card E, the rule cards RC-1 to RC-4, Big Widget's crosswalk row for SAIL 3.13 and its editorial policy BW-ED-1. Big Widget Company and everything in its bundle are made up. Do not use real records, keys or contracts from your own work.

Run the bundle yourself. Follow bundle/VERIFY.md offline: Python 3.9 or later, nothing to install, no network. Write down what each step shows before you read the model answers. The model answers quote the bundle author's own run of 19 November 2026. That is not a check. If your output differs from the author's, say so plainly; your record is the one that counts. The bundle's keys are practice keys made from public labels, so a valid signature here proves nothing about who made the files. Where the sources this module opened do not settle something, the honest answer is "not checked", with an owner and a next check.

Attribution. Quotations from OVERT's standard in this workbook are short excerpts from OVERT — Observable Verification Evidence for Runtime Trust, Version 1.1 (June 2026), by Glacis Technologies, Inc., which OVERT's governance page designates for licensing under CC BY 4.0. The course excerpted them and regrouped them into its own exercises and answers.

Exercise 1 (500.7-a)

Three layers, a privacy check, and one claim that fits none of them. Big Widget's platform lead sends a status note to the board:

"Card R7 shows our AI controls are working. The gateway's signature checks out, the record proves TS-2 ran and blocked an over-limit order, which shows TS-2 keeps parts-buyer in scope, and our runtime follows the OVERT design principles. R7 holds only hashes, so we can post it publicly. Parts-buyer is safe to run."

  1. In bundle/, run python3 verify.py --all and record the results for case V. Then run python3 verify.py --guess privacy-demo and record what it shows.
  2. Split the note into separate claims. For each one, say which layer it belongs to: integrity and origin, attested control claim, independently tested effectiveness, or none of these. Say which claims are about something else, such as privacy.
  3. Give the result of each rule card (RC-1 to RC-4) for R7 in one line. Say which ones the checker decided and which ones need a person. For RC-4, open cases/V/evidence/EV-17.txt and judge it yourself.
  4. Which of the note's claims does R7 actually support, and under what assumptions?
  5. A colleague says: "Big Widget could just get OVERT conformance. Then we could say it is safe." What questions would you ask about any conformance claim, and what does OVERT's own text say about "safe"?
  6. Rewrite the note in four or five sentences so that every sentence is supported by the bundle.
Model answer
  1. Author's run (compare, do not copy). Case V: RC-1 pass; the rule file matches; EV-17 and EV-18 are present and are the files R7 names; RC-2 operator-attested; RC-3 needs the privacy owner's review, which is not in the bundle; RC-4 needs a person to read EV-17. The guess step tried 6,000 possible orders built from public catalogue values: the plain hash was matched after 1,653 guesses; R7's keyed commitment was not matched by any of them.
  2. The claims and their layers:
    • "The gateway's signature checks out": integrity and origin. The canonical bytes of R7's signed payload are unchanged since signing; under Card T's two assumptions (the key belongs to the gateway; it was not stolen or misused before R7 was signed), the gateway's key signed them.
    • "TS-2 ran and blocked an over-limit order": attested control claim. R7 says so, but R7 is signed only by Big Widget's own gateway key, so this is the operator's word, not an independent observation. "Proves" is too strong.
    • "TS-2 keeps parts-buyer in scope": effectiveness. Nothing in the bundle establishes it. One denied order shows one value limit at work once; it does not show that TS-2 stops out-of-scope calls in general, across the tools, connectors and routes parts-buyer can reach.
    • "Our runtime follows the OVERT design principles": a self-description. The rule cards are modelled on OVERT principles but are the course's own rules, and passing them is not OVERT conformance.
    • "R7 holds only hashes, so we can post it publicly": a privacy claim, judged by RC-3, and not supported (see 3).
    • "Parts-buyer is safe to run": none of these. A runtime record cannot support it by itself.
  3. Rule cards against R7:
    • RC-1 Integrity: pass, under Card T only. Decided by the checker.
    • RC-2 Who attests: operator-attested. The signer is Big Widget's own gateway key, with no independent notary. Labelled by the checker.
    • RC-3 Privacy: not passed. The order content is a keyed commitment, so guessing part numbers and prices does not reveal it, as the guess step shows. But the tool name, the time and EV-17's storage location are still readable, and RC-3 also needs a named privacy owner to have reviewed every such field and location as safe to show a stranger, with the referenced documents behind access controls. That review is not in the bundle. A person decides this card, not the checker.
    • RC-4 Effectiveness: not established. EV-17 is present, but it was run by the platform team that also runs the gateway, it tried only 100 over-limit orders on one tool, and no pass mark was set before the test. It did not try the other tools, the scope list, the connector grants or routes around the gateway. A person decides this card. (That is the author's reading; write your own.)
  4. R7 supports that its signed payload is unchanged and, if Card T holds, that the gateway's key signed it. It also supports that Big Widget asserts TS-2 version 3 ran and denied one call. It supports nothing beyond that.
  5. A good answer asks: what level and what scope? Who assessed the conformance? Who signed each piece of evidence? OVERT expresses conformance as "a composite claim combining a maturity level (1–4) and a scope designator". At Levels 1–2 an independent notary is not required, some evidence may be operator-attested (and must be disclosed as such), and "Level 1 and Level 2 conformance MAY be self-assessed by the operator". "Not required" is not "there is none": ask what this claimant actually used. On "safe", OVERT's text says a conformance claim "does not constitute a general representation that the system is secure, compliant, or safe." Its governance page makes clear, in our words, that using the OVERT name or marks is not a certification of any product. So conformance could not make the "safe" sentence true.
  6. For example: "Card R7's signature checks out with the gateway key in Card T, so the record's signed payload is unchanged and, if that key is ours and was not misused, our gateway signed it. The record says TS-2 version 3 denied one over-limit order; that is our own attested claim, not an independent one. Our only test, EV-17, was run by our own team on one limit, so TS-2's effectiveness is not established. R7 will not be shared outside until our privacy owner has reviewed its metadata and evidence locations. We make no claim that parts-buyer is safe."

Exercise 2 (500.7-b)

Trace SAIL 3.13, then find the gaps. Use Big Widget's crosswalk row for 3.13, Card R7, Card E and the bundle's cases V, M and X.

  1. Write the trace as four steps: risk, control claim, record, evidence. Describe risk 3.13 in your own words, and say which SAIL phase it sits in.
  2. In two sentences, say what the trace has shown and what it has not. How much of 3.13 does the order limit cover?
  3. List at least three pieces of evidence still missing before anyone could say TS-2 works. For the test, say what it must state before it is run and what it must report after.
  4. Name one risk that R7-style records cannot cover by design, and explain why no signature check could catch it.
  5. Run case X (python3 verify.py cases/X, or read its row from --all). This is Card E: an earlier check of R7 passed under Card T; since then R7's signed payload, its signature, the fetched key list and Card T have not changed; EV-18 is present and its hash matches; EV-17 returns "not found". A reviewer writes: "Signature invalid: evidence missing." Correct the finding.
  6. Now run case M, then run case X with the re-fetched key list: python3 verify.py --trust trust/card-T-refetched.json cases/X. For each, say what the seal check showed and why it differs from case X with Card T.
  7. A colleague adds a column mapping 3.13 to OWASP ASI02 and ASI03. What label should that column carry?
Model answer
  1. The trace:
    1. Risk: 3.13, in the Agentic Posture (Build) phase: a build-stage review of how agents are set up, not the act of building them. In our words: an agent's tool permissions and connectors are broader than its job needs.
    2. Control claim: TS-2 keeps parts-buyer within its scope list and order limit.
    3. Record: R7 says one call to supplier.order.create was denied under TS-2 version 3, because the order was above the value limit.
    4. Evidence: EV-17, the quarterly scope-test report, and EV-18, the connector permission export. In case X, EV-17 is unavailable and EV-18 is present with a matching hash.
  2. The trace shows that Big Widget claims a control for 3.13, and that one operator-attested record says the control denied one call for one reason, the value limit. It does not show that TS-2 works, that the scope list is right, that the other tools are limited, or that the connectors hold no wider grants: the order limit is one partial control for 3.13, not the whole of it, and the crosswalk row only says where evidence should be.
  3. Any three of:
    • The test itself, with its frame stated. Before testing: which population of calls it samples from (every tool, connector and route parts-buyer can reach, not just order values, with both cases that should be allowed and cases that should be denied), how attempts are chosen, and the largest bypass rate Big Widget will accept. After: how many attempts, by whom, how many got through, an uncertainty statement, and what it did not cover. OVERT's home page asks, in our words, that safety claims come with confidence intervals, sample sizes and methods an auditor can reproduce. No finite test proves there is no bypass; it gives evidence with declared limits.
    • Independence: a tester other than Big Widget's platform team. R7 is operator-attested, and EV-17 was run by the team that runs the gateway.
    • The real permissions: an independent comparison of EV-18, Big Widget's own export, with the permissions the supplier systems actually grant the agent's credentials. A matching hash only shows EV-18 is the file R7 names.
    • Coverage: independent evidence that every route parts-buyer has to a tool goes through the gateway. Today that is Big Widget's design, not a tested fact.
  4. A second route. If parts-buyer has a credential that reaches a supplier system directly, calls on that route never pass the gateway, so they produce no gateway record at all. A signature check can only test records that exist, and a missing record is not evidence that the call did not happen. Other evidence, such as the supplier's own logs, might show it indirectly; the gateway's records cannot. (This is the course's reasoning, not a statement in OVERT's text. It fits OVERT's governance page, which, in our words, says conformance does not establish that observation was complete.) A second by-design limit is also acceptable: an in-limit order is allowed and recorded as allowed even if it is the wrong part or fraudulent, because TS-2 is about scope, not about whether an order is right. OVERT's governance page, in our words, says conformance does not establish that the recorded facts are accurate.
  5. Author's run of case X: RC-1 pass, rule file matches, EV-17 unavailable, EV-18 present, RC-2 operator-attested, RC-4 not established. A corrected finding: "Signature check still passes, as it did before: R7's signed payload, signature, the fetched key list and Card T are unchanged, so RC-1 passes under Card T. That TS-2 ran and denied the call remains Big Widget's attested claim. EV-17 is unavailable, so TS-2's effectiveness is not established. This is missing proof, not an integrity failure; the governance lead must find or replace EV-17."
  6. Author's run: case M, RC-1 fails: one value in R7 was changed after signing, so the canonical bytes of the signed payload no longer match the signature and nothing in the record can be relied on. Case X with the re-fetched key list, RC-1 also fails, but R7's signed payload did not change: the key is no longer on the list, so a trust input to the check changed. That is not evidence of tampering. Both differ from case X with Card T because one input to the seal check differs: the signed payload in M, the key list in the re-fetched run. An earlier pass carries forward only while every input to the check stays the same.
  7. It should be labelled as this team's own judgment. ASI02 "Tool Misuse and Exploitation" and ASI03 "Identity and Privilege Abuse" are from the OWASP Top 10 for Agentic Applications for 2026, but placing them next to 3.13 is not a mapping published by OWASP or Pillar. Like every crosswalk row, it is a pointer, not proof that a control works.

Exercise 3 (500.7-c)

House rule, licence, or not checked? Big Widget's legal owner asks you to sort the statements below before the crosswalk is published. For each, write E (BW-ED-1, the crosswalk's editorial policy), L (a condition in the sail-skill LICENSE file at commit 67e1691, correctly stated), N (not checked: the sources opened do not settle it) or X (contradicted or misread by the licence's own text). Give a reason, and for each L, say which use the condition applies to.

  1. "This crosswalk reproduces no SAIL risk names, descriptions or controls."
  2. "If you share the Licensed Material, including in modified form, you must give the credit and notices the licence lists."
  3. "The licence grants reproducing and sharing the Licensed Material for NonCommercial purposes only, meaning purposes not primarily intended for or directed towards commercial advantage or monetary compensation."
  4. "ShareAlike applies to every change we make, even edits we keep inside our team."
  5. "Any use for which someone is paid is automatically commercial under this licence."
  6. "The licence in the sail-skill repository covers the SAIL web page."
  7. "Pillar has given Big Widget permission for free internal use of SAIL."
  8. "Patent rights in SAIL are licensed under CC BY-NC-SA 4.0."
  9. "This licence means we may not put Pillar's logo on any slide."
  10. "Because we cite by id only, we have complied with Pillar's licence."

Then write two sentences for the file that record what was checked, what was not, and who will find out.

Model answer
  1. E. It is part of BW-ED-1, a choice the crosswalk's authors made. It does not tell you what the licence allows or forbids.
  2. L. Section 3(a): "If You Share the Licensed Material (including in modified form), You must:" give the credit and notices the section lists. It applies when you share, and Share means "to provide material to the public".
  3. L. Section 2(a)(1) grants the right to "reproduce and Share the Licensed Material, in whole or in part, for NonCommercial purposes only", and section 1(k) defines NonCommercial as "not primarily intended for or directed towards commercial advantage or monetary compensation." It applies to the licensed uses. Whether a particular reuse is NonCommercial is a separate, fact-specific question this module does not decide: not checked.
  4. X. Section 3(b): "if You Share Adapted Material You produce, the following conditions also apply." Share is defined as providing material to the public. ShareAlike is triggered by sharing an adaptation, not by edits you keep to yourself.
  5. X. The definition turns on whether the use is "primarily intended for or directed towards" commercial advantage or monetary compensation. Payment can make a use commercial, but money changing hands somewhere does not settle the question by itself. Whether a given paid use is NonCommercial is not checked.
  6. N. The licence covers "Licensed Material", defined as the material "to which the Licensor applied this Public License." The file does not say which files or assets that is, and the SAIL web page carries "© 2026 Pillar. All rights reserved." Neither text alone settles it. Not checked.
  7. N. The licence file contains no separate or internal-use permission, and nothing else opened shows one. That does not prove none exists. Not checked; do not fill the gap with a guess.
  8. X. Section 2(b)(2): "Patent and trademark rights are not licensed under this Public License."
  9. N. The licence does not grant trademark rights, so it cannot be the permission for the logo. But that is all it says about them; it does not ban every use of a logo. Whether a particular logo use is allowed is not checked.
  10. N. This mixes the two texts. Following the house rule says nothing about the licence's conditions, and whether the licence even covers the material used is not checked. Treating the editorial rule as the complete licence is the mistake this exercise is about.

Two sentences for the file, for example: "We read the sail-skill LICENSE at commit 67e1691 (CC BY-NC-SA 4.0) and quote its sections 1(k), 2(a)(1), 2(b)(2), 3(a) and 3(b) and its definition of Share; our citing rule, BW-ED-1, is our own editorial choice and is not that licence. Whether the licence covers the SAIL web page, the framework download or other SAIL assets, whether any reuse or logo use is permitted, and whether Pillar has granted any separate permission are not checked; Big Widget's legal owner will find out before the crosswalk is published."

Next module: 500.8 Designing a governance regime