Level 500: Governance, safety and research · Module 500.5

A colleague with a blank checklist tests the latch of the garden door while another holds a closed laptop.
Will it hold?

Building safeguards that hold under attack · Technical gear

Reviewed and published, version 1.0.0.

Every research result is tied to the paper version cited.

Objectives. 500.5-a: design layered safeguards for a Big Widget Company agent with tool access, and name the attack each layer cannot stop. 500.5-b: explain why adversarial robustness is unsolved, using published transferable or automated attacks on aligned models.

Primary sources. Greshake et al. (2023, cited as arXiv v2);1 Zou et al. (2023, arXiv v2);2 Debenedetti et al. (2025, arXiv v2, "CaMeL");3 NIST AI 100-2 E2025;4 OWASP Top 10 for LLM Applications, 2025 edition;5 AiGovOps Foundation Manifesto, for the Foundation's gate terms.6 Publication status: a version of Greshake et al. appeared in the proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (AISec '23), published 26 Nov 2023; this gear quotes the arXiv v2 text. Zou et al. and Debenedetti et al. are cited as arXiv v2 preprints; whether later peer-reviewed versions exist is not checked. OWASP's 2026 edition is now current and renumbers its entries; this gear quotes the 2025 entries and has not checked the 2026 text.

Product names. The cited papers name commercial models and products. This gear leaves those names out until they are cleared for use in the course, and uses neutral labels. The papers' tables and sections cited below name them, so every figure stays auditable.

1. Threat model in NIST's terms

The attack it cannot stop; Instructions: Any injection the model happens to follow anyway. It is not a security control; Input and output filtering: Hidden, chained or encoded injections; new phrasings it has not seen; Least privilege: Misuse of a tool it legitimately has; Separate the plan from the data: Text-to-text attacks that change no data flow, such as a summary that lies or a phishing link in a reply. Anything its written policies allow; Authorisation outside the model: Anything inside the allowed limits: a real order, from an approved supplier, under the limit, that the buyer did not want; Human approval for high-risk actions: A persuasive request the approver accepts. Anything outside the hold rules; Rate limits: Any harmful action inside the quota; Logging and monitoring: Anything, at the moment it happens. Logs find and explain; they do not stop.
The lesson's 'The attack it cannot stop' cells; uncited cells are labelled course, and no layer is presented as a guaranteed stop (s3, s4, s5).
Figure data as text

Safeguard layers

The attack it cannot stop

PartWordingSources
InstructionsAny injection the model happens to follow anyway. It is not a security control.[s5]
Input and output filteringHidden, chained or encoded injections; new phrasings it has not seen.[s4]
Least privilegeMisuse of a tool it legitimately has.course
Separate the plan from the dataText-to-text attacks that change no data flow, such as a summary that lies or a phishing link in a reply. Anything its written policies allow.[s3]
Authorisation outside the modelAnything inside the allowed limits: a real order, from an approved supplier, under the limit, that the buyer did not want.course
Human approval for high-risk actionsA persuasive request the approver accepts. Anything outside the hold rules.course
Rate limitsAny harmful action inside the quota.course
Logging and monitoringAnything, at the moment it happens. Logs find and explain; they do not stop.course

The lesson's 'The attack it cannot stop' cells; uncited cells are labelled course, and no layer is presented as a guaranteed stop (s3, s4, s5).

NIST AI 100-2 E2025 classifies attacks on generative AI by attacker objective, attacker capability and attacker knowledge.4

Target system (fictional). Big Widget Company's supplier desk agent. Principal: a buyer u. Tools: read_mailbox(u), read_attachment, lookup_orders, summarise, send_email(to, body), create_po(supplier, amount). Untrusted inputs: every email body and attachment from any sender. The relevant adversary holds resource control (they can send a quote) and, against a commercial model, typically black-box knowledge.

Why this is a distinct threat: such applications "blur the line between data and instructions",1 because generative models "combine the data and instruction channels".4 Greshake et al. framed the attack as remote exploitation "by strategically injecting prompts into data likely to be retrieved",1 and NIST stresses that the attacker is a third party while "in many cases, it is the primary user of the model who is harmed".4 For agents specifically, tool use lets attackers "execute arbitrary code or exfiltrate data"; NIST calls agent security research "still in its early stages".4

Evidence status of Greshake et al.: demonstrations used synthetic applications built on a commercial model whose tools returned prepared content, local HTML files opened in a commercial chat-search sidebar, and a commercial code-completion assistant; the authors avoided public, search-indexed injections and state that they "believe that, in principle, the attacks are feasible for in-the-wild retrieved injections as well".1 Treat the live-retrieval case as argued and plausible, not measured, in that paper.

2. Heuristic defences, and why they are not boundaries

Misconception: a strong system prompt or an input filter stops prompt injection.

NIST's mitigation list for indirect injection has three families: training (task-specific fine-tuning; training models to follow hierarchical trust relationships), detection, and input processing (filtering instructions out of third-party data, spotlighting, instructing models to disregard instructions in untrusted data). It then states: "current mitigations do not offer full protection against all attacker techniques".4 Separately, it lists ways to hide or obfuscate injections: non-visible parts of a resource, multi-stage injections and Base64-encoded commands.4 It does not tie these techniques to any particular filter.

OWASP LLM07:2025 is explicit that the system prompt "should not be considered a secret, nor should it be used as a security control", and that security controls be "enforced independently from the LLM".5 OWASP LLM01:2025: "it is unclear if there are fool-proof methods of prevention for prompt injection".5 CaMeL's authors characterise the heuristic class: "none of these heuristic defenses provide any guarantee of security".3 NIST's design advice is permissive, not mandatory: designers "may design systems with the assumption that prompt injection attacks are possible if a model is exposed to untrusted input sources".4 The Foundation's rule for the action side is "No direct model-to-tool path for consequential actions."6

Formally, call a defence heuristic if its block condition is a property of model or classifier behaviour on the input, and structural if its block condition is a property of the system's information flow or permissions that the input cannot change. The rest of this gear designs with that distinction. (The terms are this course's; the classification follows NIST's empirical-versus-provable framing in section 5.)4

3. Layered design and residual attack per layer

Outcome vocabulary. For an attack a, a layer's effect is one of: deny (the action cannot execute, and no approval inside the agent workflow overrides it); hold (execution waits for a named approver's decision on the exact action; it executes if approved, so a hold bounds who decides, not what is decided); or no guarantee (the layer may detect, slow or shrink the action, but its block condition is not guaranteed true). The Foundation's gate answers are "go ahead · only this far · wait for a person · no".6 The course maps them as: deny = "no"; hold = "wait for a person"; a hard limit or quota that admits an action only inside bounds = "only this far" (it constrains, it does not test intent). An action that clears every automated layer receives only a runtime permit under policies authorized earlier; it is not a named person's "go ahead" for that action. No layer tested whether the action was wanted, so an unintended action can execute with no human approval of it. The mapping is this course's, not the Foundation's.

Hard limits and exception authority (fictional design). The ordering system's supplier allow-list and amount limit are hard limits: a held approval cannot lift them. Changing either is a separate change made by the named procurement owner in the ordering system, outside the agent's tool surface. An approval binds one action: recipient, content and amount as shown to the approver.

#LayerTypeMechanismResidual attack (what it cannot stop)
1Instructions, delimiting, spotlightingheuristicSystem prompt; mark untrusted spans.54Any injection the model follows anyway; not a security control.5
2Input/output filtering and detectionheuristicClassifiers or rules on inputs and outputs.54No guarantee against obfuscated, multi-stage or novel injections: NIST lists the first two as hiding techniques and includes filtering among mitigations that "do not offer full protection against all attacker techniques";4 optimisation-found inputs (section 5).2
3Least privilegestructuralMinimise extensions, extension functionality and extension permissions; execute in the user's context.5Abuse of an authorised tool within its scope (e.g. a harmful but permitted send_email to an existing supplier).
4Control/data-flow separation with capabilitiesstructuralCaMeL: a privileged LLM (P-LLM) writes the plan as restricted Python from the trusted query; a quarantined LLM (Q-LLM) without tools parses untrusted data; a custom interpreter keeps a data-flow graph, tags values with capabilities (sources and allowed readers), and checks policies "before each tool call".3Text-to-text attacks "which have no consequences on the data flow" (including injection-induced phishing); side channels, which the authors report;3 and any flow the written policies permit, since security "will only be as strong as the capabilities the system supports, and the policies that the system implements and enforces".3 (The course's example of the last: Q-LLM-extracted values steering the parameters of a planned, policy-permitted action.)
5Complete mediation in downstream systems; output handlingstructural"Implement authorization in downstream systems rather than relying on an LLM"; treat model output with zero trust and validate it before backend functions.5 Here: create_po denied unless supplier ∈ allow-list and amount < limit, for principal u.Actions that satisfy the downstream policy but not the principal's intent.
6Human approvalprocedural (hold)Approval before high-risk actions run.5 Here: hold on send_email to any address not previously used by u, and on create_po above a review amount set below the hard limit.Persuasive requests; approval fatigue ("users become desensitized to security prompts");3 anything outside the hold predicate.
7Rate limitingstructural (quota)create_po denied once u's hourly quota is used.5Any harmful action while quota remains. A quota bounds frequency, not intent: it denies the (k+1)th call regardless of content, and nothing before it.
8Logging and monitoringdetectiveLog every tool call; a named reviewer reads the log. OWASP: logging and rate-limiting "will not prevent" excessive agency "but can limit the level of damage".5Any attack at execution time; it supports detection and recovery only.

Cross-cutting: adversarial testing that treats the model "as an untrusted user";5 and evaluation against "strong adaptive attacks", since the adversarial ML field "has a history of publishing defenses evaluated under relatively weak adversarial models that are subsequently broken by more powerful attacks" (NIST's statement in its predictive-AI evasion section).4 Testing measures; remediation is a separate design change; a passing re-test bounds only the attacks tried.

On the CaMeL guarantee. The paper models security as a game (PI-SEC) in which policies govern allowed actions "independently of the prompt", are checked before each tool call, and halt execution on failure; the adversary wins by supplying an initial tool state that causes an action outside what the trusted prompt permits.3 The guarantee is therefore about control and data flow under the policies written, not about the truth of generated text or the fitness of values inside an allowed flow. It also presumes complete mediation: an action reachable by a path outside the interpreter is outside the guarantee (the course's reading of the threat model). The authors list the cost: users "needing to codify and specify security policies and maintain them".3

Receipts. A layer-6 decision is a gate decision. The Foundation's gate "leaves a signed receipt that holds no private content".6 In this course's application, the receipt carries metadata (decision, rule, time), not the quote, competing prices or email body; those belong in the controlled layer-8 work record. A receipt evidences that a named approver decided under a rule. It is not evidence the decision was correct.

4. Exercise pattern: first denying or holding layer

Procedure: for an attack a and an ordered design L1..Ln, return the smallest i such that Li is structural or procedural and its deny or hold condition is true for a under stated assumptions, and label the result deny or hold (approver: …); otherwise return "none". Heuristic layers never count as the answer, because their block condition is not guaranteed.43 A hold is reported as a hold, not a block. This procedure is this course's; it applies the sources above to a fictional design.

5. Why adversarial robustness is unsolved

Misconception: patching the jailbreak that worked fixes the class of attack.

5.1 What the attack is, at the level needed here

Zou et al. append an adversarial suffix to a harmful request and optimise the suffix so that the model begins its answer affirmatively.2 The method "automatically produces these adversarial suffixes by a combination of greedy and gradient-based search techniques", rather than relying on manual prompt engineering, and a single "universal" suffix is optimised across several prompts and, in some runs, several models.2 This gear does not reproduce the optimisation procedure. What matters for design is that an automated search, not a person's ingenuity, produces the attack, so the space of working inputs is searched, not guessed.

5.2 Transfer, measured

The universal suffixes were optimised white-box on two open models (and, in a further run, two more open models), then applied black-box to five closed commercial models, labelled A to E here.2 Table 2 of v2 names them. Attack success rate, %, averaged over 388 behaviours:2

ConditionABCDE
Behaviour only1.88.00.00.00.0
+ suffix (first open-model set)34.334.52.60.031.7
+ suffix (both open-model sets)47.429.137.61.836.1
+ Concatenate79.624.238.41.314.4
+ Ensemble86.646.947.92.166.0

All five were 2023 snapshots, identified by version in the paper.2 What this supports: suffixes optimised on open models transferred, at very different rates, to closed models in 2023; on the ensemble row the rate ranged from 2.1% to 86.6%, and the authors say model D "appears to be more robust" than the other commercial models.2 What it does not support: any rate for a model as of November 2026, a claim that every model is equally exposed, or a claim that the same search would succeed against a current model. Note also that the paper's introduction ("up to 84%") and overview (87.9%, 53.6%) give figures that do not match Table 2.2 Cite the table and the row; state the denominator.

NIST makes the general point: while technical restrictions differ between models, "the techniques for circumventing such defenses are often common between different kinds of models".4

5.3 Why patching is not closure

The authors disclosed results to the affected companies before publication, expected that "the examples shown in the paper will likely cease to function", and wrote that "it remains unclear how the underlying challenge posed by our attack can be adequately addressed".2 A patch that defeats the published strings is evidence about those strings. It is not evidence the class is closed: the optimiser is not removed, and whether it finds new working suffixes against the patched model is an empirical question the patch does not answer. Their analogy is computer vision, where adversarial examples remain "an omnipresent phenomenon" and detection-based defences "have universally failed".2

NIST (section 4.2.5): many AML mitigations "are empirical in nature and lack theoretical or provable guarantees"; research results point to theoretical limits, including the impossibility of model-based detection preventing all impermissible outputs, and the finding that, so long as a model has any probability of an undesired behaviour, some prompt can trigger it, so alignment that "attenuates but does not remove" a behaviour "will remain vulnerable to adversarial prompting attacks".4 NIST keeps pre-deployment adversarial testing because it "can potentially foreclose many attack vectors" and raise attacker effort, while noting that organisations "may need to consider practices and measures beyond adversarial testing".4 Testing identifies; remediation changes the system; neither alone establishes robustness against attacks not yet tried.

6. Structural defence results, read critically

CaMeL "creates a protective system layer around the LLM, securing it even when underlying models are susceptible to attacks"; "the untrusted data retrieved by the LLM can never impact the program flow".3 Keep three results apart:

  1. Utility. On AgentDojo, the v2 abstract (24 Jun 2025) reports 77% of tasks solved "with provable security" against 84% for an undefended system; v1 (24 Mar 2025) reported 67%.3 These are task completion rates, not attack counts.
  2. One security comparison. In the authors' baseline comparison, run with a single model, successful attacks against CaMeL numbered 0, against 8 for the next best defence (a tool filter).3
  3. The main runs across newer models. The authors write that "the number of successful attacks is not zero", and attribute the remaining successes to attacks outside CaMeL's threat model.3

"Provable security" is scoped to the PI-SEC game, the implemented capabilities and the policies written.3 Limits, in the authors' words: "Our approach is not perfect and does not completely address every potential attack vector";3 text-to-text attacks and phishing are non-goals;3 "CaMeL is vulnerable to side-channel attacks";3 and "CaMeL does not aim to make a fully autonomous system without any need for human intervention."3 One benchmark result is not evidence about a different deployment, tool set or policy.

Connecting the objectives: because 500.5-b says model-level robustness has no guarantee,42 the design in 500.5-a should put each consequential action behind at least one layer that denies it or holds it for a named approver, whose condition the attacker's text cannot change, and state "none" where no such layer exists.35 That is the Foundation's "No direct model-to-tool path for consequential actions."6

7. Three yeses

The Foundation's four gates (Pre-pend, Policy, Operate, Audit) fall into three movements: Get to Yes "before a single action runs" and "before a rule or a system ships", Stay at Yes "while it runs, in real time", Recover to Yes "after — on demand, on incident".6

Further learning (optional, not part of this lesson, not copied here): UC Berkeley CS294/194-280, Advanced Large Language Model Agents (Spring 2025): <https://rdi.berkeley.edu/adv-llm-agents/sp25>. MIT 6.858, Computer Systems Security (Fall 2014), for capabilities and information flow: <https://ocw.mit.edu/courses/6-858-computer-systems-security-fall-2014/>. Stanford CS120, Introduction to AI Safety (Fall 2025): <https://web.stanford.edu/class/cs120/>.

Sources

  1. s1 Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. Paper. Checked 2026-11-19 by source-checker (Claude Opus 5.5, round-2 re-check session; not the writer or item writer); arXiv v2 PDF read in full and Crossref record opened, 9/9 uses supported. ↩ ↩ ↩ ↩
  2. s2 Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson (2023). Universal and Transferable Adversarial Attacks on Aligned Language Models. Paper. Checked 2026-11-19 by source-checker (Claude Opus 5.5, round-2 re-check session; not the writer or item writer); arXiv v2 PDF read in full, 29/29 uses supported. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s3 Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, and Florian Tramèr (2025). Defeating Prompt Injections by Design. Paper. Checked 2026-11-19 by source-checker (Claude Opus 5.5, round-3 re-check session; not the writer, item writer or an earlier checker); arXiv v2 PDF read in full and v1 abstract opened, 56/56 uses supported. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  4. s4 Apostol Vassilev, Alina Oprea, Alie Fordyce, Hyrum Anderson, Xander Davies, and Maia Hamin (2025). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2 E2025). Standard. Checked 2026-11-20 by source-checker (Claude Opus 5.5, round-6 re-check session; not the writer, item writer or an earlier checker); NIST PDF reopened, pp. 52-53 read by hand, the 3 round-6 gear citations (plain.md:88-90, standard.md:120) supported; other s4 uses unchanged since their round-2/3/5 supports verdicts. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  5. s5 OWASP GenAI Security Project (2025). OWASP Top 10 for LLM Applications 2025. Official doc. Checked 2026-11-20 by source-checker (Claude Opus 5.5, round-6 re-check session; not the writer, item writer or an earlier checker); LLM01:2025 entry page reopened, the round-6 use at plain.md:88 supported; LLM05-07 pages not reopened, their uses unchanged since the round-2 check (LLM01, LLM05, LLM06, LLM07 and README read, 44/44 supported). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  6. s6 AiGovOps Foundation (2026). Manifesto. Official doc. Checked 2026-11-20 by source-checker (Claude Opus 5.5, round-4 re-check session; not the writer, item writer or an earlier checker); live MANIFESTO.html page read, 16/16 uses supported, including the round-4 four-gates/three-movements wording. ↩ ↩ ↩ ↩ ↩ ↩ ↩

Next: the quiz