
Building an agent with limits · Technical gear
Reviewed and published, version 1.0.0. As of 20 Nov 2026. It is not a credential. Big Widget Company, its people, its agent, the policy format and every trace here are synthetic, written for this lesson; they are not any product's format. The course sandbox (gate, recording stubs, fixtures) runs offline in
apps/sandbox/; §3.4 gives the commands and the evidence they produce.
You will be able to:
- specify an authority table as a default-deny policy over (identity, tool, arguments), enforce it with complete mediation outside the model, and show from a dry-run trace that a blocked proposal was refused with no side effect (300.4-a);
- state a decision rule for out-of-band confirmation in terms of channels the requester controls, and apply it to the Hong Kong deepfake video-conference fraud (300.4-b);
- justify a distinct workload identity for the agent with delegated, least-privilege access, and design an audit record that carries that identity and a non-secret credential reference or authorisation event, never authenticator content (300.4-c).
Thesis: an agent's authority is the intersection of three things you control: the policy at the gate, the permissions of the agent's own identity in each downstream system, and the scope a person delegates for a task. Prompts are not in that list. Evidence that the bound holds is a trace with refusals in it.
This builds on 300.3 (the model proposes; the system decides) and is attacked in 300.6 (injection steers proposals).
1. Threat model: Excessive Agency
OWASP's LLM06:2025 Excessive Agency names three root causes, "excessive functionality", "excessive permissions" and "excessive autonomy".1 Mapped to controls:
| Root cause | OWASP's example | Control in this module |
|---|---|---|
| Excessive functionality | a document-reading extension that "also includes the ability to modify and delete documents"1 | Connect only the tools the job needs ("only the minimum necessary"1); a tool not in the policy is denied |
| Excessive permissions | an extension meant to read data whose database identity "not only has SELECT permissions, but also UPDATE, INSERT and DELETE permissions"1 | A dedicated agent identity with least privilege in each downstream system (§5) |
| Excessive autonomy | an extension that "fails to independently verify and approve high-impact actions"1 | needs_approval rows, with out-of-band confirmation where §4 requires it |
The principle under all three is least privilege. NIST SP 800-53 Rev. 5, AC-6: "Employ the principle of least privilege, allowing only authorized accesses for users (or processes acting on behalf of users) that are necessary to accomplish assigned organizational tasks."2 The NCSC-led Guidelines for secure AI system development (November 2023) apply it to AI: where AI components "trigger actions, for example amending files or directing output to external systems, you apply appropriate restrictions to the possible actions (this includes external AI and non-AI fail-safes if necessary)", and "you apply least privilege principles to limit access to a system's functionality".3
The misconception "give it admin rights so it doesn't get stuck" optimises for task completion and removes every bound that does not depend on the model behaving. A permission error is a fail-closed outcome; with admin rights the failure mode becomes fail-open. This course inverts the default: the agent does only what it has explicitly been allowed to do, and an action it cannot undo is either needs_approval with a named approver or deny. There is no direct model-to-tool path for a consequential action.
2. The authority table as a policy
2.1 Semantics
Model the gate as a function evaluated on every proposed call:
decide(agent_id, principal, tool, args, context) -> ALLOW | NEEDS_APPROVAL(approver) | DENY
In the course's plain words these are go ahead, wait for a person and no; an ALLOW whose conditions cap the arguments (amount, supplier) is "only this far". The course's verbs are allow, constrain, hold and deny.
where principal is the person the agent acts for (§5). The course's semantics (a design choice, not a standard):
- Default deny. If no rule matches,
DENY. - Deny overrides. If several rules match, the most restrictive wins:
DENYoverNEEDS_APPROVALoverALLOW. Order in the file does not matter, so adding a permissive rule cannot silently shadow a prohibition. - Conditions are on arguments, not on the tool name alone: amount, counterparty, destination.
- Every call is mediated. OWASP's complete mediation: "all requests made to downstream systems via extensions are validated against security policies", with authorisation implemented "in downstream systems rather than relying on an LLM to decide if an action is allowed or not".1 That includes the tenth call in a loop, and calls whose arguments came from tool results.
2.2 Big Widget's parts-buyer policy (synthetic)
agent: bw-parts-buyer
default: deny
combine: deny-overrides
rules:
- { tool: search_catalogue, decision: allow }
- { tool: draft_purchase_order, decision: allow }
- tool: submit_purchase_order
decision: allow
when: { supplier_on_approved_list: true, total_usd: { lte: 5000 } }
- tool: submit_purchase_order
decision: needs_approval
when: { any: [ { total_usd: { gt: 5000 } }, { supplier_used_this_year: false } ] }
approver: priya.procurement-lead # a named person; role reviewed 2027-03-01
out_of_band: callback_to_number_on_file # §4
- { tool: send_drawing, decision: needs_approval, approver: priya.procurement-lead }
- { tool: add_supplier, decision: deny }
- { tool: change_supplier_bank_details, decision: deny }
- { tool: pay_invoice, decision: deny }
Note the two submit_purchase_order rules. An order over $5,000, or to a supplier not used this year, matches the second rule and waits for Priya; under deny-overrides that wins even if the first rule also matches. An order of $5,000 or less to a supplier that is not on the approved list but was used this year matches neither rule, so default deny refuses it. Is that what the table meant? Writing conditions so that every case lands where you intended is the work; the dry run checks it.
2.3 Where NEEDS_APPROVAL is enforced
OWASP: "require a human to approve high-impact actions before they are taken", which "may be implemented in a downstream system (outside the scope of the LLM application) or within the LLM extension itself".1 Implement the approval as a record the gate checks, bound to the exact request:
approval { id, approver_id, agent_id, principal, tool, args_digest, expires_at, oob_ref? }
args_digest binds the approval to these arguments, so an approval for a $6,000 order to S-0042 cannot be replayed for a $60,000 order to S-0099. expires_at makes an approval fail closed when it lapses (the "must stop" scenario in the hold prompts). A NEEDS_APPROVAL that nobody answers before its deadline resolves to DENY; a hold never decays into a standing allow. These fields and rules are this course's design.
2.4 Defence in depth: the downstream permission
The gate is one enforcement point. The second is the agent identity's permission in each system. OWASP: minimise extension permissions, "enforced by applying appropriate database permissions for the identity that the LLM extension uses to connect to the database".1 If the purchasing system never granted bw-parts-buyer write access to supplier bank details, a bug in the gate does not become a payment redirect. That requires the identity to be the agent's own (§5).
3. Testing the bound: dry runs and negative tests
A dry run replaces each tool with a recording stub and drives the agent through scripted scenarios on synthetic data. A test suite for the policy needs, at minimum (course rules):
- for every
denyrow, a scenario in which the model proposes that call; - for every
needs_approvalrow, a scenario with no approval, one with an expired approval, and one with a valid approval for different arguments; - one proposal of a tool not in the policy, to show
default: denyis configured, not just written in the table; - for each
denyrow, isolation runs against fixture copies of the policy, because with both protections in place a single refusal cannot show that either works alone:- default set to allow: expected
DENYfrom the explicit rule; - explicit rule removed, default restored: derive the expected decision from the rules that still match. If none matches, default deny must return
DENY. If another rule matches, that rule decides and the default is not exercised; expectingDENYthere tests an assumption, not the evaluator. In §2.2 no other rule matchesadd_supplier,change_supplier_bank_detailsorpay_invoice, so all three expectDENY; - an overlapping
allowadded for the same call, explicit rule kept: expectedDENY, which tests deny-overrides directly;
- default set to allow: expected
- if you require every prohibition to survive deletion of its own rule, state that as a separate policy constraint (no other rule may match a prohibited call) and test it as one. Default deny with deny-overrides does not imply it;
- an expected decision for every scenario, written before the run;
- assertions on two things: the gate's decision in the trace against the expected decision, and zero side effects recorded by the stub for refused calls.
3.1 Expected versus actual
Outcome labels alone cannot be scored. Pair each proposal with its expected decision:
| Expected | Actual | Reading |
|---|---|---|
DENY | DENY, stub side effects 0 | the bound held for this case |
DENY | DENY, stub side effects > 0 | the gate reported a refusal but the call ran: a mediation bypass |
DENY | ALLOW | failed negative test: the bound is broken |
no scenario expects DENY | anything | missing negative coverage: the run says nothing about the bound |
The last two rows produce identical-looking traces when every actual is ALLOW. Ten ALLOW lines may be ten permitted reads (missing coverage), or may include an unlisted export the gate wrongly allowed (a failed negative test). Only the expected column separates them.
3.2 Traces
Synthetic trace, with an injected instruction in an email the agent reads:
ts=10:02:19 agent=bw-parts-buyer principal=luis.m tool=change_supplier_bank_details
args_digest=9f3c… matched=[deny] decision=DENY returned_to_model="not run: denied by policy"
stub_side_effects=0 alert=raised
ts=10:02:20 agent=bw-parts-buyer principal=luis.m tool=submit_purchase_order
args={supplier:S-0042,total_usd:3180} matched=[allow(approved_list,lte5000)] decision=ALLOW
cred_ref=po-api/bw-parts-buyer#key-3 stub_side_effects=1
The first line is the evidence 300.4-a asks for: a proposed call, the rule that matched, DENY, what the model was told, and no side effect. A trace with only ALLOW lines tells you nothing about the bound.
A failing trace from a misconfiguration:
ts=11:40:03 agent=bw-parts-buyer principal=luis.m tool=export_supplier_list matched=[] decision=ALLOW
The expected decision was DENY (unlisted tool). matched=[] with ALLOW means the effective default is allow: the table and the running system disagree. The trace does not say which fault: default: deny missing, mis-keyed, or present but not applied by the evaluator. Inspect rule presence, matching and default handling rather than infer the exact missing line; fix, re-run, and keep the re-run's trace as the evidence. This trace is written to fail. The sandbox's shipped dry-run scenario (§3.4) proposes the same unlisted export_supplier_list and its gate returns DENY with zero stub side effects; a generated trace will not reproduce this fault unless you introduce it in a fixture copy.
Traces are not controls. OWASP: logging, monitoring and rate limiting "will not prevent Excessive Agency, but can limit the level of damage caused".1
3.3 Which trace is evidence
The traces in §3.2 were written for this lesson. They teach how to read the evidence; they are not evidence that any configuration, yours included, behaves this way. A trace counts for 300.4-a only when the gate produced it from the learner's policy, with the stub's side-effect counts attached. §3.4 is how you produce one.
3.4 Running the course sandbox
The course sandbox is an offline, standard-library Python 3 runner: apps/sandbox/sandbox.py, with instructions in apps/sandbox/README.md, section "300.4 · Building a bounded agent". Its policies, scripted proposals, approval files and in-memory recording tools are synthetic fixtures owned by this repository. Its example agent is the workbook's bw-stock-keeper acting for kemal.s, not the parts buyer in §2.2. From the repo root:
python3 apps/sandbox/sandbox.py agent dry-run --policy YOUR_POLICY.json
python3 apps/sandbox/sandbox.py agent hold --policy YOUR_POLICY.json
python3 apps/sandbox/sandbox.py agent dry-run --policy YOUR_POLICY.json --step-budget 1
python3 apps/sandbox/sandbox.py agent stop --policy YOUR_POLICY.json
Omit --policy to run the shipped apps/sandbox/agent-policy.json; add --scenarios YOUR_SCENARIOS.json (a copy of agent-scenarios.json) for your own proposals and expected decisions, and --approval YOUR_APPROVAL.json for the approval tests. --trace-dir DIRECTORY, placed before agent, sets where traces go; by default they are written under apps/sandbox/dist/traces/.
Mapping from the YAML in this gear to the sandbox's JSON:
- Same keys:
agent,default,combine,rules, and per ruletool,decision,when,approver. The sandbox's conditions usesupplier_on_approved_list,total_usd_maxandtotal_usd_over; the{ lte: ... }/{ gt: ... }forms in §2.2 are notation, not its input syntax. "combine": "deny-overrides"is required. Any other value, or none, is an input error (exit code 2) before any proposal is evaluated. That is a rejected policy, not a passing run.defaultmust bedenyorallow; theallowvalue exists only for the isolation copy in §3.
What to read in each trace event, against the suite in §3:
| Field | Use |
|---|---|
expected vs decision | the expected-versus-actual pairing of §3.1 |
gate_decision | the gate's own result; score isolation runs on this, because a tool with no downstream permission is refused afterwards (decision=DENY, "not run: no downstream tool permission") even when the gate said ALLOW |
matched, rule | which rules matched; matched=[] with rule=default is the fallback |
stub_side_effects and the top-level stub_calls | zero effects for every refusal, and no refused call in stub_calls |
agent, on_behalf_of, args_digest | the identity and binding fields used in §5 and §6 |
The hold scenario stops at the over-limit reorder. An approval replay is a fresh dry run with a synthetic approval record bound to approver, agent, principal, tool and exact arguments, with an ISO 8601 expires_at; the README gives the valid, expired and wrong-argument variants. A run whose status is completed is not a pass by itself; the expected column and the stub ledger decide.
Scope: the sandbox demonstrates gate decisions on scripted proposals. It is not a real model, real access control, authentication or credential handling, and it does not provision identities. Do not substitute a real workplace system, real logs or real credentials for it.
4. Out-of-band confirmation
4.1 The case
Figure data as text
Confirm outside the request
- PAUSE · course rule · Does any of these apply? — money or value, above a set limit, and hard to undo; changes where money or data goes; urgent, secret or skips the usual process; arrived by email, chat, call or shared doc.
- CONFIRM · course rule · Use an independent channel — call a number already on file, never one supplied in the request.
- ENFORCE · course rule · Record the person’s confirmation — the system will not run the action until the approval records it.
The course’s out-of-band procedure builds on police callback advice (s1) and OWASP human approval guidance (s2).
Objective: 300.4-b. Text equivalent of figures/confirm-outside-the-request.svg.
The Hong Kong Government's written reply to Legislative Council question LCQ9 (26 June 2024): "In end-January 2024, the Police received one fraud case involving the use of deepfake technology to fabricate a pre-recorded video conference."4 "As instructed, the informant eventually gave authorisation to transfer funds to five local bank accounts and suffered a loss of about HK$200 million."4 "Since the meeting was pre-recorded, there was actually no interaction between the informant and the fraudster."4
Scope of the evidence: one government record; it does not name the organisation or describe its controls.4 This course does not name the organisation, and the analysis below is a proposal, not a finding about the case.
The police advice in the same reply mixes in-band tests (ask the person to move "in front of the camera"; use "the content of the conversation to test the authenticity of the other party") with one out-of-band step, "verifying with a phone call" when a relative or friend asks for money by video or audio.4
4.2 The channel argument
Let C_r be the set of channels the requester controls or can write to: the meeting they convened, the email thread they started, the number they supplied, any document an outsider can edit. A face, a voice, or a correct answer to a challenge delivered on a channel in C_r authenticates the channel's content, not the requester. In this case a live challenge might have exposed a recording, since there was no interaction;4 but a check on the requester's own channel is still only as strong as the requester's ability to fake it.
Out-of-band confirmation (course definition): confirmation on a channel c ∉ C_r, established before the request (directory number, number on file, the system of record), that binds to the specific transaction (payee, amount, destination).
Decision rule (course rule). Require it when any holds:
- the action transfers value above a threshold and is not readily reversible;
- the action changes a destination: payee, bank details, delivery address, access grant;
- the request carries urgency or secrecy, or bypasses the normal process;
- the request entered through an input an outsider can write to.
4.3 Agents and out-of-band confirmation
- The agent's inputs are all in-band. Every request the agent sees arrives as text on a channel someone else may control. That is why
change_supplier_bank_detailsisdenyfor the agent, notneeds_approval: rule 2 applies to every instance. - The agent must not perform the confirmation. If the agent places the "callback", it chooses the number from its context, which an attacker can write to. OWASP requires human approval for high-impact actions and allows it to be implemented in a downstream system.1 The course's design on top of that: store the out-of-band reference (
oob_ref) in the approval record, created by the named approver on a channel outside C_r, and have the gate require it. - A related pattern in the NCSC guidelines, for data sent to services outside the organisation: "requiring users to log in and confirm before sending potentially sensitive information".3
5. Identity: the agent is a principal, not a borrowed session
NIST SP 800-53 Rev. 5:
- IA-2: "Uniquely identify and authenticate organizational users and associate that unique identification with processes acting on behalf of those users."2
- IA-9: "Uniquely identify and authenticate [Assignment: organization-defined system services and applications] before establishing communications with devices, users, or other services or applications."2 The assignment bracket means the organisation chooses which services and applications; IA-9 does not by itself require every agent to have an identity. This course's recommendation is to designate it.
- AC-6 discussion: "Organizations consider the creation of additional processes, roles, and accounts as necessary to achieve least privilege."2
Running the agent under a person's shared login violates each property you need:
| Property | Own identity | Person's login |
|---|---|---|
| Least privilege (AC-6) | Grant only the agent's job | Inherits the person's whole entitlement |
| Attribution (IA-2, AU-3) | Agent and principal distinguishable | j.smith could be either |
| Revocation | Revoke the agent; the person keeps working | Revoking the agent locks out the person |
| Downstream enforcement (§2.4) | Systems can grant the agent narrower rights | Systems see only the person |
Delegation. OWASP's mitigation "execute extensions in user's context": "actions taken on behalf of a user are executed on downstream systems in the context of that specific user, and with the minimum privileges necessary"; its example authenticates the user "via OAuth and with the minimum scope required".1 Read with IA-2, the design is: the agent authenticates as itself, carries a minimum-scope grant from the principal for the task, and the effective authority is the intersection of the gate policy, the agent's own permissions and the delegated scope. The intersection is this course's framing; its point is that the agent can never do more than any one of the three allows.
6. The audit record
Required content. NIST AU-3: audit records establish (a) the type of event, (b) when, (c) where, (d) the source, (e) the outcome, and (f) the "Identity of any individuals, subjects, or objects/entities associated with the event"; the discussion names "user or process identifiers".2 OWASP's Logging Cheat Sheet lists "Who (human or machine user)" and "User identity (if authenticated or otherwise known)" as event attributes, and "Authentication successes and failures" and "Authorization (access control) failures" as events to log.5 It also lists "Use of default or shared accounts" as an event to log,5 so a shared login is a finding, not a convenience.
A record for one gate decision (course schema):
| Field | AU-3 item | Example |
|---|---|---|
event | (a) type | tool_call.decision |
ts | (b) when | 2026-11-18T10:02:19Z |
gate, target_system | (c) where | bw-gate-1, purchasing |
agent_id, principal | (d) source, (f) identity | bw-parts-buyer, luis.m |
decision, matched_rule | (e) outcome | DENY, deny:change_supplier_bank_details |
approval_id, approver_id, oob_ref | authorisation event | APR-5531, priya.procurement-lead, CB-2207 |
cred_ref | non-secret credential reference | po-api/bw-parts-buyer#key-3 |
Excluded content. OWASP's Logging Cheat Sheet says certain data "should usually not be recorded directly in the logs, but instead should be removed, masked, sanitized, hashed, or encrypted", including "Access tokens", "Authentication passwords", "Database connection strings" and "Encryption keys and other primary secrets".5 NIST IA-5(g): "Protecting authenticator content from unauthorized disclosure and modification".2
A cred_ref is an identifier for the credential (store path, key id, version), resolvable only by the secret store's access controls. It lets an investigator answer "which key, so revoke it" without the log becoming a credential store. Logs are widely read, retained and shipped to other systems, so a secret in a log is a disclosure with a long tail. Put the agent's tool arguments through the same filter: an argument can carry a token as easily as a field can. OWASP's wording is "usually"; this course's rule is stricter: no authenticator content in an agent's audit record, ever.
The NCSC guidelines ask that you "monitor and log inputs to your system" to "enable compliance obligations, audit, investigation and remediation in the case of compromise or misuse".3 The test for the record is the incident question: which agent, for which principal, under which rule or approval, with which credential, with what result. A record that reads only j.smith fails the first two.
6.1 Audit record versus receipt
The audit record above is a private, access-controlled work record: it carries principal (a person) and, in the trace, arguments such as supplier and amount. A receipt, in this course, is the shareable artefact of one gate decision. It carries identifiers and digests only: agent_id, decision, matched_rule, a digest of the policy version, args_digest, approval_id, cred_ref (a key id). It never carries prompts, model outputs, call payloads or personal data, so not principal, not args, not the injected email. A digest is not a licence to share what it was computed from, and this module does not claim every digest is anonymous. Treat args_digest as a binding, not as a privacy measure, and keep the receipt's audience in mind. Signing is a separate step: an unsigned decision record with these fields is not yet a signed receipt, and a signature shows who issued the record, not that the decision was right.
7. The three yeses
The course's lifecycle, Get to Yes · Stay at Yes · Recover to Yes, applied to a bounded agent:
| Movement | When | This module's controls |
|---|---|---|
| Get to Yes | before any action runs | Policy with default deny and deny-overrides; named approvers with review dates; out_of_band flags; a dedicated agent identity with least privilege downstream; an expected-decision test suite run to green, with zero side effects on refusals; a named owner who can revoke the agent's credential. The yes covers these tools within these conditions only. |
| Stay at Yes | while it runs | Read gate decisions (ALLOW, NEEDS_APPROVAL, DENY), not just results; approvals expire; unanswered holds resolve to DENY; any change to tools, permissions, supplier data, approvers or the evaluator re-opens Get to Yes and re-runs the suite. |
| Recover to Yes | on incident | Revoke the agent's credential by cred_ref without touching the principal; answer the incident question from the audit record; name the owner of the fix; patch the policy or permissions; re-run the suite and keep the new trace. A disabled agent is contained, not yet recovered. |
Unknowns stay visible. Whether a given downstream system can enforce per-identity permissions, or supports delegated minimum-scope grants, is a property of that system; if you have not checked it, the design records "not checked" and an owner, not an assumption.
8. Summary
- Policy: default deny, deny overrides, conditions on arguments, complete mediation on every call, enforced outside the model.1 Least privilege is the principle.32 The combining rules are this course's.
- Evidence: an expected-decision suite; a dry-run trace with a
DENYand zero side effects for each prohibited call, a negative test for an unlisted tool, and each protection tested alone, with the expected decision derived from the rules that still match. Supplied traces are teaching material; your evidence is a trace the course sandbox generates from your policy (§3.4). - Out-of-band (this course's definition and rule, §4.2): confirm on a channel outside the requester's control, bound to the transaction, for value transfers, destination changes, urgency or outsider-writable inputs. The case that motivates it is the Hong Kong pre-recorded deepfake conference.4
- Identity: the agent authenticates as itself, with a minimum-scope delegation from the principal.12
- Audit: AU-3 content plus agent and principal, the authorisation event and a credential reference.25 No authenticator content is this course's rule; OWASP says "usually".
- Receipt: identifiers and digests only; the audit record stays private.
Words to know: excessive agency, least privilege, allow, constrain, hold, deny, default deny, deny overrides, complete mediation, approval binding, dry run, expected decision, negative test, negative coverage, out-of-band confirmation, requester-controlled channel, workload identity, principal, delegation, credential reference, audit record, receipt, authenticator.
Further learning (optional, advanced, not part of this lesson): UC Berkeley, CS294/194-280 Advanced Large Language Model Agents, Spring 2025, archived lectures and slides: <https://rdi.berkeley.edu/adv-llm-agents/sp25>. Harvard, CS50's Introduction to Cybersecurity, free OpenCourseWare: <https://cs50.harvard.edu/cybersecurity/>. Nothing here is copied from either.