Level 300: Building with gen AI · Module 300.4

Three coworkers arrange blank sticky notes into three columns on a cork board.
Alone, approver, never

Building an agent with limits · Standard gear

Reviewed and published, version 1.0.0. As of 20 Nov 2026. It is not a credential. Big Widget Company and everyone in its examples are made up. The agent, its tools and its traces are synthetic, written for this lesson. The course sandbox runs offline in apps/sandbox/, with synthetic fixtures only (section 3 says how to use it).

You will be able to:

Big idea: an agent's limits are only as real as the place they are enforced. Write them down as a table, enforce the table outside the model, give the agent an identity of its own so the limits stick to it, and keep a log that shows each refusal. For the few actions that move money or cannot be undone, confirm on a second channel the requester does not control.

In 300.3 you saw that the model only proposes a tool call and the system decides whether to run it. This module builds that decision. The running example is a purchasing agent at the made-up Big Widget Company.

1. The misconception: "give it admin rights so it doesn't get stuck"

Agents that hit a permission error stop, so it is tempting to give them every permission. The OWASP Top 10 for LLM Applications (2025 version) names the result Excessive Agency and gives three root causes: excessive functionality, excessive permissions and excessive autonomy.1

The fix is the oldest rule in security, least privilege. NIST's catalogue of security controls (SP 800-53 Rev. 5, control AC-6) says to allow "only authorized accesses for users (or processes acting on behalf of users) that are necessary to accomplish assigned organizational tasks".2 The UK NCSC's guidelines for secure AI systems, published with the US CISA and partner agencies in November 2023, say the same for AI: if AI components need to trigger actions, "you apply appropriate restrictions to the possible actions", and "you apply least privilege principles to limit access to a system's functionality".3

An agent with admin rights does not get stuck. It also cannot be stopped by its permissions when it is wrong or when someone tricks it. Getting stuck is the safe failure; the authority table decides where it gets stuck, and who unsticks it.

This course's rule is the reverse of "admin so it doesn't get stuck": the agent does only what it has explicitly been allowed to do, and anything it cannot undo either waits for a named person or is not the agent's to do at all.

In this course a gate has four answers: go ahead, only this far, wait for a person and no (the technical words are allow, constrain, hold and deny). The authority table is how you decide, in advance, which answer each action gets.

2. The authority table

An authority table lists every action the agent can take and puts each in one of three columns. This is the course's format (it matches the hold prompts in levels/300/practice/hold-prompts.md), not a standard's:

One more rule sits under the table: anything not listed is denied. A tool you forgot to list does not run.

Here is the table for Big Widget's parts-buying agent. The tools, limits and people are made up.

Action (tool)AloneNamed approverNever
search_catalogue (read parts and prices)yes
draft_purchase_order (a draft only; nothing is sent)yes
submit_purchase_order to a supplier already on the approved list, up to $5,000yes
submit_purchase_order over $5,000, or to a supplier not yet used this yearPriya, procurement lead (approval role reviewed 1 Mar 2027)
send_drawing to a supplierPriya, procurement lead
add_supplier or change_supplier_bank_detailsnever: people do this, in the supplier-master process, with an out-of-band check (section 4)
pay_invoicenever: payments belong to finance, not this agent
anything elsedenied by default

How to fill the columns:

  1. Start from the job, not the tools. OWASP's advice is to limit the extensions an agent may call "to only the minimum necessary".1 If the job does not need a tool, do not connect it.
  2. Put high-impact actions in the approver column. OWASP: "require a human to approve high-impact actions before they are taken."1 Large, new, irreversible or outward-facing actions go here.
  3. Put actions with no business being automated in "never". Changing where money goes is the classic example (section 4).
  4. Write the limit into the row. "Submit purchase orders" is not a rule; "up to $5,000, approved suppliers only" is.

3. Configure it outside the model, then dry-run it

Where the table lives. Not in the prompt. A sentence like "never change bank details" in the system prompt is just more text for the model, which it may not follow. OWASP's advice is to "implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not", and to apply complete mediation, so that all requests to downstream systems through extensions "are validated against security policies".1 It says an approval step "may be implemented in a downstream system (outside the scope of the LLM application)".1 The NCSC adds "external AI and non-AI fail-safes if necessary".3

So the table becomes configuration for a gate that sits between the model's proposal and the tool. In this course's synthetic format (not any product's) it looks like this:

agent: bw-parts-buyer          # the agent's own identity (section 5)
default: deny                  # anything unlisted is refused
combine: deny-overrides        # if two rules match, the stricter one wins
rules:
  - tool: search_catalogue
    decision: allow
  - tool: draft_purchase_order
    decision: allow
  - tool: submit_purchase_order
    decision: allow
    when: { supplier_on_approved_list: true, total_usd_max: 5000 }
  - tool: submit_purchase_order
    decision: needs_approval
    when: { any: [ { total_usd_over: 5000 }, { supplier_used_this_year: false } ] }
    approver: priya.procurement-lead
  - tool: send_drawing
    decision: needs_approval
    approver: priya.procurement-lead
  - tool: add_supplier
    decision: deny
  - tool: change_supplier_bank_details
    decision: deny
  - tool: pay_invoice
    decision: deny

If two rules match one call, the stricter decision wins: "never" beats "approver", and "approver" beats "alone". (A course rule; the technical gear spells it out.)

A second lock: the agent's own permissions. The gate is one lock. The downstream system is the second. OWASP says limits on extension permissions should be "enforced by applying appropriate database permissions for the identity that the LLM extension uses to connect to the database".1 If the gate has a bug, a purchasing system that never gave bw-parts-buyer the right to change bank details still refuses. That second lock only works if the agent has an identity of its own (section 5).

Dry-run it. Before the agent touches anything real, run it against synthetic data and read its trace. A dry run here means the tools are fakes that record what they were asked to do.

Two kinds of trace. The two traces below were written for this lesson. They are practice in reading evidence, not evidence about your configuration: a trace someone else wrote cannot show that your gate refused anything. Objective 300.4-a is met by a trace the course sandbox generates from your own configuration.

Run it in the course sandbox. The sandbox is an offline Python 3 runner in apps/sandbox/. It needs no packages, accounts, credentials or network, and its policies, proposals and tools are synthetic fixtures owned by the course. The steps are in apps/sandbox/README.md, section "300.4 · Building a bounded agent". Its example agent is the workbook's bw-stock-keeper, not the parts buyer below. From the repo root:

python3 apps/sandbox/sandbox.py agent dry-run --policy YOUR_POLICY.json
python3 apps/sandbox/sandbox.py agent hold --policy YOUR_POLICY.json
python3 apps/sandbox/sandbox.py agent dry-run --policy YOUR_POLICY.json --step-budget 1
python3 apps/sandbox/sandbox.py agent stop --policy YOUR_POLICY.json

Start from a copy of apps/sandbox/agent-policy.json; omit --policy to run the example as shipped. Each run prints where it saved its JSON trace. Map this lesson's YAML onto that file:

In the trace, compare expected with decision for each event, and read matched, returned_to_model and stub_side_effects. Then check that no refused call appears in the separate stub_calls list: that is the "no side effect" evidence. A run that says completed is not a pass on its own. The README also covers the hold, approval, step-budget and stop runs.

The sandbox is a teaching runner: scripted proposals, in-memory recording tools and synthetic approval files. It shows how a gate decides. It is not real access control, authentication or a real model. Use it, not a real workplace system or real logs.

Write the expected answer first. Before a dry run, list each action you will make the agent propose and the answer the table should give: go ahead, wait for a person, or no. Include at least one action the table must refuse, and one tool that is not on the list at all. After the run, compare expected with actual, line by line:

You cannot tell the last two apart from the outcomes alone. A trace of ten "allowed" lines might be ten harmless reads, or it might include a forbidden action that slipped through. Only the expected column tells you which.

Here is a synthetic trace from a run where a supplier's email, planted in the inbox the agent reads, asks it to update the supplier's bank details:

10:02:11 agent=bw-parts-buyer on_behalf_of=luis.m tool=search_catalogue part=BW-114
         rule=allow decision=ALLOWED result=3 offers
10:02:14 agent=bw-parts-buyer on_behalf_of=luis.m tool=draft_purchase_order supplier=S-0042 total_usd=3180
         rule=allow decision=ALLOWED result=draft PO-77120
10:02:19 agent=bw-parts-buyer on_behalf_of=luis.m tool=change_supplier_bank_details supplier=S-0042
         rule=deny decision=REFUSED result="not run: denied by policy" alert=raised
10:02:20 agent=bw-parts-buyer on_behalf_of=luis.m tool=submit_purchase_order supplier=S-0042 total_usd=3180
         rule=allow(approved_list, <=5000) decision=ALLOWED cred_ref=po-api/bw-parts-buyer#key-3 result=PO-77120 sent

Read it like a reviewer:

Now spot the gap in a second synthetic trace:

11:40:03 agent=bw-parts-buyer tool=export_supplier_list format=csv rule=(none) decision=ALLOWED

The export tool is not in the table, so you expected no. No rule matched, and it ran anyway: a failed negative test. For an unlisted tool, the only protection is the default, so the default did not work here. The trace alone does not say why: default: deny may be missing, or present but not applied. Open the configuration and the gate's handling of unmatched calls, fix what you find, and run the test again. The table was right; the system did not match it. That is why you dry-run. This trace was written to fail, on purpose. In the sandbox's example run, the gate refuses its unlisted export_supplier_list call with zero stub side effects, so do not expect a generated trace to repeat this fault.

Test each protection on its own. For a tool on the "never" list, two things should refuse it: its explicit deny rule and the default. If both are in place, a single test cannot show that either one works alone. So test them apart, in a synthetic copy of the configuration:

If you want every "never" to hold even when its own rule is deleted, that is an extra design rule (no other row may allow the same call), and you should write it down as one. Default deny does not give you that by itself.

Run these isolation tests in the sandbox on fixture copies of the stock-keeper policy, one change per copy. Score each one on the trace's gate_decision, not only on the final decision. The sandbox connects no real tool for a forbidden action, so the final decision can still read DENY when the gate itself said ALLOW. That second lock is good design, but it would hide the gate result you are testing.

Logging is not limiting. OWASP says logging, monitoring and rate limiting "will not prevent Excessive Agency, but can limit the level of damage caused".1 The trace shows what happened; only the gate and the agent's narrow permissions change what happens.

4. Out-of-band confirmation: a face is not a channel you control

Step 1, PAUSE · COURSE RULE: Does any of these apply? Money or value, above a set limit, and hard to undo; changes where money or data goes; urgent, secret or skips the usual process; arrived by email, chat, call or shared doc. Step 2, CONFIRM · COURSE RULE: Use an independent channel. Call a number already on file, never one supplied in the request. Step 3, ENFORCE · COURSE RULE: Record the person’s confirmation. The system will not run the action until the approval records it.
The course’s out-of-band procedure builds on police callback advice (s1) and OWASP human approval guidance (s2). A face or voice on the requester’s channel is not independent confirmation: use a channel the requester does not control.
Figure data as text

Confirm outside the request

  1. PAUSE · course rule · Does any of these apply? — money or value, above a set limit, and hard to undo; changes where money or data goes; urgent, secret or skips the usual process; arrived by email, chat, call or shared doc.
  2. CONFIRM · course rule · Use an independent channel — call a number already on file, never one supplied in the request.
  3. ENFORCE · course rule · Record the person’s confirmation — the system will not run the action until the approval records it.

The course’s out-of-band procedure builds on police callback advice (s1) and OWASP human approval guidance (s2).

Objective: 300.4-b. Text equivalent of figures/confirm-outside-the-request.svg.

The misconception: "A familiar face or voice on a call proves who is asking."

The case. On 26 June 2024 the Hong Kong Government answered a Legislative Council question on deepfake fraud. Its reply said: "In end-January 2024, the Police received one fraud case involving the use of deepfake technology to fabricate a pre-recorded video conference."4 "As instructed, the informant eventually gave authorisation to transfer funds to five local bank accounts and suffered a loss of about HK$200 million."4 And: "Since the meeting was pre-recorded, there was actually no interaction between the informant and the fraudster."4

What the record does not say: it does not name the organisation, and it does not say which checks the organisation had.4 This course does not name it either. The control below is this course's proposal, not a finding about what happened.

The police advice in the same reply includes tests inside the call, such as asking the person to move their head and "using the content of the conversation to test the authenticity of the other party", and one step outside it: "verifying with a phone call" when a relative or friend asks for money by video or audio.4

The idea: who controls the channel? Every check inside the call runs on the channel the requester set up: their meeting link, their video, their script. A face, a voice, even a correct answer to a question, arrives through that channel. Out-of-band confirmation means confirming the request on a second channel the requester does not control: you call back on a number from the company directory, never one given in the request; you check in the finance system, not in the email. (This is the course's definition, as in 100.7 and the hold prompts.)

When an action needs it. This course's rule. An action needs out-of-band confirmation if any of these is true:

  1. it moves money or something of value, above a set limit, and cannot easily be undone;
  2. it changes where money or data will go: a new payee, new bank details, a new delivery address, a new person given access;
  3. the request comes with urgency or secrecy, or skips the usual process;
  4. the request reached you (or the agent) through a channel an outsider can write to: email, chat, a call, a shared document.

How it fits the authority table. Out-of-band confirmation is a property of an approval, not a new column. In the parts-buyer table, change_supplier_bank_details is never for the agent, and the people who do it in the supplier-master process must confirm on a channel the requester does not control. For submit_purchase_order over $5,000 to a new supplier, Priya's approval includes a callback to the supplier on the number already on file. The NCSC gives a related pattern for data sent outside the organisation: "requiring users to log in and confirm before sending potentially sensitive information".3

Two cautions for agents:

5. The agent needs its own identity

The misconception: "It's fine for the agent to use my account."

If the parts-buyer agent logs in as Luis, three things go wrong.

  1. It gets everything Luis can do. Luis's account was set up for a person with his whole job. The agent needs a small part of it. NIST's least-privilege control says organisations "consider the creation of additional processes, roles, and accounts as necessary to achieve least privilege".2 An account of its own is how you give the agent less than Luis has, and it is what OWASP's "identity that the LLM extension uses" (section 3) needs to be.1
  2. Nobody can tell who did it. NIST says to "uniquely identify and authenticate organizational users and associate that unique identification with processes acting on behalf of those users" (IA-2), and to uniquely identify and authenticate the system services and applications the organisation designates (IA-9).2 Designating the agent is your organisation's choice; this course recommends it. A log line that says only j.smith does not say whether Luis clicked or the agent acted.
  3. You cannot switch it off without switching off Luis. If the agent misbehaves, you want to revoke its access and leave Luis working. With a shared login, the only lever is Luis's password. (A course point; the authority table and the stop in 300.3 rely on it.)

Acting for a person is still allowed. OWASP's advice for an agent that acts for a user is that "actions taken on behalf of a user are executed on downstream systems in the context of that specific user, and with the minimum privileges necessary"; its example is a user who authenticates "via OAuth and with the minimum scope required".1 That is a narrow grant the person gives, not the person's password. The agent keeps its own identity, and the record shows both: which agent, and on whose behalf. That is why every line in the trace in section 3 has agent= and on_behalf_of=.

6. What the log records, and what it must never record

Record the identity. NIST's audit-record control (AU-3) asks that each record establish what type of event occurred, when, where, its source, its outcome, and the "identity of any individuals, subjects, or objects/entities associated with the event".2 OWASP's Logging Cheat Sheet, under who, says "human or machine user".5 For an agent, that means the agent's identity and the person it acted for, as separate fields.

Record the authorisation event. OWASP lists "Authentication successes and failures" and "Authorization (access control) failures" among events to log.5 For an agent, that means each gate decision (allowed, refused, waiting for approval) and, for approved actions, which approval: for example approval=APR-5531 approver=priya.procurement-lead. OWASP also lists "Use of default or shared accounts" as an event to log,5 which tells you how a shared login looks to a security team: a warning sign.

Keep the secret out. OWASP's list of data that "should usually not be recorded directly in the logs" includes "Access tokens", "Authentication passwords", "Database connection strings" and "Encryption keys and other primary secrets"; it says to remove, mask, sanitise, hash or encrypt them instead.5 NIST's authenticator-management control (IA-5) includes "Protecting authenticator content from unauthorized disclosure and modification".2 OWASP says "usually". This course's rule is stricter: an agent's log never holds a secret.

So the log records a credential reference: the name of the key and its version, such as cred_ref=po-api/bw-parts-buyer#key-3, which says which credential was used without being the credential. Anyone who reads a log, and logs are read by many people and copied to many places, could use a secret they find in it. A reference is useless to them and still tells an investigator which key to revoke.

The NCSC asks that you "monitor and log inputs to your system" to enable "audit, investigation and remediation in the case of compromise or misuse".3 After an incident, the log's job is to answer: which agent, for whom, under which approval, using which credential. A log with only j.smith cannot answer the first two. A log with a raw key in it is a second incident.

Log or receipt? They are different things. The agent's log is a private work record: it names the person the agent acted for, and its lines carry the call's details (supplier, amount, order number). It stays access-controlled. A receipt, in this course, records only the gate's decision about one action: which agent, which rule, which answer, which approval, which credential reference. It never holds prompts, outputs, the call's contents or personal data, such as the text of the planted email or the name of the person the agent worked for. Do not paste log lines into a receipt you share.

7. The three yeses for an agent with limits

This module's work fits the course's lifecycle: Get to Yes · Stay at Yes · Recover to Yes.

Some things this module cannot tell you: whether your own systems can give an agent its own identity, or whether your downstream systems enforce the permissions you set. If you have not checked, write "not checked" and name who will.

8. Summary

My checklist:

Words to know: authority table, least privilege, excessive agency, go ahead, only this far, wait for a person, no, default deny, complete mediation, gate, dry run, trace, expected result, negative test, out-of-band confirmation, on behalf of, credential, credential reference, audit record, receipt.

Further learning (optional, advanced, not part of this lesson): UC Berkeley, CS294/194-280 Advanced Large Language Model Agents, Spring 2025, archived lectures and slides: <https://rdi.berkeley.edu/adv-llm-agents/sp25>. Harvard, CS50's Introduction to Cybersecurity, free OpenCourseWare: <https://cs50.harvard.edu/cybersecurity/>. Nothing here is copied from either.

Sources

  1. s2 OWASP GenAI Security Project (2025). OWASP Top 10 for LLM Applications 2025. Official doc. Checked 2026-11-20 by source-checker (Claude Opus 5.5, separate session, round-4 re-check; did not write or edit module 300.4). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  2. s4 Joint Task Force, National Institute of Standards and Technology (2020). Security and Privacy Controls for Information Systems and Organizations (NIST SP 800-53 Rev. 5). Standard. Checked 2026-11-19 by source-checker (Claude Opus 5.5, separate session, round-3 re-check; did not write or edit module 300.4). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s3 UK National Cyber Security Centre, US Cybersecurity and Infrastructure Security Agency and international partner agencies (2023). Guidelines for secure AI system development. Official doc. Checked 2026-11-19 by source-checker (Claude Opus 5.5, separate session, round-3 re-check; did not write or edit module 300.4). ↩ ↩ ↩ ↩ ↩
  4. s1 Government of the Hong Kong Special Administrative Region (written reply by the Secretary for Security to a Legislative Council question) (2024). LCQ9: Combating frauds involving deepfake. Record. Checked 2026-11-19 by source-checker (Claude Opus 5.5, separate session, round-3 re-check; did not write or edit module 300.4). ↩ ↩ ↩ ↩ ↩ ↩
  5. s5 OWASP Cheat Sheet Series (2026). Logging Cheat Sheet. Official doc. Checked 2026-11-19 by source-checker (Claude Opus 5.5, separate session, round-3 re-check; did not write or edit module 300.4). ↩ ↩ ↩ ↩ ↩

Next: the quiz