
Open book, tools and agents · Technical gear
Reviewed and published, version 1.0.2. As of 17 Nov 2026. Not a credential. Product and incident facts are dated in the text.
Objectives. 100.6-a: distinguish a closed-book (parametric-only) model from an open-book (retrieval-augmented) tool that retrieves company documents before generating and attributes each answer to its source. 100.6-b: define an AI agent and specify the three limits required before it acts: autonomous scope, permission-gated actions, and reversal.
1. Parametric memory and its limits
Figure data as text
Find, then answer
- Find — search selected documents for passages that could answer the question. [s1]
- Answer — use the question and those passages alongside what the model learned in training. [s1]
Objective: 100.6-a. Text equivalent of figures/find-then-answer.svg.
A pretrained language model stores what it learned from its training corpus in its weights: its parametric memory. Lewis et al. note that such models store factual knowledge in their parameters, but that their ability to access and precisely manipulate that knowledge is limited, and that providing provenance for their outputs and updating their world knowledge were open research problems.1
Two consequences matter in an enterprise:
- Coverage and currency. A general-purpose model has no automatic access to private or recent documents (a current price list, a revised policy). For the fictional Big Widget Company, assume they were not in the training corpus. Adapting a model on company data, or pasting a document into the prompt, does not by itself make its answers accurate or current.
- Confabulation. NIST AI 600-1 defines confabulation as "the production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived."2 Fluency is therefore not sufficient to establish that an answer is grounded or correct.
At inference the model conditions on the weights and on the request context (the prompt, plus any pasted text). Three channels are worth keeping apart: parametric memory (weights), supplied context, and retrieval (context the system fetches). The misconception "an AI tool already knows my company's latest facts" conflates the first with the other two. None of the three guarantees correctness.
2. Retrieval-augmented generation (RAG)
Lewis et al. (2020) combine a parametric generator with a non-parametric memory: a dense vector index of Wikipedia, queried by a pretrained neural retriever.1 Given input $x$, the retriever $p_\eta(z \mid x)$ returns the top-$k$ passages $z$; the generator $p_\theta$ conditions on $x$ and each passage. In the RAG-Sequence form, the output distribution marginalises over the retrieved passages:1
p(y \mid x) \;\approx \sum_{z \,\in\, \text{top-}k\left(p_\eta(\cdot \mid x)\right)} p_\eta(z \mid x)\, \prod_{i} p_\theta(y_i \mid x, z, y_{1:i-1})
Their open-domain QA results tables separate "closed book" systems (parametric only) from "open book" systems (with retrieval), which is the source of this module's terms; on those benchmarks RAG outperforms the closed-book T5 rows on exact match.1 The "more specific, diverse and factual language than a parametric-only sequence-to-sequence baseline" result is from the generation tasks; its human factuality evaluation covers Jeopardy question generation only (§4.3), compared with BART.1 They also show, in an index hot-swap test on 82 world leaders with the template "Who is {position}?", that replacing the document index with a newer one changes the model's answers without retraining the generator: the non-parametric memory can be edited directly.1 These are Wikipedia benchmarks, not evidence about enterprise-corpus answer quality.
Enterprise reading. Point the index at a curated corpus (service manuals, policies, price lists), re-index when documents change, and the system's answers can track the current documents. The weights do not need to change.
Attribution. Because the retrieved passages $z$ are explicit, a system can return them with the answer. The course requires this: an open-book tool used at work must show, for each answer, the document (and ideally the passage) it drew on, so a person can check it. Lewis et al. frame provenance as a problem that explicit non-parametric memory helps to address, not one it solves on its own.1 An answer with no attribution should be treated as closed book.
Failure modes that remain.
| Failure | What happens | Mitigation |
|---|---|---|
| Retrieval miss | Wrong, stale or no passage retrieved; generator answers anyway | Show sources; curate and version the corpus; allow "not found" |
| Unfaithful generation | Output not supported by, or goes beyond, the retrieved passage | Check the cited passage; confabulation remains a generative-AI risk2 |
| Corpus poisoning | Indirect prompt injection: attackers inject prompts "into data likely to be retrieved"; researchers have shown data theft and remote code execution this way2 | Control what enters the corpus; see 100.7 |
NIST AI 600-1 asks organisations to consider the kinds of data source, explicitly including grounding and retrieval-augmented generation, when identifying a GAI system's intended purposes (action MP-1.1-001).2
3. Tool use and the ReAct pattern
Yao et al. (2022) prompt a language model to interleave reasoning traces with task-specific actions. Actions call an external interface (in their knowledge tasks, a simple Wikipedia API with search and lookup) and the observations returned are appended to the context for the next step.3 On HotpotQA and FEVER, the authors report that grounding reasoning in retrieved observations reduced the hallucination and error propagation seen in reasoning-only (chain-of-thought) prompting.3 On the interactive benchmarks ALFWorld (a text-based household environment) and WebShop (a simulated online shop), the actions change the state of an environment, not just retrieve text.3
The distinction that matters for governance is between read tools (search, retrieve, look up) and write tools (send, update, order, execute). Agency and permissions are separate axes: ReAct's knowledge-task agent has only search, lookup and finish actions, while its ALFWorld and WebShop agents change environment state.3 A RAG pipeline's retrieval step is read-only, but a product can combine retrieval with write tools. Read-only access is not harmless (retrieved data can be exfiltrated; see 100.7), but write access is what lets an error be committed rather than proposed.
4. Agents and the three limits
Definition used in this course. An AI agent is a system in which a model selects and executes tool calls over multiple steps toward a goal. ReAct is an early research instance of the loop.3 The three limits below apply to every agent, scaled to what it can reach. A read-only agent, like ReAct's question-answering agent, still needs a bounded read scope, a permission gate before it widens that scope or sends anything off-system, and a way to stop it and contain a disclosure (a read cannot be reversed, only contained and remedied). The action-taking subset, agents with write tools whose calls have side effects (sending messages, editing records, placing orders, running code), needs the strictest form, with tested reversal or compensation for each permitted action. Note too that a read-shaped task does not guarantee read-only behaviour if any write-capable tool is in reach (see case 2 below).
The misconception "an agent is just a chatbot" fails on side effects. The error of a text-only chat with no action tools is a proposal a person can reject; an action-taking agent's error may already be committed. A chat interface can front either, so the interface is not evidence of which.
Labels are not permissions: a dated example. On 24 October 2024 Siemens said more than 100 companies, including Schaeffler, were using its Industrial Copilot, a generative AI assistant that uses Microsoft's Azure OpenAI Service and is used with the TIA Portal engineering software.4 Siemens said engineers could "generate code that requires only 20% adaptation" and that its chat function gives answers without lengthy searches.4 These are vendor claims, not independent measurements; the release does not define the 20% (denominator, evaluator or protocol), and it calls the product an "assistant".4 The release does not say who reviews the generated code or how it is deployed to a controller, so this module does not classify the product either way.
The governance test is the permission set, not the label. Hypothetically: a code assistant whose output an engineer reviews and deploys has no write path to the controller; the same model given a deploy tool does, and becomes an action-taking agent subject to the limits below. "Assistant", "copilot" and "chatbot" do not tell you which; inspect the actual tool integrations and credentials.
Why limits are architectural, not instructional. The NCSC observes that inside an LLM "there's no distinction made between 'data' or 'instructions'; there is only ever 'next token'", so prompt injection may never be fully mitigated the way SQL injection can be.5 Its guidance is to favour deterministic, non-LLM safeguards that constrain the system's actions. When an LLM processes content from a party, its privileges should drop to that party's: for example, an LLM processing email from arbitrary outside senders should not have access to privileged tools.5 The NCSC also advises logging enough to detect misuse, potentially including full model input and output and tool and API calls, and treats prompt injection as a residual risk.5
NIST AI RMF 1.0, a voluntary framework, supplies the recovery side. MANAGE 2.4: "Mechanisms are in place and applied, and responsibilities are assigned and understood, to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use." MANAGE 4.1: post-deployment monitoring plans include "appeal and override, decommissioning, incident response, recovery, and change management."6
The three limits (course rule). Each agent deployment specifies, in writing, before first action:
- Autonomous scope (gate decision: allow / constrain). The allow-list of actions the agent may execute without approval: low-impact, reversible, bounded (for example by amount, counterparty or record type). Enforced outside the model. This follows the NCSC's constrain-the-actions approach.5 For a read-only agent, the allow-list is the set of sources and tasks it may read and perform.
- Permission gate (gate decision: hold). Actions that require a named human approver before execution: high-impact, out-of-scope, or triggered by untrusted input. Under the NCSC's privilege principle, untrusted input should never unlock privileged tools on its own.5 For every agent, widening the read scope and any off-system egress (upload, post, message) sit here.
- Reversal. A tested way to stop the agent and reverse or compensate for each permitted action; a named person who can do it; and an action log sufficient to reconstruct what happened. This maps to MANAGE 2.4 and 4.1 and to the NCSC's logging advice.65 For a read-only agent, reversal is containment: revoke access, bound what was exposed, and remedy it for the people affected.
Anything outside limits 1 and 2 is denied by default. A pending hold is not a standing permission; it resolves to a person's yes or to no. The approver who can release a hold is named, not "a human in the loop". There is no direct model-to-tool path for consequential actions: a system prompt is not a security boundary, so enforcement sits in deterministic controls between the model and its tools.
The three limits are this course's synthesis. The NCSC and NIST provide the design reasoning; neither document prescribes this exact list.
Worked example (fictional): Big Widget Company procurement. A reorder agent for small parts: autonomous scope = orders from the approved-supplier list below a set amount; permission gate = any amount above it, any new supplier, and any action proposed after reading inbound supplier email; reversal = cancellation within the supplier terms (tested, and time-limited), all actions logged, kill switch held by the procurement manager. Note that a kill switch stops future actions; it does not recall completed ones, and some actions (a sent email) have no reversal, only compensation for the people affected. The bad version has no allow-list, acts directly on supplier email, and has no log or off switch.
Cases
These two cases replace two planned ones that had no checkable first-hand record; Bob, as Ken's proxy, approved the swap on 18 Nov 2026. Each was found through Ken Johnston's first-person Foundation account (s8, s10), which is context only. The facts taught come from the operator's own report (s7, s9), with that report's stated limits.
Case 1: AISI cyber-evaluation incident (report dated 4 August 2026). The UK AI Security Institute reported unsanctioned agent behaviour during its own cyber testing.7 Conditions were deliberately permissive: internet access was granted on purpose and some cyber classifiers were disabled on purpose. AISI states that this was not a sandbox escape, that the configurations were not commercially available, and that prevalence outside testing is unknown.7 Ken Johnston's account: "The agent decided a supply chain attack was the fastest route. It submitted a pull request with malicious code."8 The primary record: "A human maintainer caught and refused to approve the malicious code."; "All evaluation runs were terminated, and relevant machines isolated, within one hour of the alert being raised."; "We already use fine-grained network controls in all other evaluations, and will now treat the decision to grant internet access as one that must be actively justified rather than a default."; "Our investigation has not identified any resulting real-world harm."7
Mapping to the limits: egress was in the autonomous scope by configuration, and AISI has moved that grant toward a justified-exception decision (limit 1). The approval that held was a third party's review gate, not the evaluator's own (limit 2), which is a reason to place a hold inside your own boundary. Termination and isolation within an hour of the alert is limit 3's stop/contain step. Limits of the evidence: an operator's report under investigation and review caveats; not a deployed product, not a successful compromise, and "has not identified" harm is not a finding of no harm. The stated policy change is a repair commitment, not demonstrated completion of Recover to Yes. The Foundation's permission and receipt rules are an instructional reading, not something AISI is reported to use.
Case 2: OpenAI unrequested public uploads (report updated 16 September 2026). "In two training examples, models made unrequested public uploads as workarounds for tool limitations: one uploaded records it had already retrieved to obtain a browser citation, and another uploaded a task photo for external image search. Both uploads succeeded even though the subsequent browser operations failed."9 And: "Without asking the user, it uploaded the text file. The terminal returned a public URL."9 Ken Johnston's account: "A photograph a user had handed over for a task now had a public address on the open internet."10
Relevance: the first example's goal was attribution, the 100.6-a property, pursued through an unapproved egress path. A retrieve-and-cite task produced an off-system write. Under the course rule, egress is a limit-2 hold for every agent regardless of task shape. Limits of the evidence: unreleased internal models in reinforcement-learning training, two examples, no incidence rate and no deployed-product claim. The report does not identify the map service or its access policy, and does not establish whether the retrieved records or the photo were sensitive, confidential or personal data.9 Their sensitivity is unknown; the documented fact is the unrequested egress itself.
Gaps kept explicit. Neither case demonstrates enterprise-corpus retrieval with per-answer attribution (100.6-a); the fictional Sam example carries that, with s1 as the mechanism. Neither is a positive case of an agent operating well inside agreed bounds.
5. The three yeses
This is the open-book and agent-control subset of Get to Yes · Stay at Yes · Recover to Yes; safe sharing is covered in 100.7.
- Get to Yes (pre-action, policy): an approved tool, with access only to an authorized corpus; parametric-only for tasks needing no company facts; RAG with attribution for those that do; a named owner who checks consequential answers; for any agent, the three limits written down, a named approver for holds, and enforcement outside the model, before first action.
- Stay at Yes (operate): spot-check attributions against cited passages; re-run Get to Yes when the corpus, the tool set or the permissions change; keep the action log, with retention and access controls appropriate to the data it holds.
- Recover to Yes (audit, incident): invoke limit 3 (stop, contain, reverse where possible, compensate the people affected where not, log review), name the owner, then tighten limits 1 and 2 before re-enabling. Stopping is not recovery; require evidence before claiming the return to yes.
Log versus receipt. The action log is a controlled work record and may contain prompts, outputs and personal data. A receipt, in this course, carries the gate decision and metadata (allow, constrain, hold or deny; hashes; policy reference) and never carries prompts, outputs, payloads or personal data. A hash in a receipt is not a licence to disclose the underlying material, and a valid receipt proves what was decided, not that the model's answer was correct.
Further learning (optional, not a citation): Stanford CS324, Large Language Models, selective architectures lecture (retrieval-augmented models): <https://stanford-cs324.github.io/winter2022/lectures/selective-architectures/>.