Level 300: Building with gen AI · Module 300.2

Two coworkers sort coloured folders into three trays; one holds a small padlock.
Who may see this?

Building a retrieval assistant · Technical gear

Reviewed and published, version 1.0.0. As of 20 Nov 2026. It is not a credential. Big Widget Company, its people, documents, test users and traces are made up. The exercises run in the course's offline sandbox, apps/sandbox/ (synthetic fixtures owned by this course); apps/sandbox/README.md gives the exact commands. The YAML and the assertion code in sections 2 to 4 are illustrative and product-neutral: they state the design, and each section says how it maps to the sandbox's JSON files and Python functions. The call-counting script in section 3 has been run against the sandbox. Reading the workbook's printed traces alone does not meet objective 300.2-b.

You will be able to:

1. The system, not the prompt

Misconception: "A good prompt is the whole build."

Lewis et al. define RAG with a retriever $p_\eta(z \mid x)$ that returns "(top-K truncated)" distributions over passages $z$ for an input $x$, and a generator $p_\theta$ that conditions on $x$ and $z$.1 For RAG-Sequence they write

p_{\text{RAG-Sequence}}(y \mid x) \approx \sum_{z \in \text{top-}k(p(\cdot \mid x))} p_\eta(z \mid x)\, p_\theta(y \mid x, z)

where top-$k(p_\eta(\cdot \mid x))$ is "the list of k documents z with highest prior probability", computed as a maximum inner product search over dense document and query encodings.1

Two consequences for a builder:

  1. The prompt only shapes $p_\theta$. What reaches the generator is decided upstream: which collections exist, which documents the requesting user may receive, and how $k$ passages are chosen.
  2. Top-$k$ is defined by rank, not by relevance. The retriever returns the $k$ highest-scoring passages whether or not any of them answers the question. Abstention is therefore a design decision the build must make explicitly (section 3).

NCSC's secure-design and secure-development guidance asks builders to threat-model the system, to "document the creation, operation, and life cycle management" of models, datasets and system prompts, and to have "processes and controls in place to manage what data AI systems can access".2 The permission matrix (section 2) and the spec (section 3) are this course's artefacts for those requirements in a retrieval build; the guidance does not prescribe their form.

2. Authorisation: the permission matrix and the test pair

Step 1, SET UP · COURSE STEP: Use synthetic fixtures. Run the same question as an allowed user and a denied user. Step 2, INSPECT · COURSE STEP: Read both traces. Check retrieved passages, sources shown and the answer against the permission. Step 3, DECIDE · COURSE STEP: Both runs must pass. Denied: no restricted passage, source or content, even in a harmless answer. Allowed: gets and cites it. No trace means no pass.
Course test procedure, using OWASP retrieval logging guidance (s1). The denied run must pass all checks; a person also reads answers for paraphrased disclosure.
Figure data as text

Test both users

  1. SET UP · course step · Use synthetic fixtures — run the same question as an allowed user and a denied user.
  2. INSPECT · course step · Read both traces — check retrieved passages, sources shown and the answer against the permission.
  3. DECIDE · course step · Both runs must pass — denied: no restricted passage, source or content, even in a harmless answer. Allowed: gets and cites it. No trace means no pass.

Course test procedure, using OWASP retrieval logging guidance (s1).

Objective: 300.2-a. Text equivalent of figures/test-both-users.svg.

Misconception: "If a document is in the index everyone can see it, if it is out it is safe, and checking the final answer is enough."

OWASP's 2025 list (as of November 2026) treats the retrieval store as an attack surface. LLM08 (Vector and Embedding Weaknesses) warns that "inadequate or misaligned access controls can lead to unauthorized access to embeddings", that "the model could retrieve and disclose personal data, proprietary information", and describes cross-context leaks "in multi-tenant environments where multiple classes of users or applications share the same vector database".3 Its mitigations start with "fine-grained access controls and permission-aware vector and embedding stores" and "strict logical and access partitioning of datasets in the vector database".3 LLM02 (Sensitive Information Disclosure) adds least privilege: "Only grant access to data that is necessary for the specific user or process."3

"Out of the index" is a narrow guarantee. indexed: false on a collection means this retrieval path will not ingest that source. It says nothing about copies of the same text in an indexed source (a case note pasted into a policy, a copy saved to an indexed folder), about content supplied in the user's input, or about other tools with access to the original. Excluding HR case files is a sound scope restriction; it is a claim to test, not an assumption. The exclusion check is in the test section below. (This is course reasoning, not a claim from the cited sources.)

The matrix as configuration

# permission-matrix.yml (synthetic, course sandbox)
collections:
  - id: service-manuals
    owner: service-docs-lead          # a named person in a real build
    audience: [staff, approved-dealers]
    authorisation: query_time_document_acl
  - id: hr-policies-published
    owner: hr-policy-owner
    audience: [staff]
    authorisation: query_time_document_acl
  - id: hr-case-files
    indexed: false                    # never ingested by this assistant
    exclusion_check: canary_phrases   # synthetic marker phrases from these fixtures must not
                                      # appear in any indexed collection (tested below)

Course rules for the matrix:

The allowed/denied test pair as assertions

Run one question as two synthetic principals and assert on the retrieved set, the citations and the answer text itself. A decision label is the system's self-report; do not assert on it alone.

# Synthetic users and documents only; never real employee records.
Q = "What is the policy on overtime?"
FACT = "ZX-41"     # synthetic fact planted only in the staff-only overtime fixture:
                   # "Overtime is booked under pay code ZX-41."
NO_ANSWER = SPEC["no_relevant_passage"]["message"]

def denied_failures(r):
    """Every reason a denied run fails; an empty list is a pass."""
    fails = []
    if any(p.collection.startswith("hr-") for p in r.passages):
        fails.append("restricted passage retrieved")              # retrieved set
    if any(c.collection.startswith("hr-") for c in r.citations):
        fails.append("restricted source shown")                   # sources shown
    if FACT in r.answer:
        fails.append("restricted content in answer")              # answer text
    # The dealer has no permitted source for this question, so the spec requires abstention.
    if r.answer.strip() != NO_ANSWER or r.citations:
        fails.append("not the spec's no-answer response")
    return fails

allowed = run(Q, user="test-staff-01")
denied  = run(Q, user="test-dealer-01")

# Allowed run: retrieved, cited and used.
assert any(p.collection == "hr-policies-published" for p in allowed.passages)
assert any(c.doc == "overtime-policy" for c in allowed.citations)
assert FACT in allowed.answer

# Denied run: every check must hold.
assert denied_failures(denied) == []

# Negative fixture: clean route metadata, restricted fact in the answer. The checker must reject it.
leak = Result(passages=[], citations=[], decision="abstain",
              answer="Overtime is booked under pay code ZX-41.")
assert "restricted content in answer" in denied_failures(leak)

# Exclusion check: canaries planted in the hr-case-files fixtures must not be in any indexed chunk.
CANARIES = ["QX-CASE-7731", "QX-CASE-9902"]    # synthetic
for coll in index.collections():
    for chunk in coll.chunks():
        assert not any(c in chunk.text for c in CANARIES), (coll.id, chunk.id)

run, SPEC, Result and index stand for the sandbox's harness; the names are illustrative. Two limits to state honestly. A literal fact or canary check catches copying, not paraphrase, so a person still reads the denied answers. And when the denied user does have a permitted source for the question, replace the last check in denied_failures with the expected permitted answer and citation.

Why the retrieved-set check carries the weight: once a passage is in the retrieved set it has been disclosed to the generator, and it is in the trace and the logs. NCSC asks builders to treat logs "as sensitive data" and protect their confidentiality, integrity and availability.2 OWASP recommends "detailed immutable logs of retrieval activities".3 A denied run with a benign answer and an hr-* passage in denied.passages is a fail, and the failing matrix line is that collection's authorisation.

At Big Widget Company. Lee's staff help desk build ingests hr-policies-published, but a folder of HR case notes was mis-filed into the same collection. test-dealer-01 asks about overtime; the answer cites only the overtime policy, but denied.passages contains a case-note chunk. Two defects: the ingestion scope (case files were meant to be indexed: false, and the exclusion check would have caught their canaries in hr-policies-published), and the authorisation (a query-time ACL check would have dropped a staff-only chunk for a dealer whatever collection it sat in).

In the sandbox (README Exercise 1)

The sandbox runs offline on the Python 3 standard library. From the repo root:

python3 apps/sandbox/sandbox.py retrieve --user test-staff-01 'What is the amber spindle repair step?'
python3 apps/sandbox/sandbox.py retrieve --user test-dealer-01 'What is the amber spindle repair step?'
python3 apps/sandbox/sandbox.py audit-index

Each invocation writes a JSON trace (path printed; default apps/sandbox/dist/traces/, or --trace-dir DIRECTORY before the subcommand). How the illustrative names above map to it:

Illustrative (this gear)Sandbox
permission-matrix.yml collection owner, audienceapps/sandbox/documents.json → collections.<id>.owner, .audience (roles)
document ACLeach passage's permission_labels; users map to roles in users
audience ∩ ACL, at query time, before generationretrieval.retrieve keeps a passage only if the role is in both, before ranking
indexed: falsethe collection is absent from the profile's collections in retrieval-config.json; single documents via excluded_documents
CANARIES and the exclusion loopexclusion_markers in documents.json; audit-index scans every indexed passage
r.passages, r.citations, r.answertrace keys passages, citations, answer

Assert on the dealer trace's passages, citations and answer (the marker "violet hinge jig" must not appear), not on decision. The default audit should report no hit for "copper teacup detour"; the README then has you add shared-folder to the service profile in a configuration copy and re-run audit-index --config YOUR_COPY.json, which finds the marker in a copy. The audit is a literal substring check and cannot catch paraphrase.

3. The spec, the configured path and the abstention trace

The spec is one page and is written before configuration:

# spec.yml (synthetic): Big Widget dealer service assistant
scope:
  answers: [service procedures, part numbers, warranty rules for current models]
  refuses: [prices, HR, legal advice, questions about a named person]
collections: [service-manuals, warranty-policy]       # rows from permission-matrix.yml
citation: "{title}, v{version}, section {section}"    # for every passage the answer relies on
relevance:
  check: <threshold and/or per-passage check>         # tuned on answerable and unanswerable tests
  on_fail: no_relevant_passage                        # routed in code, before generation
no_relevant_passage:
  rule: abstain                                       # the generator is not called (the
                                                      # test harness counts calls; see below)
  message: >-
    I couldn't find this in the service manuals or warranty policy.
    Please contact dealer support.
  citations: none
owner: service-docs-lead
tests: tests/dealer-assistant.yml                     # includes at least one unanswerable question

The spec's lines map to named risks. NIST AI 600-1 defines confabulation as "the production of confidently stated but erroneous or false content" and notes that outputs "may also include confabulated logic or citations that purport to justify or explain the system's answer".4 Its suggested action MS-2.5-003 is to "review and verify sources and citations in GAI system outputs" during pre-deployment risk measurement and ongoing monitoring, and MS-2.5-005 asks builders to verify that "retrieval-augmented generation data is grounded".4 It lists data privacy (leakage and unauthorised disclosure of personal or sensitive data) as a distinct generative-AI risk, which the refuses line narrows.4

Deciding "no relevant passage". Because top-$k$ always returns $k$ candidates,1 the build needs an explicit relevance decision between retrieval and generation: for example a score threshold, a separate relevance check on each passage, or both. Its parameters are tuned on the spec's own test questions, including unanswerable ones, and re-checked whenever the embedding model or the collections change. The route matters as much as the check: when the check fails, the pipeline returns the spec's message without calling the generator. An instruction in the system prompt ("say you couldn't find it") is a request to $p_\theta$, not an enforced route, and a model conditioned on weak passages may still answer. This is a course design rule; the cited sources do not prescribe a method.

A passing abstention trace (synthetic):

{
  "question": "What is the towing limit of the Model 12 widget?",
  "user": "test-dealer-01",
  "collections": ["service-manuals", "warranty-policy"],
  "passages": [
    {"doc": "service-manual-m7", "version": 3, "section": "4.2", "relevant": false},
    {"doc": "service-manual-m9", "version": 2, "section": "1.1", "relevant": false},
    {"doc": "warranty-policy",   "version": 5, "section": "2",   "relevant": false}
  ],
  "decision": "abstain",
  "answer": "I couldn't find this in the service manuals or warranty policy. Please contact dealer support.",
  "citations": []
}

The same passages with "answer": "2,000 kg (Service manual, section 4)" fail the spec: an unsupported value with a citation that does not support it.4

Prove the route, not just the words. Everything in that trace is written by the pipeline under test, including decision. A pipeline that ignores the relevance check, calls the generator with a "say you couldn't find it" prompt, and gets the fixed message back would produce the same answer, the same empty citations and, if it sets the label, the same decision. Its answer looks right; its route breaks the spec. So the harness, not the pipeline, observes the generator boundary: it wraps the generator in a call counter the pipeline cannot write to.

# Illustrative harness. The fixtures decide which questions are unanswerable:
# no synthetic document mentions the Model 12 or towing.
Q_NONE = "What is the towing limit of the Model 12 widget?"
Q_OK   = "How do I reset the service light on the Model 9 widget?"   # answered by a fixture
NO_ANSWER = SPEC["no_relevant_passage"]["message"]

def abstention_failures(r, generator_calls):
    """Every reason a no-answer run fails; an empty list is a pass."""
    fails = []
    if r.answer.strip() != NO_ANSWER:
        fails.append("not the spec's no-answer message")
    if r.citations:
        fails.append("citation on a no-answer")
    if generator_calls != 0:
        fails.append("generator was called")          # observed by the harness, not self-reported
    return fails

gen = CallCounter(generator)                           # harness-owned spy around the real generator
nofit = run(Q_NONE, user="test-dealer-01", generator=gen)
assert abstention_failures(nofit, gen.count) == []

# Positive control: an answerable question must reach the generator and cite its passage.
gen = CallCounter(generator)
ok = run(Q_OK, user="test-dealer-01", generator=gen)
assert gen.count == 1
assert ok.answer.strip() != NO_ANSWER
assert ok.citations and all(c.doc in {p.doc for p in ok.passages} for c in ok.citations)

# Negative fixture: right words, no citations, "abstain" label, but the generator was called once.
# This is the prompt-only build. The checker must reject it.
prompted = Result(passages=nofit.passages, citations=[], decision="abstain", answer=NO_ANSWER)
assert abstention_failures(prompted, generator_calls=1) == ["generator was called"]

CallCounter, run and Result are illustrative names, not the sandbox's API; any wrapper the pipeline cannot reset will do. Whether the positive control's answer matches its passage still needs the fixture's expected fact or a person reading it; a citation check alone does not show support. And a threshold strict enough to abstain on everything fails the positive control, so it fails the spec too. (Course design; the cited sources do not prescribe a harness.)

In the sandbox (README Exercise 2)

Copy apps/sandbox/retrieval-config.json and edit its warranty profile. The spec maps to it:

spec.yml (illustrative)warranty profile (sandbox JSON)
collectionscollections, plus top_k
citationcitation_format, e.g. "{title}, version {version}, section {section}"
relevance.checktopics (topic → trigger phrases) and min_score
relevance.on_fail / rule: abstainfixed in the runner: no sufficient passage → no model call, no_answer, no citations
no_relevant_passage.messageno_answer

The sandbox's relevance decision: the question must match exactly one configured topic, and a passage passes only if it carries that topic and shares at least min_score distinct words with the question. Ranking is by shared-word count over the permitted passages and returns the best top_k even when weak. Run the abstention case and the positive control on your copy:

python3 apps/sandbox/sandbox.py retrieve --config YOUR_COPY.json --profile warranty --user test-dealer-02 'Does the warranty cover flood damage to a Model 7 widget?'
python3 apps/sandbox/sandbox.py retrieve --config YOUR_COPY.json --profile warranty --user test-dealer-02 'How long do I have to make a warranty claim?'

Compare relevance, decision, model_called, answer and citations. Set min_score to 999 to reproduce the over-strict failure (workbook Trace C), then lower it and re-test both. The model is a scripted extractive stub (retrieval.model_stub) that joins the qualifying passages' text, so the sandbox never produces the workbook's fabricated Trace B; it tests routing and citation on synthetic fixtures, not a real model's behaviour.

model_called is not the harness count. The runner sets model_called from its own route decision (bool(evidence)), so it is the pipeline's self-report, exactly like decision. To observe the generator boundary from outside, wrap the stub. This script does that without editing the sandbox:

# count_calls.py: save it outside apps/sandbox/ and run it from the repo root:
#   python3 count_calls.py YOUR_COPY.json
import json
import sys

sys.path.insert(0, "apps/sandbox")
import retrieval

calls = []
real_model = retrieval.model_stub

def counted_model(passages):
    calls.append(len(passages))
    return real_model(passages)

retrieval.model_stub = counted_model    # retrieve() looks this name up on every call

with open("apps/sandbox/documents.json", encoding="utf-8") as f:
    fixtures = json.load(f)
with open(sys.argv[1], encoding="utf-8") as f:
    profile = json.load(f)["warranty"]

for question in ["Does the warranty cover flood damage to a Model 7 widget?",
                 "How long do I have to make a warranty claim?"]:
    calls.clear()
    trace = retrieval.retrieve(fixtures, profile, "test-dealer-02", question)
    print(len(calls), "|", trace["decision"], "|", trace["model_called"], "|", trace["answer"])

Run against the shipped retrieval-config.json on 20 Nov 2026, it printed 0 | no-answer path | False | … for the flood question and 1 | answer generated | True | … for the claim question: zero observed calls on the abstention case and one on the positive control. Apply the same three checks as abstention_failures above, using the observed count in place of gen.count; the retrieval.retrieve result is a dict, so read trace["answer"] and trace["citations"]. The script imports the sandbox's internal names retrieval.retrieve and retrieval.model_stub; it is a test-side spy, not a sandbox command. If those names change, or you do not run it, record the call count as "not checked" rather than reading model_called as proof.

Reading is not configuring. Reading the synthetic traces printed in the workbook practises the reading half of 300.2-b. The evidence for 300.2-b is your configuration copy and your own traces for both questions (and, for the stricter route proof, the observed call counts). Until you have them, record "configured in the sandbox: not yet", not a pass.

4. Freshness and duplicates

Misconception: "Once a document is indexed, the answers stay correct."

The answer distribution follows the index. Lewis et al. built indexes from December 2016 and December 2018 Wikipedia dumps and asked about 82 world leaders who changed between them.1 RAG answered 70% correctly with the 2016 index for 2016 leaders and 68% with the 2018 index for 2018 leaders; with mismatched indexes, 12% (2018 index, 2016 leaders) and 4% (2016 index, 2018 leaders).1 They conclude "we can update RAG's world knowledge by simply replacing its non-parametric memory".1 Read operationally: the non-parametric memory is the system's knowledge of your documents, so a stale index makes stale answers likely for questions that depend on the changed facts. It does not make any single answer certainly wrong, and the answer alone will not tell you which ones are.

Duplicates and versions compete in ranking. If v2 and v3 of a torque spec are both indexed, along with a second copy of v3, the top-$k$ set for a given phrasing may contain v2, v3 or both, and the generator may answer from either or blend them. The citation can be accurate and still point at a superseded document. OWASP LLM08 names this class "data federation knowledge conflict", errors that "can occur when data from multiple sources contradict each other".3 The mitigation it ties to combined sources is data review: "When combining data from different sources, thoroughly review the combined dataset", and "tag and classify data within the knowledge base to control access levels and prevent data mismatch errors".3 Its integrity audit and source-validation items target hidden codes and data poisoning, not version conflicts.

Metadata that makes freshness enforceable (course design, illustrative):

- doc_id: torque-spec-wheel-hub
  version: 3
  effective_from: 2026-10-01          # synthetic date
  supersedes: 2
  content_hash: <sha256 of normalised text>   # shared by exact duplicates
  sources:                                    # one record per place the text lives; never merged
    - path: <source path A>
      source_acl_ref: <pointer to A's ACL in the source system>
    - path: <source path B>
      source_acl_ref: <pointer to B's ACL in the source system>

Ingestion rules: index only the current version (or filter on effective_from and supersedes at query time); reject documents with no owner. De-duplicate storage, not identity: identical content_hash values may share one embedding, but every source record keeps its own path, version and ACL reference. At query time a passage is returned, and cited, only through a source record the user's current ACL allows. Permissions are never unioned across duplicates.

Keep two operations apart:

Test both with a synthetic pair: the same text at a staff-only path and a dealer-visible path. After compaction, a dealer still receives it, and only with the dealer-visible citation. After the dealer-visible record is retired, a dealer does not receive it at all, even though the staff-only copy and the shared embedding remain.

Each change to the collections, the embedding model, the generator or the prompt triggers the spec's test set, the allowed/denied pair and the exclusion check.

In the sandbox (README Exercise 3).

python3 apps/sandbox/sandbox.py retrieve --user test-dealer-01 'What is the gearbox oil-change interval?'

The trace's citations show gearbox-v3 (80 hours), gearbox-v4 (120 hours) and gearbox-v4-copy; compare version, section and path. The sandbox's ranking has no freshness term, so v3 competes on equal terms. Set "excluded_documents": ["gearbox-v3", "gearbox-v4-copy"] in the service profile of your configuration copy and re-run with --config YOUR_COPY.json: only canonical v4 remains. This filters the searchable fixtures; it does not model effective_from/supersedes metadata, storage compaction or per-source ACL records, and it is not a substitute for the owner's maintenance and re-test.

Ownership is the control. Sculley et al. observe that "data dependencies in ML systems carry a similar capacity for building debt" as code dependencies, "but may be more difficult to detect", and describe unstable inputs that "change behavior over time".5 Their mitigation, a versioned copy, has its own cost: "potential staleness".5 NCSC asks for processes to "track, authenticate, version control and secure your assets", and says update processes should reflect that "changes to data, models or prompts can lead to changes in system behaviour".2 NIST's MG-3.1-003 is to "re-assess model risks after fine-tuning or retrieval-augmented generation implementation".4

The course rule: every collection row names an owner who retires superseded versions on the day the new one is ingested, removes duplicates, runs the test set on every trigger above, and is the contact for a stale-citation report. For the torque conflict, that is the service-manuals owner, working with the engineer who owns the torque document; not a generic "AI team".

5. The three yeses for a retrieval build

Get to Yes · Stay at Yes · Recover to Yes, for a retrieval build:

Trace versus receipt. Traces and retrieval logs contain queries, passages and outputs: they are access-controlled work records. A course receipt records a gate decision (allow, constrain, hold, deny) and never carries prompts, outputs, payloads or personal data. Do not embed trace content in a receipt you share.

Words to know: retriever, generator, top-$k$, maximum inner product search, collection, access-control list (ACL), query-time authorisation, principal, allowed/denied test pair, negative fixture, canary, exclusion check, trace, relevance decision, abstention, generator call counter, positive control, knowledge conflict, superseded version, content hash, source record, storage compaction, retiring a source record, collection owner, hold, receipt.

Further learning (optional, advanced, not part of this lesson): Carnegie Mellon University, Machine Learning in Production (free book and public course materials): <https://mlip-cmu.github.io/>. Carnegie Mellon University, 11-711 Advanced Natural Language Processing (Spring 2025; public syllabus, slides and readings): <https://cmu-l3.github.io/anlp-spring2025/>.

Sources

  1. s3 Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Paper. Checked 2026-11-19 by source-checker (Claude Opus 5.5, round-3 re-check session; did not write or edit module 300.2). ↩ ↩ ↩ ↩ ↩ ↩
  2. s2 UK National Cyber Security Centre and partner agencies (2023). Guidelines for secure AI system development. Official doc. Checked 2026-11-19 by source-checker (Claude Opus 5.5, round-3 re-check session; did not write or edit module 300.2). ↩ ↩ ↩ ↩
  3. s1 OWASP GenAI Security Project (2025). OWASP Top 10 for LLM Applications 2025. Official doc. Checked 2026-11-19 by source-checker (Claude Opus 5.5, round-3 re-check session; did not write or edit module 300.2). ↩ ↩ ↩ ↩ ↩ ↩ ↩
  4. s5 National Institute of Standards and Technology (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). Official doc. Checked 2026-11-19 by source-checker (Claude Opus 5.5, round-3 re-check session; did not write or edit module 300.2). ↩ ↩ ↩ ↩ ↩
  5. s4 D. Sculley, Gary Holt, Daniel Golovin, et al. (2015). Hidden Technical Debt in Machine Learning Systems. Paper. Checked 2026-11-19 by source-checker (Claude Opus 5.5, round-3 re-check session; did not write or edit module 300.2). ↩ ↩

Next: the quiz