Try each exercise first, then open its model answer.
Building a retrieval assistant · Workbook
Three exercises, one for each thing this module teaches. Everything here is set at Big Widget Company, a made-up company. Its people, documents, test users, marker phrases and traces are made up too. Use only what the lesson taught. You do not need to look anything up.
Run the exercises in the course's offline sandbox. apps/sandbox/README.md (section "300.2 · Building a retrieval assistant") gives the exact commands for each exercise, starting with python3 apps/sandbox/sandbox.py --help from the repo root. The traces printed below are synthetic examples for reading practice. Reading them does not show that you configured a working path, so on its own it does not meet objective 300.2-b: for that, configure your own copy of the sandbox's warranty profile and keep your own traces for both test questions. Until you have, write "configured in the sandbox: not yet", not a pass. Never use a real employee's records, or real company documents, as test material.
Exercise 1 (300.2-a)
Priya is building a parts-and-service assistant for Big Widget Company. It will answer staff and approved dealers. She has four sets of documents:
- Service manuals. Owned by the service documentation lead. Most are open to all staff and approved dealers, but a few engineering notes inside the same folder are open to staff only.
- Dealer price lists. Owned by the sales operations manager. Open to all staff and to approved dealers.
- Supplier contracts. Owned by the head of purchasing. Open to the purchasing team only.
- Supplier dispute notes. Kept by the purchasing team. Open to the purchasing team only. Buyers sometimes copy paragraphs from them into other documents.
Priya's manager says three things:
- (i) "Index the supplier contracts too. Everyone at the company can see what's in the index anyway, so it doesn't matter."
- (ii) "Before go-live, read the answers the dealer test account gets. If no answer mentions a contract, we're fine."
- (iii) "Leave the dispute notes out of the index. Then they're safe, and there's nothing to check."
- Write Priya's permission matrix: one row per set of documents, with the four columns from the lesson.
- For each of the manager's three statements, say what is wrong with it.
- For statement (iii), say what Priya can do to check that "out" is really out, and what she writes if she cannot run that check.
- A dealer asks about a repair step that one of the staff-only engineering notes covers. What should happen, and which rule decides it?
- Write an allowed/denied test pair for the engineering notes. Say who the two test users are, the one question both ask, and the three things you check in the denied user's trace. How do you check the answer's actual words?
- The denied run gives a polite, harmless answer. But the trace lists a passage from an engineering note. Pass or fail? Which line of the matrix failed, and what do you do?
Model answer
A matrix along these lines:
Collection Owner (a named person) Who may see answers from it How access is checked Service manuals Service documentation lead All staff and approved dealers At query time, against each document's own access list Dealer price lists Sales operations manager All staff and approved dealers At query time, against each document's own access list Supplier contracts Not indexed Nobody, through this assistant Not in any collection this assistant searches Supplier dispute notes Not indexed Nobody, through this assistant Not in any collection this assistant searches; the collections are searched for copies Putting the supplier contracts in a collection for the purchasing team only would also be defensible, but it is outside this assistant's job and adds risk for no gain. Either way, every indexed collection has a named owner.
- (i) "Everyone can see what's in the index." Wrong: in the index does not mean everyone may see it. Access can and should be checked for each user. But in the index does mean it can be retrieved unless access is checked. OWASP warns that weak access controls can let a model retrieve and disclose proprietary information.
- (ii) "Read the answers." Wrong: checking only the final answer misses the leak. A passage that was retrieved has already been handed to the model, and can show up in the sources panel, in the logs or in the answer to the next, slightly different question. The test has to check the passages and the sources too.
- (iii) "Out of the index, so safe." Wrong: out of the index is a narrow promise. It says this search step will not find that file. It says nothing about copies of its text (here, paragraphs buyers copy into other documents, which might sit in an indexed folder), text a user pastes into a question, or other tools that reach the original. Leaving the notes out is still a good scope decision. It is something you check, not something you assume.
- (i) "Everyone can see what's in the index." Wrong: in the index does not mean everyone may see it. Access can and should be checked for each user. But in the index does mean it can be retrieved unless access is checked. OWASP warns that weak access controls can let a model retrieve and disclose proprietary information.
- Put a few distinctive made-up phrases into the synthetic dispute notes used for testing, then search every collection the assistant uses for those phrases. A hit means a copy got in. Repeat the search whenever a collection changes. If Priya cannot run the search, the honest status is "not checked", and she names the collection owner as the person who can run it.
- The dealer does not get the engineering note. The rule: the narrowest list wins. The row says dealers may see answers from the service manuals, but that can only narrow a document's own access list, never widen it. Because access is checked at query time against each document's own list, the staff-only note is never handed to the model for a dealer's question.
A good test pair:
- Allowed user: a synthetic staff test account (for example
test-staff-01). - Denied user: a synthetic dealer test account (for example
test-dealer-01). - Question: the same for both, worded to match the engineering note, for example a question about the repair step it covers.
- Three checks on the denied user's trace, all of which must hold:
- Passages retrieved: no passage from a staff-only engineering note.
- Sources shown: the engineering note is not cited or named.
- Answer: contains nothing from it, and says it cannot help with that.
- The answer's actual words: a label in the trace such as "answered from permitted sources" is the system's own report, not a check. Put a made-up marker phrase or fact in the synthetic engineering note, and check that it never appears in the denied user's answer. A marker check catches copying, not paraphrase, so a person still reads the denied answers.
The allowed user's trace should show the note retrieved and cited, and an answer that matches what the note says, so you know the test question really reaches it.
- Allowed user: a synthetic staff test account (for example
- Fail. A harmless answer with a restricted passage in the trace still fails. The line that failed is the service manuals row's "how access is checked": the query-time check let a staff-only passage through. Put the collection on hold (stop answering from it) and tell its owner and whoever Big Widget names for data incidents. Fix the access check so it uses each document's own current access list, re-run the test pair, and use the retrieval logs to work out who saw what. If the logs cannot tell you, write "unknown" and name who will find out; do not assume nobody saw it. The collection owner lifts the hold only after the re-test passes. A better prompt is not the fix.
Exercise 2 (300.2-b)
Sam is building a warranty assistant for Big Widget dealers.
- Write Sam's one-page spec. Include: the questions it answers, the questions it does not answer, the collections it searches, the citation it shows, the relevance check, what it says when nothing relevant is found, and the owner.
- List the five steps Sam takes to configure the path in the sandbox, and say which test questions prove the no-answer case works and the answerable case still works.
- Read these two synthetic traces for the same question. Which one meets the spec? Give three reasons from the trace.
TRACE A
question: "Does the warranty cover flood damage to a Model 7 widget?"
user: test-dealer-02 (synthetic)
collections: warranty-policy
passages: 3 returned; they cover wear parts, shipping damage and claim deadlines; none mentions flood
relevance: check failed (no passage passed)
decision: no-answer path
model calls: 0 (counted by the test harness)
answer: "I couldn't find this in the warranty policy. Please contact dealer support."
citations: none
TRACE B
question: "Does the warranty cover flood damage to a Model 7 widget?"
user: test-dealer-02 (synthetic)
collections: warranty-policy
passages: 3 returned; they cover wear parts, shipping damage and claim deadlines; none mentions flood
relevance: no check configured
decision: answer generated
model calls: 1 (counted by the test harness)
answer: "Yes, flood damage is covered for 12 months (Warranty policy, section 3)."
citations: Warranty policy, section 3
- Sam then runs a question the warranty policy does answer. Does this trace meet the spec? Why or why not?
TRACE C
question: "How long do I have to make a warranty claim?"
user: test-dealer-02 (synthetic)
collections: warranty-policy
passages: 3 returned; one is the claim-deadlines section, which states the deadline
relevance: check failed (no passage passed)
decision: no-answer path
model calls: 0 (counted by the test harness)
answer: "I couldn't find this in the warranty policy. Please contact dealer support."
citations: none
- A teammate rebuilds the path a different way and re-runs the flood question. Does Trace D meet the spec? What would you write if the harness could not count model calls at all?
TRACE D
question: "Does the warranty cover flood damage to a Model 7 widget?"
user: test-dealer-02 (synthetic)
collections: warranty-policy
passages: 3 returned; they cover wear parts, shipping damage and claim deadlines; none mentions flood
relevance: check failed (no passage passed)
decision: no-answer path
model calls: 1 (counted by the test harness)
answer: "I couldn't find this in the warranty policy. Please contact dealer support."
citations: none
- Sam's teammate says, "We don't need a relevance check. We'll put 'say you couldn't find it' in the prompt, and anyway the search only returns passages when it finds a match." Explain what is wrong with both halves.
Model answer
- A spec along these lines:
Line Sam's warranty assistant Questions it answers What the warranty covers, for how long, and how to make a claim, for current widget models Questions it does not answer Prices, HR, legal advice, anything about a named person Collections it searches Warranty policy (a row in the permission matrix) Citation shown Document title, version and section for every passage the answer uses Relevance check How the system decides the passages found are good enough to answer from; tested on questions it should answer and questions it cannot When nothing relevant is found The check fails, no answer is written, and it returns: "I couldn't find this in the warranty policy. Please contact dealer support." No guess, no citation. Owner A named person who keeps the warranty policy collection current
- Choose the collections (only those in the spec and the matrix). Set the citation format. Set the relevance check, and connect its failure to the no-answer path. Set the no-answer message, word for word. Run the spec's test questions: at least one the collections cannot answer, whose trace proves the no-answer case, and at least one they can answer, whose trace proves the check does not refuse everything.
Trace A meets the spec:
- the passages really were irrelevant (none mentions flood);
- the relevance check failed and the no-answer path ran;
- the test harness counted zero model calls, so the model was never asked to write anything;
- the answer is the spec's wording, makes no claim about cover, and cites nothing.
Trace B fails. It has no relevance check, so it wrote an answer from passages that say nothing about floods: a confident "yes" with a 12-month period and a section number. NIST calls confidently stated but false content confabulation, and warns that it can include made-up citations. The citation points at a section that does not support the answer.
- No, Trace C fails the spec. The question is one the spec says it answers, and a retrieved passage states the deadline, yet the relevance check failed and the no-answer path ran. A check so strict that it refuses answerable questions also fails the spec. A passing trace would show the check passed, one model call, an answer that matches the claim-deadlines passage, and a citation to that passage. Sam adjusts the check and re-runs both kinds of test question.
- No, Trace D fails the spec. Its answer, citations and decision line look exactly like Trace A's. But the harness counted one model call, so the model was asked to write even though the check failed. That is what a build looks like when it sends the weak passages to the model with "say you couldn't find it" in the prompt and gets the fixed message back. The decision line is the system's own report; the count kept by the test harness, outside the system, is the evidence. If the harness could not count calls, the honest status of the no-answer route is "not checked", not a pass. Name who can add the count (the assistant's owner or whoever runs the harness).
"Put it in the prompt." An instruction in the prompt is a request to the model, not a control: the model may still answer from weak matches. The control is a relevance check plus a no-answer path that skips writing an answer when the check fails, and only tests on answerable and unanswerable questions show they work.
"The search only returns passages when it finds a match." In the 2020 paper that named retrieval-augmented generation, the search step returned the top few passages by score. That list is defined by rank alone, so in that design it is not empty just because nothing fits, and nothing guarantees any passage is relevant. Without a relevance check, the model may write from weak matches or from what it learned in training, and the answer looks just like a real one. If Sam's search does use a score cut-off, that cut-off is itself a relevance check, and it still has to be set and tested.
Exercise 3 (300.2-c)
The Big Widget dealer service assistant searches the service manuals collection. Last month:
- Engineering issued version 4 of the gearbox manual with a new oil-change interval.
- Version 3 is still in the collection.
- A dealer saved a copy of version 4 to a shared folder, and that folder is in the collection too.
- A dealer asks for the oil-change interval. What answers might come back, and why?
- The answer cites "Gearbox manual, version 3, section 5". Is the citation fake? Is the answer safe to use?
- A colleague says, "Once a document is indexed, the answers stay right." Use what the RAG researchers found when they swapped indexes to answer them.
- Who fixes this, and what exactly do they do? Who does not own the fix?
Model answer
- The old interval, the new one, or a mix of the two. In this example the search returns whichever passages score highest for the way the question is worded. Being newer does not make a passage score higher, so it may find version 3, version 4, the extra copy of version 4, or a mix. OWASP warns that errors can occur when data from multiple sources contradict each other.
- Version 3 is a real document, so the citation may point at something that exists. That alone does not prove section 5 says what the answer says: a citation is what lets you check, and NIST's suggested action is to review and verify sources and citations. Either way, version 3 is superseded, so do not use its interval. A real citation can still point at the wrong document.
- The answers follow the index. The researchers who named retrieval-augmented generation built one index from an older Wikipedia and one from two years later, and asked about 82 world leaders who had changed in between. With the index that matched the year, the system got most answers right (70% and 68%). With the wrong year's index, it got very few right (12% and 4%). The good news is that you can update what the system knows by swapping the documents, with no retraining. The warning is the mirror image: if old documents stay in the index, old answers are a real risk. Not every answer will be wrong, but you cannot tell from the answer alone which ones are.
The owner of the service manuals collection, working with the engineer who owns the gearbox manual. The owner:
- removes or marks version 3 the day version 4 goes in;
- deals with the duplicate copy without changing who may see the text. The shared folder may have a different access list from the original's. Tidying the index's storage keeps each copy's own access list and source. Retiring the shared-folder copy is different: it deliberately removes that route to the text, so the owner decides it on purpose and re-runs the allowed/denied test afterwards;
- re-runs the spec's test questions after the change (and after any change to the model or the prompt);
- is the person dealers contact when an answer cites something out of date.
Not "the AI team", and not the dealer who noticed. Sculley and colleagues found that data a system depends on builds up costs that can be harder to detect than code, which is why keeping a collection current needs a named person, not a setting.