
How retrieval works · Standard gear
Reviewed and published, version 1.0.0. As of 19 Nov 2026. It is not a credential. This module teaches research findings, not product facts; findings are dated by their papers.
You will be able to:
- explain retrieval-augmented generation as two steps: finding passages in a chosen collection, then having the model write an answer from them (300.1-a);
- explain what retrieved passages change inside the model's input, and what they cannot guarantee (300.1-b);
- tell whether a wrong answer came from retrieval (the passages needed were not found) or from generation (they were found, and the answer is still wrong) (300.1-c).
Big idea: retrieval does not teach a model your documents. The assistant looks some up, adds them to the question, and writes. Each of those steps can fail on its own, and with the trace you can usually tell which one did.
In Level 100 (module 100.6) you met the open-book tool: it finds company documents first, then answers. This module opens the box. In 300.2 you build one.
1. Two steps: find, then write
Figure data as text
Find, then write
- FIND [s1] · Search the chosen collection — the retriever returns passages relevant to this question.
- WRITE [s1] · Write from the input — the model receives the question and retrieved passages, then writes an answer.
Objective: 300.1-a. Text equivalent of figures/find-then-write.svg.
Retrieval-augmented generation (RAG) joins two parts. The 2020 paper that named it describes a retriever, which looks up passages in a document index, and a generator, a language model that writes its output from the input together with the passages the retriever returned.1 In that paper the index was Wikipedia, held outside the model as what the authors call a "non-parametric" memory, next to the "parametric" memory stored in the model's weights.1
So a RAG system answers in two steps:
- Find. The retriever searches a chosen collection (the documents someone decided to put in the index) and returns the few passages that score as most relevant to the question.
- Write. The model receives the question and those passages, and writes an answer.
The misconception this corrects: "The model has read our documents and remembers them." Retrieval does not work that way. The collection sits in the index, outside the model's weights, and retrieving a passage does not train the model on it. When it writes, the model draws on three things: your input (the question and any instructions or text you supplied), what it learned in training, and the passages retrieved for this question. A document the model did not learn in training, and that you did not supply, reaches it only if the find step returns it.
The RAG authors showed how much the index matters. When they swapped the Wikipedia index for one from a different year, the system's answers about world leaders changed to match the index, with no retraining of the model.1 That shows answers can be updated by changing the collection. It does not show that every answer always follows the index, or that the model knows nothing else.
Each step can be measured on its own. The RAG paper's retriever is based on Dense Passage Retrieval,1 and the Dense Passage Retrieval paper studies the find step by itself. It scores a retriever by whether a passage holding the answer is among the top passages it returns, before any model writes anything.2 That separation is the basis of section 3.
At Big Widget Company: the missing policy page. Big Widget is a made-up company. Its service assistant answers dealer questions from a collection of service policies. The service team replaces the warranty-labour policy. The new policy was written after the model was trained, and nobody pastes it into a question. Nobody adds it to the collection either.
In our scenario, this is what the trace and the screen show:
- A dealer asks about warranty labour. The find step cannot return a page that is not in the collection. It returns the old policy.
- The write step answers fluently from the old policy and cites it. NIST calls the general risk confabulation: "the production of confidently stated but erroneous or false content".3
- The dealer sees a confident answer, with a citation to the old page, and no warning that the new page was never there.
The root failure is in the first step, and specifically in the collection. Rewording the prompt cannot retrieve a page that is not in the collection. An instruction to say "I could not find a current policy" might stop the confident wrong answer; that limits the damage but does not fix coverage.
The fix has two parts. First, add the new page and retire the old one, and name who keeps the collection current (300.2 covers who that is). Second, test it: re-ask the failing question, check that the new page is retrieved, and check that the answer matches it. Adding the page fixes coverage; only the retest shows the answer is fixed.
2. What the passages change, and what they cannot guarantee
They become more text in the input
Retrieval does not change the model. It changes the model's input. In the RAG paper, the generator combines the input and a retrieved passage by simply joining them into one sequence.1 This course uses that picture for every retrieval assistant: the passages become more text in the model's context window, alongside the question and any instructions, and the model writes from all of it. Whether a particular product does exactly this is something to check in its documentation, not assume.
"Does not change the model" is about the model's weights. It is not a promise about storage. The application around the model may keep the question, the retrieved passages and the answer in logs or conversation history, and a provider may have its own rules about later use. Check the tool's retention policy, or ask its owner; do not assume either way.
They are used unevenly by position
Text in the context window is not all used equally. In Lost in the Middle, researchers gave models a question and a set of documents, only one of which held the answer, and moved that document to different positions. Performance was often highest when the relevant information was at the beginning or end of the input, and it dropped significantly when the model had to use information in the middle of a long input. This held even for models built for long contexts.4
So a retrieval assistant that returns twenty passages has not "given the model the answer" just because the right passage is somewhere among them. Where it sits matters. The study measured specific models in 2023 on research tasks; it shows the effect exists and is worth testing for, not how large it is in any given product.4
What they cannot guarantee
Adding the right passage to the input raises the chance of a right answer. It does not guarantee three things:
- That the model reads it correctly. The model can still misread, mix up or go beyond the passage. Confabulation is a risk of generative AI in general, not only when nothing was retrieved.3
- That the model uses a passage in the middle of a long input. See Lost in the Middle, above.4
- That the model treats an instruction inside a passage as data. Researchers who studied indirect prompt injection argue that applications built on language models "blur the line between data and instructions". They showed that attackers can plant instructions in content an application is likely to retrieve, such as web pages, without ever talking to the system directly, and that this worked against real applications.5 A retrieved passage is text in the same context window as the system's own instructions. To take an illustration of our own: a line in a supplier document that says "ignore the other passages" may be followed. So an instruction in the system prompt is not a security boundary. The course's rule: no direct path from the model to a consequential action, such as sending or changing something; a gate outside the model stands between them. Module 300.6 covers the attack and the defences.
Why a citation does not prove the answer
The misconception this corrects: "If it cites a document, the answer is right." A citation in an answer is a claim, and it can fail in two ways.
- The citation itself can be wrong. A citation the model writes is generated text. NIST warns that generative AI outputs "may also include confabulated logic or citations that purport to justify or explain the system's answer".3 The course's reading of that warning: a generated citation can name a passage that was never retrieved, or a document that does not exist. Some tools attach citations from the retrieval step rather than letting the model write them; check which yours does, and do not assume.
- A real, retrieved passage may not support the answer. Researchers who build evaluation tools for RAG treat this as its own question. Their faithfulness score asks whether the claims in an answer can be inferred from the retrieved context. It is measured on its own, apart from whether the right context was retrieved.6
The RAG authors listed providing provenance for a model's decisions as an open research problem, and presented retrieval as a way to help with it.1 The course's own conclusion: retrieval makes checking possible; it does not do the checking for you.
In practice, check a citation in this order:
- Exists and was retrieved. Is the cited passage in the trace, among the passages actually retrieved for this question?
- Supports. Does that passage say what the sentence citing it says?
- Current. Is it the current version of the document?
3. Was it retrieval or generation?
The misconception this corrects: "Every wrong answer is a hallucination, and nothing more can be said." Calling every wrong answer a hallucination names the symptom and stops there. In a RAG system you can usually say which step failed, as long as you can see the trace: the question, the passages the find step returned, and the answer.
RAG evaluation research separates these on purpose. RAGAs, for example, scores context relevance (did the retrieved passages fit the question?) apart from faithfulness (does the answer stick to those passages?) and answer relevance (does it address the question?).6 The Dense Passage Retrieval paper measures the find step alone, before any answer is written.2
Ask one question of the trace: were all the passages needed for a right answer among the passages found?
| The trace shows | Which step failed first | What to fix |
|---|---|---|
| A needed passage is not among those found (wrong passages, an old version, none at all, or only some of the passages the answer needs) | Retrieval | The collection (is the page in it, and current?) or the search (does it find the page for this wording?) |
| All the needed passages are among those found, and the answer is wrong, unsupported or ignores them | Generation | How passages are given to the model (how many, in what order), the instructions, or the model; and keep checking answers against passages |
Three notes on the table:
- Some answers need more than one passage. If the answer needs the warranty policy and the list of covered parts, finding only the policy is still a retrieval failure. The table tells you where to look first. It is not a full causal proof: both steps can fail on the same question.
- No passage, answer anyway. If nothing relevant was found and the system still answered, the root failure is retrieval; answering anyway is a second problem. What the system should say when nothing relevant is found is part of the spec you write in 300.2.
- The passage was there, but buried. If the right passage was found but sat in the middle of a long list and the answer missed it, that is a generation failure. The fix may still be on the retrieval side: return fewer, better-ranked passages.4
At Big Widget Company: two traces. Both are made up for this lesson, and each question needs only one passage.
Trace A. Question: "How many days does a dealer have to file a warranty claim?" Passages found: (1) Travel and expenses policy, section 2; (2) Canteen opening hours. Answer: "Dealers have 30 days to file a warranty claim." Retrieval failure. Neither passage is about warranty claims, so the answer cannot have come from them. The model wrote a plausible number anyway.
Trace B. Question: the same. Passages found: (1) Warranty policy, section 4: "Dealers must file a claim within 60 days of the repair date." Answer: "Dealers have 30 days to file a warranty claim (Warranty policy, section 4)." Generation failure. The right passage was found and even cited, and the answer still contradicts it. This is the citation misconception from section 2 in one line.
If you cannot see the trace, say so. Without the passages found, you cannot tell the two failures apart. "I don't know yet: the tool does not show what it retrieved" is an honest finding. The next step is to ask the tool's owner for the trace.
4. The three yeses for retrieval
These are the retrieval parts of the three yeses: Get to Yes · Stay at Yes · Recover to Yes.
The assistant decides none of them. A named person does, at a gate, with one of four answers: go ahead (allow), only this far (constrain), wait for a person (hold) or no (deny). A yes is conditional: it covers the collection, questions and checks it names, and nothing more.
- Get to Yes (before anyone relies on it). Who chose the collection, and which named person keeps it current? Is the page you need in it? Does the tool show the passages it found, not only a citation? Can you see a trace? Do you know what the tool keeps in its logs, or do you know that you do not? Only this far might read: only questions this collection covers, and a person checks the cited passage before an answer goes to a dealer.
- Stay at Yes (while it runs). Are you checking that cited passages were really retrieved, and reading the sentence each one supports? Has the collection changed? If so, the answer is wait for a person: re-test the questions that matter before you rely on it again. If someone quotes how often the assistant "finds the right page", ask "out of how many?": how many questions, on which collection, measured when, and did "found" mean all the passages needed?
- Recover to Yes (when it goes wrong). Hold the affected questions: stop relying on the assistant for them. Read the trace. Retrieval or generation? Tell the owner of that step; they fix it. Re-test the question that failed and keep it as a regression question. A fix is not a recovery until the re-test shows the right passage retrieved and the right answer; show that evidence before you say "recovered". If there is no trace, the honest status is "I don't know yet", and the next step is to ask the tool's owner for one.
Trace or receipt? They are different things. The trace holds the question, the retrieved passages and the answer, so it is a private work record: keep it access-controlled, and only where your workplace's policy allows. A receipt, in this course, records the gate's decision (go ahead, only this far, wait for a person, no). It never holds prompts, outputs, payloads or personal data. Do not paste a trace into a receipt you share. A receipt shows what was decided; it does not show that an answer was right.
My checklist:
- I can say which collection this assistant searches, who chose it, and who keeps it current.
- I know who can say yes, or wait, for this assistant, or I know that I do not.
- I check that a cited passage was retrieved, and I read it before I trust the sentence.
- When an answer is wrong, I look at the passages found before I call it a hallucination.
- When I cannot see the trace, I say "I don't know yet" and ask the tool's owner for it.
Words to know: retrieval-augmented generation (RAG), retriever, generator, collection, index, passage, context window, trace, retrieval failure, generation failure, faithfulness, confabulation, indirect prompt injection, gate, receipt.
Further learning (optional, advanced, not part of this lesson):
- Stanford University, CS224N Natural Language Processing with Deep Learning (partial public materials): <https://web.stanford.edu/class/cs224n/>
- Carnegie Mellon University, 11-711 Advanced Natural Language Processing (Spring 2025): <https://cmu-l3.github.io/anlp-spring2025/>
- University of Oxford, Deep Natural Language Processing (2017 lectures): <https://github.com/oxford-cs-deepnlp-2017/lectures>