
Grounding it in your documents · Technical gear
Reviewed and published, version 1.0.0. As of 18 Nov 2026. Not a credential. Research results are dated in the text.
Objectives. 200.3-a: explain what grounding changes (generation conditioned on documents the user supplies or a retriever returns) and what it does not fix. 200.3-b: elicit answers with claim-level quotation and location, and verify each quotation against the source of record. 200.3-c: diagnose grounded-answer failures caused by the corpus (contradiction, staleness, gaps), assign ownership of the fix, and escalate failures whose sources look correct. Localising a failure to retrieval or generation is Level 300 and out of scope here.
Big Widget Company and its documents are fictional.
1. Grounding: three channels of evidence
At inference a model conditions on its weights (parametric memory) and on its context window. Grounding adds evidence to the context in one of two ways:
- User-supplied context. The user attaches or pastes a document; the model conditions on the question and that text.
- Retrieval. A retriever selects passages from a corpus and the generator conditions on them.
Lewis et al. (2020) introduced retrieval-augmented generation (RAG): a parametric generator paired with a non-parametric memory, a dense vector index of Wikipedia queried by a neural retriever.1 Given input $x$, the retriever $p_\eta(z \mid x)$ returns the top-$k$ passages $z$, and in the RAG-Sequence form the output marginalises over them:1
p(y \mid x) \;\approx \sum_{z \,\in\, \text{top-}k\left(p_\eta(\cdot \mid x)\right)} p_\eta(z \mid x)\, \prod_{i} p_\theta(y_i \mid x, z, y_{1:i-1})
Their motivation is the point for practitioners: parametric models' ability to access and precisely use stored knowledge is limited, and providing provenance for outputs and updating world knowledge were open research problems.1 Their index hot-swap test on world leaders showed that replacing the document index with a newer one changed answers without retraining the generator.1 These are Wikipedia benchmarks, not evidence about enterprise-corpus answer quality.
What grounding changes. The evidence is now your corpus, and it can be updated without touching the weights. What it does not change. Nothing in the formula above forces $y$ to be entailed by $z$. The generator conditions on the passages; it is not constrained to them.
2. Failure modes that survive grounding
The misconception is "if I attach the document, the answer comes from the document". Three classes of failure remain.
2.1 Uneven use of context. Liu et al. (arXiv 2023; TACL 2024) ran a multi-document QA task with input contexts of 10, 20 and 30 documents, exactly one of which contains the answer, and varied that document's position.2 Models tested included GPT-3.5-Turbo, GPT-3.5-Turbo (16K), Claude-1.3, Claude-1.3 (100K), MPT-30B-Instruct and LongChat-13B (16K).2 Performance was highest when the relevant information was at the beginning or end of the context, giving a U-shaped curve over position.2 GPT-3.5-Turbo's multi-document QA performance could drop by more than 20% depending on position, and in the 20- and 30-document settings its worst-case performance fell below its closed-book score of 56.1% (against 88.3% when given only the answer-bearing document, the "oracle" setting).2 Extended-context models were "not necessarily better than their non-extended counterparts at using their input context".2 These are 2023 models; the result is evidence that conditioning on a passage does not guarantee using it, not a measurement of current systems.
2.2 Unfaithful generation. NIST AI 600-1 defines confabulation as "the production of confidently stated but erroneous or false content".3 Section 2.2 extends it: "Confabulations also include generated outputs that diverge from the prompts or other input or that contradict previously generated statements in the same context."3 For grounded systems, that is the failure where the output is not supported by, or goes beyond, the supplied passages.
2.3 Corpus defects. A faithful generator reading a contradictory, stale or incomplete corpus produces a faithful wrong answer. See section 5.
Worked example (fictional). Corpus: Big Widget returns policy, "This policy covers all returns, including returns from dealers. Unused parts may be returned within 30 days of delivery with the original invoice. Custom-built widgets cannot be returned." Query: return an unused part at day 40?
| Output | Entailed by the passage? | Failure class |
|---|---|---|
| "No; unused parts must be returned within 30 days of delivery." | Yes | none |
| "No, but store credit is available with a 15% restocking fee." | No; plausible but unsupported addition | 2.2, extrinsic content |
| "Yes, with the original invoice." | No; drops a condition, contradicts the passage | 2.2, contradiction of input |
3. Eliciting checkable answers
Figure data as text
Quote, point, verify
- Quote and point (course step) — ask for exact words, the document version or date, and the page or section. Ask for “not found” if there’s no answer.
- Open the source — find the exact quote and read around it. Check that it supports the claim. [s4]
Objective: 200.3-b. Text equivalent of figures/quote-point-verify.svg.
Require claim-level attribution in the instruction:
Answer only from the attached document(s).
For each claim: quote the supporting sentence verbatim in quotation marks,
and give document title, version/date, and section.
If the documents do not answer the question, reply "not in the documents".
This addresses the provenance problem Lewis et al. name, at the interface rather than in the model.1 It is a request, not a guarantee: the quoted span and the location are themselves model output and must be verified.
4. Verifying attributions
The misconception is that a citation in the output implies support in the source. NIST AI 600-1: "GAI outputs may also include confabulated logic or citations that purport to justify or explain the system's answer, which may further mislead humans into inappropriately trusting the system's output."3 Its suggested action MS-2.5-003 is to "Review and verify sources and citations in GAI system outputs during pre-deployment risk measurement and ongoing monitoring activities."3
For each claim $c_j$ with quoted span $q_j$ and location $\ell_j$, check against the source of record $D$ (the current published version, not the model's copy):
- Existence. $q_j$ occurs verbatim in $D$ at or near $\ell_j$.
- Context. The surrounding text of $q_j$ in $D$ does not qualify or reverse it (exceptions, scope, effective dates).
- Support. $q_j$ entails $c_j$. This is the check most often failed: a real quote attached to a claim it does not support (the restocking-fee row above).
- Coverage. Every claim has an attribution; unattributed claims are treated as unsupported.
Because position effects are documented (section 2.1), verify every claim in long outputs, not a sample from the opening.2 Deciding the sampling rate for a task (spot checks to full checking) is 200.4.
5. Corpus defects and ownership
The misconception is that wrong answers are always the model's fault. A corpus can be:
- Inconsistent: two documents assert incompatible facts; both attributions verify.
- Stale: a superseded version remains retrievable or was supplied.
- Incomplete: no document covers the query; the generator fills the gap unless told to abstain.
NIST AI 600-1 section 2.8 describes high-integrity information as able to "be linked to the original source(s) with appropriate evidence" and as creating "reasonable expectations about when its validity may expire".3 Relevant suggested actions: verify "that fine-tuning or retrieval-augmented generation data is grounded" (MS-2.5-005), and "Track dataset modifications for provenance by monitoring data deletions, rectification requests, and other changes that may impact the verifiability of content origins" (MG-4.1-006).3 Operationally (the course's practice, not NIST's wording): every document in a grounding corpus has a named owner, a version and a review-by date.
Anchor case: Moffatt v. Air Canada, 2024 BCCRT 149
The case is cited by the customer's name; the body text says "the customer", since the analysis does not need the name. In November 2022, after a family death, the customer asked the chatbot on Air Canada's website about bereavement fares.4 The chatbot said a reduced fare could be requested after travel, within 90 days of the ticket's issue date; the same answer linked to Air Canada's "Bereavement travel" page, which said the policy did not apply to completed travel.4 The customer brought a small claim at the British Columbia Civil Resolution Tribunal.4 Air Canada submitted that it could not be liable for information from its chatbot, as though the chatbot were a separate entity responsible for its own actions; in a decision issued 14 February 2024 (Tribunal Member Christopher C. Rivers) the tribunal called this a "remarkable submission".4 It held the chatbot to be part of Air Canada's website and Air Canada responsible for all information on the website, whether from a static page or a chatbot.4 It did not expect the customer to double-check one part of the site against another, found reliance on the chatbot reasonable, found negligent misrepresentation, and awarded C$650.88 in damages plus pre-judgment interest and CRT fees.4
Reading for grounding. The deployer owns the whole published surface. When two published statements conflict, reconciling them is the publisher's job; the user is not the consistency check.4 Limits. The decision does not describe how the chatbot produced its answer (no account of retrieval or generation), so the case cannot be attributed to a corpus defect or to unfaithful generation; the locus is unknown.4 It is a single small-claims decision of a provincial tribunal.4
6. Triage and escalation
| Observation after section 4 checks | Likely locus | Owner of the fix | Action |
|---|---|---|---|
| Attributions verify; sources conflict | Corpus (inconsistent) | Document owners; topic owner reconciles | Report both documents; hold the answer |
| Attribution verifies against a superseded version | Corpus (stale) | Document owner; corpus maintainer | Report; remove or retire the old version |
| No source covers the query; model answered anyway | Corpus (incomplete) and abstention failure | Topic owner; tool owner | Report the gap; escalate the non-abstention |
| Sources current and consistent; output unsupported | Undetermined (retrieval or generation) | Tool owner | Escalate with query, output, attributions and source of record |
The last row is the escalation boundary. A user cannot repair it by editing documents, and re-sampling until an acceptable answer appears hides the defect from everyone else. NIST AI 600-1 suggests procedures to "record and track GAI system reported errors, near-misses, and negative impacts" (MG-4.3-002) and verifying that those monitoring reported issues can "promptly escalate issues for response" (MG-4.1-007).3 Localising the fault (retriever recall, ranking, context position, or generator faithfulness) is Level 300.
Every "owner" in the table is a named person with a known channel, not a team alias. Where the owner is not known, record "owner unknown" and resolve that before closing the item.
Big Widget triage (fictional). Ana owns the returns policy; Tom owns the dealer assistant. (a) The assistant answers "60 days" for unused parts and quotes a "Dealer returns" page that says 60; the returns policy states its scope as "all returns, including returns from dealers" and says 30. The context check (section 4, check 2) finds no exception, effective date or precedence statement on either page that resolves the conflict. Both attributions verify, both documents are current and both claim the same scope: inconsistent corpus. Which rule governs is a policy decision for Ana, not for the user; hold the answer and report both documents to her. (b) The assistant says custom-built widgets are returnable within 30 days, quoting the 30-day sentence; the current policy excludes custom-built widgets. The quote exists but does not support the claim and the corpus is sound: escalate to Tom.
7. The three yeses as gate points
The Foundation's three movements are Get to Yes ("before a single action runs"), Stay at Yes ("while it runs, in real time") and Recover to Yes ("after — on demand, on incident"), and a gate decides "go ahead · only this far · wait for a person · no".5 Mapping them onto grounded answers is this course's application, not a quotation.
| Movement | Grounded-answer controls | Gate outcomes |
|---|---|---|
| Get to Yes (before reliance) | Corpus inventory with a named owner, version and review-by date per document; prompt contract requiring claim-level quote, location and abstention; named tool owner and escalation channel. Unknowns are recorded as "not checked" or "unknown" with who will resolve them | Do not rely on the tool for an answer class until these exist |
| Stay at Yes (per answer) | Section 4 checks on every answer that matters; corpus changes re-open Get to Yes; escalations are recorded, not resampled away | All claims verify: go ahead. Strike unsupported claims and use the verified remainder: only this far. Corpus defect, non-abstention, or unsupported output over a sound corpus: wait for a person (the named document or tool owner). Unverifiable answer: no |
| Recover to Yes (after a wrong answer is relied on) | Stop relying on the answer; notify the owner; correct it for those affected once the owner has ruled; fix the corpus or escalate the tool | Claim recovery only after re-running the query against the repaired corpus or tool and re-verifying. Withdrawn is stopped, not recovered |
Evidence versus receipts. The escalation packet (query, output, attributions, source of record) contains outputs and possibly customer data. It is a controlled work record, kept under the organisation's data rules. The Foundation's signed receipt is a separate object that "holds no private content";5 it carries "hashes, decision, tier, policy hash, key id" and never "prompts, outputs, payloads, personal data".6 A receipt shows that a decision was made under a policy; it does not show that the grounded answer was correct.
8. Summary
| Objective | Practice |
|---|---|
| 200.3-a | Grounding changes the evidence, not the guarantee: context can be under-used2 and outputs can diverge from input3 |
| 200.3-b | Require verbatim quote plus location per claim. Review and verify sources and citations in outputs;3 this course's method does so with four checks (existence, context, support, coverage) against the source of record |
| 200.3-c | Classify corpus defects and route them to named document owners; the deployer owns its whole published surface.4 Escalate non-abstention and unsupported outputs over sound sources to the tool owner, under procedures to record reported errors and promptly escalate issues3 |
Further learning (optional, technical; not part of this lesson): Carnegie Mellon University, 11-667 Large Language Models: Methods and Applications (Fall 2025), schedule includes a retrieval-augmented generation lecture and readings: <https://2025.cmu-llms.org/schedule/>. Stanford University, CS224N: Natural Language Processing with Deep Learning, lecture "Agents, Tool Use, and RAG" (free public videos are the 2024 series): <https://web.stanford.edu/class/cs224n/>. For the non-technical companion: University of Washington, Modern-Day Oracles or Bullshit Machines? (Bergstrom and West): <https://thebullshitmachines.com/>.