
How it writes · Technical gear
You will be able to: describe autoregressive generation over a token vocabulary; explain why next-token prediction yields fluent output with no built-in guarantee of truth (confabulation); and name a concrete verification step.
Tokens and the autoregressive loop
Figure data as text
One piece at a time
A loop: step 3 leads back to step 1.
- Read the context: the request and the text written so far. [s1]
- Work out how likely each possible piece is to come next. [s1]
- Choose and add one token at the end. [s1]
- Repeat from step 1 with the new token in the context. [s1]
Objective: 100.3-a. Text equivalent of figures/next-token-loop.svg.
Scope. This section describes autoregressive language models, the class GPT-3 belongs to.1 It does not claim that every text-generating model is autoregressive or uses the same tokenisation.
Tokens. In the original Transformer, text was modelled as a sequence of subword units drawn from a fixed vocabulary: byte-pair encoding with a shared vocabulary of about 37,000 tokens for English–German, and a word-piece vocabulary of 32,000 tokens for English–French.2 In such a vocabulary a token can be a whole word or a fragment of a word.
Next-token distribution. At each position $t$, the decoder maps the context to a hidden state $h_t$, and a learned linear transformation followed by a softmax converts it into a probability distribution over the vocabulary $V$:2
p_\theta(x_t \mid x_{<t}) = \operatorname{softmax}(W h_t), \qquad \sum_{v \in V} p_\theta(v \mid x_{<t}) = 1 .
Autoregression. Generation is auto-regressive: the model consumes the previously generated symbols as additional input when generating the next one.2 GPT-3 is described by its authors as an autoregressive language model of this type.1 Given the per-step conditionals, the probability of a whole sequence follows from the chain rule of probability (a mathematical identity, not an empirical finding):
p_\theta(x_1, \dots, x_T) = \prod_{t=1}^{T} p_\theta(x_t \mid x_{<t}).
Context window. The conditioning context is finite. GPT-3 used a context window of $n_{ctx} = 2048$ tokens.1 Earlier text that does not fit is not part of $x_{<t}$ at that step.
Decoding. Producing text means repeatedly selecting a token from $p_\theta(\cdot \mid x_{<t})$, appending it to the context and recomputing. The selection rule is a separate design choice from the model itself; Vaswani et al., for example, decoded their translation models with beam search (beam size 4).2 The sources cited here report that choice but do not specify how beam search, or any other decoding rule, ranks candidates; this lesson does not describe that mechanism (not checked).
What this does and does not imply about "planning." Within any one candidate sequence, tokens are generated left to right, and each generated token is fixed as conditioning context for later steps.2 The per-step quantity is a probability given the context;2 statistical prediction of this kind can produce accurate or inaccurate output.3 So the generation pass is not itself a fact check: the procedure described here contains no step that checks a finished answer against the world, the way a person might check and edit a document. Three separate layers should not be confused:
- Generation: the autoregressive pass just described.
- Display: how a product presents the output to a user. This lesson makes no claim about any particular product's display behaviour (not checked); whatever it is, it does not change how the text was generated.
- Surrounding workflow: a system may wrap the model in other steps, such as retrieving documents before generation;4 any fact check has to come from such a step or from a person, not from token generation itself.
The papers cited here describe the generation procedure; they do not settle what a network represents internally about tokens it has not yet emitted, and this lesson makes no claim either way.
Link to truth. Every step chooses tokens by their probability given the context. Nothing in $p_\theta$ marks a token as true. NIST draws the same connection: generative models "generate outputs that approximate the statistical distribution of their training data; for example, LLMs predict the next token or word," which "can produce factually accurate and consistent outputs" but "can also produce outputs that are factually inaccurate or internally inconsistent."3
Confabulation: fluent, confident and false
Definition. NIST AI 600-1 (July 2024) defines confabulation as "the production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived," and calls confabulations "a natural result of the way generative models are designed."3 It also notes that outputs may include confabulated logic or citations that purport to justify the answer, further misleading people into trusting it.3
Fluency does not establish accuracy. In Brown et al.'s human-detection study, evaluators' mean accuracy at identifying ~200-word news articles generated by the 175B-parameter GPT-3 as machine-written was about 52%, where 50% is chance.1 That study measured provenance detection (human or machine author), not factual accuracy. It shows that surface quality gave evaluators little signal about authorship; it does not measure how fluency and truth correlate. The separate support for fluent falsehood is NIST's definition above: confabulated content is confidently stated.3 The operational conclusion is narrower than "decoupled": fluency is not evidence sufficient to establish that content is true.
Measured rates (dated, task-specific). We found no general frequency for confabulation in NIST AI 600-1, and none of the sources here offers one. Before repeating a rate, ask out of how many, on which model, measured how and when. The following are measurements of particular models on particular tasks, not rates for current products.
- TruthfulQA (2021). 817 questions across 38 categories (including health, law, finance and politics), built so that some humans would answer falsely through misconception. The best model tested was truthful on 58% of questions; the largest models were generally the least truthful, and many false answers mimicked popular misconceptions.5 The human baseline of 94% came from one participant who answered 250 randomly sampled questions with internet access, not from people taking all 817 items.5 The authors suggest that scaling up alone is less promising for truthfulness than fine-tuning with training objectives other than imitating web text.5
- InstructGPT (2022). Fine-tuning GPT-3 with human feedback reduced the hallucination rate on closed-domain tasks from OpenAI's API prompt distribution (outputs that should contain only information present in the input, such as summarisation) from 41% to 21%.6 The authors still list "make up facts" among InstructGPT's remaining simple mistakes.6
These results show that, on the tasks measured, confabulation was measurable and was reduced but not removed by tuning. Together with NIST's statement that confabulation is a natural result of generative model design,3 they correct the misconception that made-up facts only come from a malfunction.
Retrieval helps but does not certify. Retrieval-augmented generation conditions the model on documents fetched from an index. Lewis et al. report that RAG models generated more specific, diverse and factual language than a parametric-only seq2seq baseline.4 The human factuality judgement behind that claim was on one task, Jeopardy question generation: over 452 pairs, raters judged RAG's question more factual in 42.7% of cases and BART's in 7.1%.4 The same abstract names providing provenance for decisions as an open research problem for large pre-trained language models in general.4 Retrieval can help; it does not verify any particular claim, and the retrieved documents themselves still need checking.
Case: citations that could not be resolved
In Mata v. Avianca (No. 22-cv-1461 (PKC), sanctions opinion of 2023), the court found that the respondents "abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question."7 From finding 6: "Before executing the Affirmation, Mr. LoDuca did not review any judicial authorities cited in his affirmation."7
Lookups were attempted, and they failed. The filing cited decisions "said to be published in the Federal Reporter, the Federal Supplement and Westlaw" (finding 5). Avianca's reply stated that its counsel had been "unable to locate most of the case law cited" (finding 7). The Court "conducted its own search for the cited cases but was unable to locate multiple authorities" (finding 9). Mr. Schwartz, who researched and wrote the filing (finding 6), testified that he looked up "Varghese" and "couldn't find it", and confirmed that he had cited it (finding 12).7
What the case supports: confabulated citations can be specific and well-formed enough to reach a court filing, and a failed lookup did not, by itself, stop their use. The control this course draws from it (the course's application, not the court's words) is to resolve each authority against an authoritative report, database or record, and to hold the citation when resolution fails. What the case does not support: any claim about how the tool generated those citations, about confabulation frequency, or about other incidents. The RECAP host is a public mirror of the filed record, not the court's own site.
A Foundation blog post by Bob Rapp (November 2026), which describes itself as an editorial summary, gives the Foundation's gate "every consequential AI output is independently verified against the system of record before it is used or filed."8 That is the Foundation's rule, not the court's holding.
A verification step you can name
NIST's suggested action MS-2.5-003, addressed to organisations managing GAI risk, is to "review and verify sources and citations in GAI system outputs during pre-deployment risk measurement and ongoing monitoring activities."3 The procedure below is this course's application of that control to an individual user's workflow; it is not NIST's text.
- Resolve every citation. Retrieve the cited document from its publisher, registry or court record. Unresolvable means unsupported.
- Match the claim to the text. Locate the quoted words, figure or holding in the source; check that the source supports the sentence as written, not a nearby claim. A claim with no supporting source is removed, not re-cited.
- Escalate by consequence. Anything going to a customer, a regulator, a board or a safety decision gets a named second human checker. Re-asking the same model is not independent verification: "a second question to the same model is not a check."8
An unresolved citation is recorded as "not checked" and routed to a person, not silently dropped or assumed fine. The Foundation's Practice site: "I don't know yet" is always an answer, and it always pairs you with someone.9
At Big Widget Company
Big Widget Company is a made-up company; its people are made up too.
Luis in purchasing asks an AI tool to explain last quarter's parts-cost increase, with sources. The draft is fluent and fully referenced; to the model, references are generated tokens like any other.
- Good AI: Luis resolves every reference. Two do not exist. For each claim they were attached to, he searches for a real source, rewrites the claim to what that source supports, or deletes the claim if nothing supports it. Priya in finance then checks the figures as an additional step.
- Bad AI: the draft goes into the board report unresolved.
The three yeses (source-verification subset). The Foundation's lifecycle is Get to Yes, Stay at Yes and Recover to Yes.10
- Get to Yes: citations are resolved before use, and a named person (here, Luis's manager) makes the gate decision. The Foundation's gate answers "go ahead · only this far · wait for a person · no";10 a draft with any unresolved citation is a "wait for a person". The scene assumes an approved tool and that this purchasing data may be shared with it; those questions, plus a written action rule, are covered in 100.7 and 100.9.
- Stay at Yes: every later draft gets the same resolution step. One clean check is not a standing approval.
- Recover to Yes: if a confabulated fact ships, stop, own it, fix it for the people affected and change the rule. A planned repair is not a demonstrated recovery; check the fix took effect.
Evidence without private content. Luis's check notes (sources opened, what each said) live in his team's controlled work record. A Foundation signed receipt "holds no private content";10 it never carries "prompts, outputs, payloads, personal data".9 The model output and purchasing data therefore stay out of the receipt. A receipt records that a decision was made; it does not prove the summary was true.
Words to know: token, vocabulary, softmax, autoregressive, context window, decoding, confabulation, retrieval-augmented generation, provenance, gate, receipt.
Further learning (optional, not citations)
Free public university course materials. They assume programming or university mathematics.
- Stanford University, CS324: Large Language Models (Winter 2022): https://stanford-cs324.github.io/winter2022/
- University of Oxford, Deep Learning for Natural Language Processing (Hilary Term 2017), lecture slides and videos: https://github.com/oxford-cs-deepnlp-2017/lectures