Level 100: How AI works: AI at work and in life · Module 100.3

Two coworkers talk while one arranges paper slips in a row on the table.
One piece at a time.

How it writes · Technical gear

You will be able to: describe autoregressive generation over a token vocabulary; explain why next-token prediction yields fluent output with no built-in guarantee of truth (confabulation); and name a concrete verification step.

Tokens and the autoregressive loop

Read the request and text so far, estimate the next token, choose and append one, and repeat with the new context.
The model reads what is in context, estimates how likely each next token is, adds a chosen token and repeats with that piece in its context (s1).
Figure data as text

One piece at a time

A loop: step 3 leads back to step 1.

  1. Read the context: the request and the text written so far. [s1]
  2. Work out how likely each possible piece is to come next. [s1]
  3. Choose and add one token at the end. [s1]
  4. Repeat from step 1 with the new token in the context. [s1]

Objective: 100.3-a. Text equivalent of figures/next-token-loop.svg.

Scope. This section describes autoregressive language models, the class GPT-3 belongs to.1 It does not claim that every text-generating model is autoregressive or uses the same tokenisation.

Tokens. In the original Transformer, text was modelled as a sequence of subword units drawn from a fixed vocabulary: byte-pair encoding with a shared vocabulary of about 37,000 tokens for English–German, and a word-piece vocabulary of 32,000 tokens for English–French.2 In such a vocabulary a token can be a whole word or a fragment of a word.

Next-token distribution. At each position $t$, the decoder maps the context to a hidden state $h_t$, and a learned linear transformation followed by a softmax converts it into a probability distribution over the vocabulary $V$:2

p_\theta(x_t \mid x_{<t}) = \operatorname{softmax}(W h_t), \qquad \sum_{v \in V} p_\theta(v \mid x_{<t}) = 1 .

Autoregression. Generation is auto-regressive: the model consumes the previously generated symbols as additional input when generating the next one.2 GPT-3 is described by its authors as an autoregressive language model of this type.1 Given the per-step conditionals, the probability of a whole sequence follows from the chain rule of probability (a mathematical identity, not an empirical finding):

p_\theta(x_1, \dots, x_T) = \prod_{t=1}^{T} p_\theta(x_t \mid x_{<t}).

Context window. The conditioning context is finite. GPT-3 used a context window of $n_{ctx} = 2048$ tokens.1 Earlier text that does not fit is not part of $x_{<t}$ at that step.

Decoding. Producing text means repeatedly selecting a token from $p_\theta(\cdot \mid x_{<t})$, appending it to the context and recomputing. The selection rule is a separate design choice from the model itself; Vaswani et al., for example, decoded their translation models with beam search (beam size 4).2 The sources cited here report that choice but do not specify how beam search, or any other decoding rule, ranks candidates; this lesson does not describe that mechanism (not checked).

What this does and does not imply about "planning." Within any one candidate sequence, tokens are generated left to right, and each generated token is fixed as conditioning context for later steps.2 The per-step quantity is a probability given the context;2 statistical prediction of this kind can produce accurate or inaccurate output.3 So the generation pass is not itself a fact check: the procedure described here contains no step that checks a finished answer against the world, the way a person might check and edit a document. Three separate layers should not be confused:

The papers cited here describe the generation procedure; they do not settle what a network represents internally about tokens it has not yet emitted, and this lesson makes no claim either way.

Link to truth. Every step chooses tokens by their probability given the context. Nothing in $p_\theta$ marks a token as true. NIST draws the same connection: generative models "generate outputs that approximate the statistical distribution of their training data; for example, LLMs predict the next token or word," which "can produce factually accurate and consistent outputs" but "can also produce outputs that are factually inaccurate or internally inconsistent."3

Confabulation: fluent, confident and false

Definition. NIST AI 600-1 (July 2024) defines confabulation as "the production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived," and calls confabulations "a natural result of the way generative models are designed."3 It also notes that outputs may include confabulated logic or citations that purport to justify the answer, further misleading people into trusting it.3

Fluency does not establish accuracy. In Brown et al.'s human-detection study, evaluators' mean accuracy at identifying ~200-word news articles generated by the 175B-parameter GPT-3 as machine-written was about 52%, where 50% is chance.1 That study measured provenance detection (human or machine author), not factual accuracy. It shows that surface quality gave evaluators little signal about authorship; it does not measure how fluency and truth correlate. The separate support for fluent falsehood is NIST's definition above: confabulated content is confidently stated.3 The operational conclusion is narrower than "decoupled": fluency is not evidence sufficient to establish that content is true.

Measured rates (dated, task-specific). We found no general frequency for confabulation in NIST AI 600-1, and none of the sources here offers one. Before repeating a rate, ask out of how many, on which model, measured how and when. The following are measurements of particular models on particular tasks, not rates for current products.

These results show that, on the tasks measured, confabulation was measurable and was reduced but not removed by tuning. Together with NIST's statement that confabulation is a natural result of generative model design,3 they correct the misconception that made-up facts only come from a malfunction.

Retrieval helps but does not certify. Retrieval-augmented generation conditions the model on documents fetched from an index. Lewis et al. report that RAG models generated more specific, diverse and factual language than a parametric-only seq2seq baseline.4 The human factuality judgement behind that claim was on one task, Jeopardy question generation: over 452 pairs, raters judged RAG's question more factual in 42.7% of cases and BART's in 7.1%.4 The same abstract names providing provenance for decisions as an open research problem for large pre-trained language models in general.4 Retrieval can help; it does not verify any particular claim, and the retrieved documents themselves still need checking.

Case: citations that could not be resolved

In Mata v. Avianca (No. 22-cv-1461 (PKC), sanctions opinion of 2023), the court found that the respondents "abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question."7 From finding 6: "Before executing the Affirmation, Mr. LoDuca did not review any judicial authorities cited in his affirmation."7

Lookups were attempted, and they failed. The filing cited decisions "said to be published in the Federal Reporter, the Federal Supplement and Westlaw" (finding 5). Avianca's reply stated that its counsel had been "unable to locate most of the case law cited" (finding 7). The Court "conducted its own search for the cited cases but was unable to locate multiple authorities" (finding 9). Mr. Schwartz, who researched and wrote the filing (finding 6), testified that he looked up "Varghese" and "couldn't find it", and confirmed that he had cited it (finding 12).7

What the case supports: confabulated citations can be specific and well-formed enough to reach a court filing, and a failed lookup did not, by itself, stop their use. The control this course draws from it (the course's application, not the court's words) is to resolve each authority against an authoritative report, database or record, and to hold the citation when resolution fails. What the case does not support: any claim about how the tool generated those citations, about confabulation frequency, or about other incidents. The RECAP host is a public mirror of the filed record, not the court's own site.

A Foundation blog post by Bob Rapp (November 2026), which describes itself as an editorial summary, gives the Foundation's gate "every consequential AI output is independently verified against the system of record before it is used or filed."8 That is the Foundation's rule, not the court's holding.

A verification step you can name

NIST's suggested action MS-2.5-003, addressed to organisations managing GAI risk, is to "review and verify sources and citations in GAI system outputs during pre-deployment risk measurement and ongoing monitoring activities."3 The procedure below is this course's application of that control to an individual user's workflow; it is not NIST's text.

  1. Resolve every citation. Retrieve the cited document from its publisher, registry or court record. Unresolvable means unsupported.
  2. Match the claim to the text. Locate the quoted words, figure or holding in the source; check that the source supports the sentence as written, not a nearby claim. A claim with no supporting source is removed, not re-cited.
  3. Escalate by consequence. Anything going to a customer, a regulator, a board or a safety decision gets a named second human checker. Re-asking the same model is not independent verification: "a second question to the same model is not a check."8

An unresolved citation is recorded as "not checked" and routed to a person, not silently dropped or assumed fine. The Foundation's Practice site: "I don't know yet" is always an answer, and it always pairs you with someone.9

At Big Widget Company

Big Widget Company is a made-up company; its people are made up too.

Luis in purchasing asks an AI tool to explain last quarter's parts-cost increase, with sources. The draft is fluent and fully referenced; to the model, references are generated tokens like any other.

The three yeses (source-verification subset). The Foundation's lifecycle is Get to Yes, Stay at Yes and Recover to Yes.10

Evidence without private content. Luis's check notes (sources opened, what each said) live in his team's controlled work record. A Foundation signed receipt "holds no private content";10 it never carries "prompts, outputs, payloads, personal data".9 The model output and purchasing data therefore stay out of the receipt. A receipt records that a decision was made; it does not prove the summary was true.

Words to know: token, vocabulary, softmax, autoregressive, context window, decoding, confabulation, retrieval-augmented generation, provenance, gate, receipt.

Further learning (optional, not citations)

Free public university course materials. They assume programming or university mathematics.

Sources

  1. s2 Tom B. Brown et al. (2020). Language Models are Few-Shot Learners. Paper. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩ ↩ ↩
  2. s1 Ashish Vaswani et al. (2017). Attention Is All You Need. Paper. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩
  3. s3 Chloe Autio et al. (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. Standard. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩
  4. s6 Patrick Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Paper. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩ ↩ ↩
  5. s5 Stephanie Lin, Jacob Hilton, and Owain Evans (2021). TruthfulQA: Measuring How Models Mimic Human Falsehoods. Paper. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩ ↩
  6. s4 Long Ouyang et al. (2022). Training language models to follow instructions with human feedback. Paper. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩
  7. s7 United States District Court (court-filed opinion; RECAP copy hosted by CourtListener) (2023). Mata v. Avianca, No. 22-cv-1461 (PKC), Opinion and Order on Sanctions (docket entry 54). Record. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩ ↩
  8. s8 Bob Rapp (AiGovOps Foundation) (2026). Asking the bot to check the bot — Mata v. Avianca, and the citations nobody opened. Official doc. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩
  9. s10 AiGovOps Foundation (n.d.). AiGovOps Foundation Practice (home page). Official doc. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩
  10. s9 AiGovOps Foundation (n.d.). AiGovOps Foundation Manifesto. Official doc. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩ ↩

Next: the quiz