
What a model is, and isn't · Technical gear
You will be able to: describe a generative model as a learned approximation of its training data, and an autoregressive language model (the kind studied here, such as GPT-3) as a predictor over tokens rather than a retrieval system; explain why the fluency of its output is not evidence of truth; and assign work tasks to "assist with human verification" or "not to be delegated", with reasons grounded in the risk literature.
Model names, sizes, benchmark scores and cases below are dated to their sources. This module is as of November 2026; the figures are historical, not measurements of current systems. Reviewed and published, version 1.0.2. It is not a credential.
1. An autoregressive predictor, not a lookup table
Figure data as text
Predicting or looking up?
| Part | What it does | Lesson sources |
|---|---|---|
| Text model | Uses learned patterns and the text you give it to predict a likely continuation. | s1, s2 |
| Separate lookup tool | Searches for relevant documents and can pass their text to the model. | s5 |
NIST's Generative AI Profile states the general property: generative models "generate outputs that approximate the statistical distribution of their training data; for example, LLMs predict the next token or word in a sentence or phrase".1 The first clause is the claim about generative models in general; the second is NIST's example. This module studies that example: autoregressive language models. It makes no claim that every generative model, or every text model, generates left to right; other generation procedures are outside its scope.
An autoregressive model generates a sequence one element at a time, conditioning each step on what it has already generated. This is the generation procedure of the Transformer decoder: given the encoder's representations, the decoder "generates an output sequence ... of symbols one element at a time", and "[a]t each step the model is auto-regressive, consuming the previously generated symbols as additional input when generating the next".2 Its self-attention is masked so that no position can attend to later positions, which preserves this auto-regressive property.2 GPT-3 (2020) is described by its authors as "an autoregressive language model with 175 billion parameters".3
Three consequences follow for a non-technical audience.
- No answer store. What the model retains from training is encoded in its weights. Lewis et al. call this parametric memory.4 There is no table of question-answer pairs to consult. An answer is generated from a learned conditional distribution, so a fact can be reproduced, distorted or invented by the same mechanism; NIST calls the invented case a natural result of how these models are designed.1
- Context is not weights. A response is conditioned on the current input as well as the weights. GPT-3 was applied "without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model".3 Information placed in the context window can therefore shape an answer without being added to the parameters. Learned memory and working context are different places for information to come from.
- Retrieval is an addition, not the default. Retrieval-augmented generation (RAG) pairs a pre-trained generator with a non-parametric memory: in Lewis et al. (2020), a dense vector index of Wikipedia searched by a pre-trained neural retriever, whose results condition generation. They report that RAG models generated more specific, diverse and factual language than a parametric-only baseline.4 A product that searches documents first is therefore a system built around a model. Module 100.6 treats such systems. The model alone predicts from its weights and its context; it does not look up.
Misconception corrected: "The model looks the answer up somewhere."
2. Likelihood is not truth
A generative model's outputs approximate the statistical distribution of its training data.1 Probable and true are different properties, and fluency is a property of the first.
Fluency is high. In the GPT-3 news-article study, mean human accuracy at identifying articles generated by the 175B-parameter model was "barely above chance at ∼52%" (50% being chance).3 Human readers could not reliably tell the model's prose from human prose, so surface quality carried little signal about provenance. The study measured provenance detection, not factual accuracy; it says nothing either way about how fluency relates to truth. That fluent output can be false is a separate point, supported below.
Confident falsehood follows from the design. NIST AI 600-1 defines confabulation as "the production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived".1 It extends the term to outputs that diverge from the prompt or contradict earlier statements in the same context. It states that confabulations "are a natural result of the way generative models are designed".1 That is a statement about cause, not frequency: a model working as designed can confabulate, and no rate across models or tasks is claimed here. It locates the harm in human response: risks arise "when users believe false content – often due to the confident nature of the response". It also notes that outputs "may also include confabulated logic or citations that purport to justify or explain the system's answer".1
Scale did not fix it, in at least one measured case. TruthfulQA (Lin, Hilton and Evans, 2021) posed 817 questions spanning 38 categories, including health, law, finance and politics. The best model tested was truthful on 58% of questions, and "the largest models were generally the least truthful".5 The paper's human baseline was 94%, but it was a single "participant" who answered "250 randomly sampled questions" with "access to the internet": a reference point, not a population estimate on the same 817-question test.5 The paper's title names the mechanism it studied: models that "mimic human falsehoods". These are 2021 results on 2021 models. They show that likelihood-trained fluency and truthfulness can come apart; they are not a measurement of any model in use as of November 2026.
Long outputs drift. The GPT-3 authors list, among the model's limitations, that samples "start to lose coherence over sufficiently long passages, contradict themselves, and occasionally contain non-sequitur sentences or paragraphs".3
A vendor-reported instance. Google's 21 March 2023 post opening Bard to users stated that such systems "can provide inaccurate, misleading or false information while presenting it confidently", and annotated one of Bard's own responses: "Bard got some things wrong, like the scientific name for the ZZ plant — actually Zamioculcas zamiifolia, not Zamioculcas zamioculcas."6 The wrong name is a near-neighbour of the correct binomial, and no reading of the text alone will catch it; a check against a botanical reference will. Scope: this is one corporate self-reported error. It is not an error rate, and it says nothing about the mechanism that produced this particular output.
The operational rule: a claim's truth must be established by something independent of the generating model, such as a primary document, a measurement or a qualified person. Re-querying the same model is not independent. A Foundation editorial summary by Bob Rapp states the principle directly: "a second question to the same model is not a check".7 Where no independent check has been made, the honest status is not checked, not an inferred "probably fine"; the Foundation's Practice site treats "I don't know yet" as always an answer.8
Misconception corrected: "If it sounds sure, it is right."
3. Delegation by consequence and checkability
NIST groups the relevant failure modes under two risks. Confabulation is one. Human-AI configuration is the other: arrangements that can lead people to "automation bias, over-reliance" on generative AI systems.1 It flags confabulation as "especially important to monitor when integrating GAI into applications involving consequential decision making". Its illustration is a confabulated summary of patient information that "could cause doctors to make incorrect diagnoses".1 Its suggested actions include assessing output accuracy "by comparing it to a set of known ground truth data and by using a variety of evaluation methods (e.g., human oversight and automated evaluation…)" (MP-2.3-001), and deploying "fact-checking techniques to verify the accuracy and veracity of information generated by GAI systems" (MP-2.3-003).1 It notes that organisations' use of GAI "may also warrant additional human review, tracking and documentation, and greater management oversight".1
Capability is also uneven. The GPT-3 paper reports strong results on many tasks and identifies others where few-shot performance "still struggles".3 Module 100.4 develops this as the jagged frontier. The general point for 100.1: breadth of output, the ability to write about anything, is not evidence of reliability on any particular task.
The course's sorting rule uses two variables: the cost of an undetected error and the checkability of the output by a qualified person before anyone relies on it.
| Output easy to check before use | Output hard to check before use | |
|---|---|---|
| Low cost of error | Assist, light review before use (rewording, brainstorming names, first-draft emails) | Assist; a named owner reviews the output before anyone relies on it, and the owner decides how deep that review goes |
| High cost of error | Assist only with full human verification against primary records before use (a summary of a contract the reviewer also reads; a fault-code explanation checked against the manual) | Do not delegate: safety settings, entries in the books, customer prices and commitments, legal positions, decisions about people |
In every cell, the check happens before anyone relies on the output. The table covers model output that a person reads and then acts on. It does not cover a model whose output is executed directly, with no person in between; that is a separate question about gating actions, taken up in modules 100.6 and 100.9. The Foundation's rule for that case is explicit: "No direct model-to-tool path for consequential actions."9 Any consequential action stays gated by a named person.
The bottom-right cell is not "AI may not be involved". It is "the model is not the decision-maker". A person makes the decision, owns it and can explain it. In the Foundation's terms, a gate returns one of four decisions, "go ahead · only this far · wait for a person · no", and leaves a receipt.9 The matrix is the course's application of that idea to human-read output: the three "assist" cells resolve to go ahead or only this far, with a named reviewer before reliance; the "do not delegate" cell resolves to wait for a person or no for the model acting alone. "Human in the loop" without a named person who can release the hold is not a gate.
A judicial record. In Mata v. Avianca (2023), the court found that the respondents "abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question", and that "[b]efore executing the Affirmation, Mr. LoDuca did not review any judicial authorities cited in his affirmation."10 Legal citations in a filing are high cost and highly checkable against the system of record, yet the court found that a signatory had not reviewed the cited authorities. Scope: the opinion establishes these findings about these respondents. It does not establish a citation-fabrication rate or the generation mechanism. A Foundation editorial summary by Bob Rapp on the case states the corresponding gate, under the heading "The gate that would have caught it": "every consequential AI output is independently verified against the system of record before it is used or filed."7 That is the Foundation's proposed rule, not the court's holding.
Receipts carry metadata, not content. If a check is recorded, the shareable evidence must not leak what was checked. The Foundation's receipt carries "hashes, decision, tier, policy hash, key id" and never carries "prompts, outputs, payloads, personal data"; per the Manifesto it "holds no private content".9 8 Prompts, model outputs and source figures, where the workplace permits keeping them, belong in a separately controlled work record. A receipt shows that a decision was made under a policy. It does not show that the model's output was correct.
At Big Widget Company (fictional). Both examples assume a tool the company has approved for the task. Maya, a paint-shop supervisor, uses an AI tool to interpret a robot fault code: high cost of error, but checkable against the manual and the maintenance team, so the tool may assist under full verification. Changing a safety setting on the tool's say-so is bottom-right: not delegable. Luis in finance may have the tool draft a variance commentary, provided the tool is approved to receive that ledger data and every figure is reconciled to the ledger by him before release. Reconciling the figures does not cure sending confidential data to a tool that was not approved for it.
The three yeses, scoped to this module. The Foundation's Manifesto maps four gates to three movements: Get to Yes (Pre-pend, "before a single action runs"; Policy, "before a rule or a system ships"), Stay at Yes (Operate, "while it runs, in real time") and Recover to Yes (Audit, "after — on demand, on incident").9 For 100.1 that gives:
- Get to Yes: is the task one AI is good at; is the tool approved and the data safe to share (module 100.7); is the action allowed; and which named person checks and owns the result? Fact-checking alone does not make a use permitted, and the model's confidence satisfies none of these conditions.
- Stay at Yes: verification is per output, not a one-time qualification of the tool. A correct answer yesterday is not evidence about today's.
- Recover to Yes: when unverified output has been relied on, stop relying on it, name the owner, correct the record and change the rule. A stated repair is not a demonstrated recovery; claim it only with evidence.
Module 100.9 treats all three in full.
Misconception corrected: "A tool that can write anything can be trusted with anything."
Further learning (not sources)
Free public course materials. They are links only, not citations; nothing here is copied from them.
- Stanford University, CS324: Large Language Models: <https://stanford-cs324.github.io/winter2022/>
- University of Oxford, Deep Learning for Natural Language Processing (2017 lectures, advanced): <https://github.com/oxford-cs-deepnlp-2017/lectures>
- Harvard University, CS50's Introduction to Artificial Intelligence with Python: <https://cs50.harvard.edu/ai/>