Level 100: How AI works: AI at work and in life · Module 100.3

Two coworkers talk while one arranges paper slips in a row on the table.
One piece at a time.

How it writes · Standard gear

You will be able to: explain that an autoregressive language model, such as GPT-3, writes one small piece of text at a time by choosing a likely next piece; explain how that lets a model state something false in a fluent, confident voice; and name one way to check what it wrote.

One small piece at a time

Read the request and text so far, estimate the next token, choose and append one, and repeat with the new context.
The model reads what is in context, estimates how likely each next token is, adds a chosen token and repeats with that piece in its context (s1).
Figure data as text

One piece at a time

A loop: step 3 leads back to step 1.

  1. Read the context: the request and the text written so far. [s1]
  2. Work out how likely each possible piece is to come next. [s1]
  3. Choose and add one token at the end. [s1]
  4. Repeat from step 1 with the new token in the context. [s1]

Objective: 100.3-a. Text equivalent of figures/next-token-loop.svg.

This lesson is about autoregressive language models. They write text one piece after another. GPT-3, a 2020 language model, is described by its makers as this kind of model.1 This is not a claim about every AI system that produces text.

These models split text into small pieces called tokens. In the design examined here, tokens come from a fixed list of about 32,000 to 37,000 pieces, and a token can be a whole word or a piece of a word.2

The model writes by repeating one step:

  1. It reads the text in its context: your request, plus whatever it has written so far.
  2. It works out how likely each token on its list is to come next.2
  3. One token is chosen and added to the end.
  4. The step runs again, with the new token now part of what it reads.2

The context has a limit. GPT-3 used a context window of 2,048 tokens.1 Text that does not fit is not part of what the model reads at that step.

So the text is generated a token at a time. Each token, once chosen, is part of what the next one is based on.2 That is not the same as drafting a whole answer, rereading it and fixing it. A system can add other steps around the model, such as looking up documents first (see below).3 But generating tokens does not, by itself, check the facts.

Why it matters. Each token is picked because it is likely to come next. Likely is not the same as true. The US standards agency NIST makes the same link: large language models "predict the next token or word," and that kind of statistical prediction can produce accurate text or text that is factually wrong.4

What this lesson does not cover. It describes how the text is generated. It does not cover how a particular app shows the text to you, for example whether the words appear as they are made or all at once. We have not checked any particular app. It also does not settle what happens inside the model while it picks each token. That is a separate, harder question.

Fluent is not the same as true

AI-written text can be very hard to tell from a person's. In one test, people were shown short news articles and asked which were written by a person and which by GPT-3. They picked correctly about 52% of the time, where guessing would get 50%.1 That study measured whether people could tell who wrote the text, not whether the text was true. What it shows is how polished the writing can be. Smooth, confident writing tells you the model writes well. It is not evidence that the content is right.

When a model states something false as if it were true, people often call it a hallucination. NIST calls it confabulation: "the production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived."4

Made-up facts are not a sign that the model is broken. NIST says confabulations are "a natural result of the way generative models are designed."4 A model working normally can produce them. We found no single rate for every model and task in these sources. Two dated measurements show the problem on specific tasks. Before you repeat either number, note what was measured, on which model, and when.

The made-up part can look like proof. NIST warns that AI output can include made-up logic or citations that seem to justify the answer, which can lead people to trust it when they should not.4 A reference list, a quotation or a precise number is still just more tokens. It may point to nothing real.

The danger comes when people believe the false content, often because the answer sounds so sure, and then act on it or pass it on.4

A real case: citations that could not be found

In Mata v. Avianca, a US federal court found in 2023 that lawyers and their firm "abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question."7 The court also found that before one lawyer signed his affirmation, he "did not review any judicial authorities cited in his affirmation."7

The fake opinions had names, quotes and citations: the look of proof. They were said to be published in the Federal Reporter, the Federal Supplement and Westlaw.7

People did try to look them up, and the lookups failed. Avianca's lawyers wrote that they had been "unable to locate most of the case law cited."7 The court ran its own search and could not locate multiple authorities cited in the filing.7 The lawyer who researched and wrote the filing told the court he had looked up one of the cases, "Varghese," and "couldn't find it." Asked "And yet you cited it," he answered, "I did."7

So the lesson is not just "look it up." In this course's words, not the court's: resolve each cited authority in the published law reports, a legal database or the official record, and if it cannot be found, stop. Do not use it until it is found. This case shows what happens when fluent output is used without that check, or after the check has failed. It does not show how the tool produced those citations, or how often AI tools invent them.

A Foundation blog post by Bob Rapp (November 2026), which describes itself as an editorial summary, gives the Foundation's gate for this: "every consequential AI output is independently verified against the system of record before it is used or filed."8 That is the Foundation's rule, not the court's ruling.

One way to check: open the source yourself

NIST's suggested actions for organisations managing this risk include to "review and verify sources and citations in GAI system outputs during pre-deployment risk measurement and ongoing monitoring activities."4 That is written for organisations that build or run AI systems. This course applies the same idea to anyone reading an AI answer:

  1. Open every source it gives. Find the actual document, page or court record. If you cannot find it, it does not count.
  2. Find the exact words. Check that the source really says what the answer claims, including any quote or number. If no real source supports a claim, remove the claim, not just the citation.
  3. Ask a named second person when the answer matters, for example when it goes into a report, to a customer or into a decision. Asking the same AI "are you sure?" does not count. As the Foundation's post puts it, "a second question to the same model is not a check."8

If you cannot find a source and do not know whether a claim is true, say so: write "not checked" and take it to someone who can help. The Foundation's Practice site puts it this way: "I don't know yet" is always an answer, and it always pairs you with someone.9

Tools that look up documents first and then write from them can help. In one 2020 study, people compared Jeopardy-style quiz questions written by two models: one that retrieved documents first and one that wrote from what it had learned alone. Over 452 pairs, raters judged the retrieval model's question more factual in 42.7% of cases and the other model's in 7.1%.3 That is one task in one study. More factual is not always right, and the same paper describes showing where a large language model's answers come from as an open research problem.3 So you still open the source and check each claim.

At Big Widget Company

Big Widget Company is made up, and so are its people.

Luis works in purchasing. He asks an AI tool to explain why parts costs went up last quarter, and to list its sources.

The answer read just as smoothly both times. It was written one likely token at a time, and generating it did not check the sources. Only Luis's check did.

The three yeses, for this scene. The Foundation names three movements: Get to Yes, Stay at Yes and Recover to Yes.10 This scene covers the source-checking part of each.

Keep the evidence, not the secrets. Luis keeps a note of which sources he opened and what he found, in his team's own work files. The Foundation's signed receipt is a different thing: it "holds no private content."10 It never carries "prompts, outputs, payloads, personal data".9 So the AI's answer and the purchasing data stay in the work files, not in a receipt.

Words to know: token, autoregressive, context window, hallucination, confabulation, source, gate, receipt.

Further learning (optional, not citations)

Free public university course materials, for learners who want more. Some assume programming or university mathematics.

Sources

  1. s2 Tom B. Brown et al. (2020). Language Models are Few-Shot Learners. Paper. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩ ↩
  2. s1 Ashish Vaswani et al. (2017). Attention Is All You Need. Paper. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩ ↩ ↩
  3. s6 Patrick Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Paper. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩ ↩
  4. s3 Chloe Autio et al. (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. Standard. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩
  5. s5 Stephanie Lin, Jacob Hilton, and Owain Evans (2021). TruthfulQA: Measuring How Models Mimic Human Falsehoods. Paper. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩
  6. s4 Long Ouyang et al. (2022). Training language models to follow instructions with human feedback. Paper. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩
  7. s7 United States District Court (court-filed opinion; RECAP copy hosted by CourtListener) (2023). Mata v. Avianca, No. 22-cv-1461 (PKC), Opinion and Order on Sanctions (docket entry 54). Record. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩
  8. s8 Bob Rapp (AiGovOps Foundation) (2026). Asking the bot to check the bot — Mata v. Avianca, and the citations nobody opened. Official doc. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩
  9. s10 AiGovOps Foundation (n.d.). AiGovOps Foundation Practice (home page). Official doc. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩
  10. s9 AiGovOps Foundation (n.d.). AiGovOps Foundation Manifesto. Official doc. Checked 2026-11-17 by source-checker agent (Claude, separate round-3 re-check session; did not write or revise the module), 17 Nov 2026. ↩ ↩ ↩

Next: the quiz