
How it writes · Plain gear
You will be able to: say how an AI like GPT-3 writes, one small piece at a time; say why an AI can sound sure and still be wrong; and name one way to check.
Laying track one piece at a time
Figure data as text
One piece at a time
A loop: step 3 leads back to step 1.
- Read the context: the request and the text written so far. [s1]
- Work out how likely each possible piece is to come next. [s1]
- Choose and add one token at the end. [s1]
- Repeat from step 1 with the new token in the context. [s1]
Objective: 100.3-a. Text equivalent of figures/next-token-loop.svg.
Picture a toy train. There is no finished track. As the train rolls, one new piece of track is laid in front of it. Then the next. Then the next. Each piece is chosen to fit the pieces already down.
Some AIs write a bit like that. This lesson is about that kind of AI. GPT-3 is one example.1
It cuts writing into small pieces called tokens. In the design we look at here, a token can be a whole word or part of a word.2
Then it writes one token at a time. Each time, it looks at the text it has in front of it: your question, plus what it has already written. It works out how likely each possible token is to come next. It picks one and adds it. Then it does the same thing again.2 Experts have a long name for this: autoregressive.1
There is a limit to how much text it can look at once. GPT-3 could look at about 2,000 tokens at a time.1 Text outside that limit is not part of what it looks at.
So the words are made one small piece at a time. Each piece it has written becomes part of what it looks at for the next piece.2
Here is the key point. It picks the likely next piece. Likely is not the same as true. A US government standards group, NIST, says the same thing. These AIs predict the next word, and that can give you true words or false ones.3 So picking a likely piece is not checking the facts.
What we do not cover here. This lesson is about how the words are made. It is not about how an app shows them to you, all at once or bit by bit. We have not checked any app. And it does not say what goes on inside the AI while it picks. That is a harder question.
Sounding sure is not being right
AI writing can sound very real. In one test, people read short news stories. Some were written by people and some by an AI. People had to guess which was which. They were right only a little more often than if they had tossed a coin.1
That test was about who wrote it, not whether it was true. But it shows how good the writing can be. So smooth writing does not prove the facts are right.
Sometimes an AI makes things up. It might invent a book that does not exist, a quote nobody said or a number from nowhere. And it says it in the same calm, sure voice. People call this a hallucination. NIST calls it confabulation: false content stated with confidence, which can fool people.3
It is not just a glitch in a broken AI. NIST says it is a natural result of how these AIs are built.3 So an AI that is working normally can still make things up. How often depends on the AI and the task. Here are two older tests:
- In a 2021 test, AIs answered hundreds of tricky questions about health, law, money and politics. The best AI was truthful on 58 in 100 questions. Many wrong answers repeated common myths. And in general, the biggest AIs were the least truthful.4 (For comparison, one person answered 250 of the questions, with the internet to help, and was truthful on 94 in 100.4)
- In 2022, researchers trained an AI with feedback from people. On some kinds of tasks, like summing up a text, it made things up in about 21 out of 100 answers, down from about 41 before. The researchers said it could still make up facts.5
Those numbers are about those AIs, on those tasks, back then. They are not about the AI you use today.
Here is the sneaky part. The made-up bit can look like proof. An AI can add sources or reasons that make its answer look trustworthy, even when they are invented.3 The trouble starts when someone believes it, often because it sounds so sure, and acts on it.3
A real case: court cases that did not exist
This really happened. In 2023, a US court looked at a case called Mata v. Avianca. The court found that lawyers sent in court opinions that did not exist, "with fake quotes and citations created by the artificial intelligence tool ChatGPT."6 When the judge asked about them, they kept standing by them.6 And one lawyer signed his papers without reading any of the cases he cited.6
The fake cases looked real. They had names and quotes. They were said to be in well-known law books and a legal database.6
People did try to look them up. Lawyers for the other side said they could not find most of them.6 The court searched too, and could not find several of them.6 And the lawyer who wrote the papers told the judge he had looked up one of the cases and could not find it. He had cited it anyway.6
So "look it up" was not enough on its own. The rule this course takes from the story is: if you cannot find it, stop. Do not use it until you find the real thing.
This story does not tell us how the AI made the fake cases, or how often AIs do this. It shows what happens when a check fails and the work goes ahead anyway.
The AiGovOps Foundation wrote about this case in a blog post by Bob Rapp (November 2026). The post is an editorial summary. It gives the Foundation's rule: "every consequential AI output is independently verified against the system of record before it is used or filed."7 That means: anything that matters gets checked against the real record before you use it. That is the Foundation's idea, not what the court said.
One way to check: open the source
NIST advises organisations that run AI systems to review and verify the sources and citations in what the AI writes, both before the system goes live and while it is in use.3 This course turns that into three steps you can take yourself:
- Open every source. Find the real page, report or book. If you cannot find it, it does not count.
- Find the exact words. Make sure the source really says what the AI says it does. If no real source backs a claim, take the claim out.
- Ask a real person when it matters, like a report, a customer or a big decision. Asking the same AI "are you sure?" does not count. The Foundation's post puts it this way: "a second question to the same model is not a check."7
Stuck? It is fine to say "I don't know yet." Write "not checked" and ask someone. The Foundation's practice site says "I don't know yet" is always an answer, and it always pairs you with someone.8
Some AI tools look up documents first and then write. In one 2020 study, an AI that did this wrote quiz questions that people judged more factual than ones from an AI that did not.9 That is one test on one kind of task. More factual is not perfect, so you still check each claim.
At Big Widget Company
Big Widget Company is made up, and so are its people.
Luis buys parts. He asks an AI why parts cost more last quarter, and asks for sources.
- Good AI: Luis opens every source. Two of them do not exist! He looks for real sources for those parts of the answer. Where a real source says something, he rewrites the sentence to match it. Where he cannot find one, he cuts the sentence. He asks Priya in finance to check the numbers too.
- Bad AI: Luis copies the smooth answer into a report for the bosses. He never opens a source.
The answer sounded just as sure both times. It was written one likely piece at a time. The only thing that caught the fake sources was Luis.
The three yeses. The Foundation uses three short phrases: Get to Yes, Stay at Yes and Recover to Yes.10 Here is the source-checking part of each.
- Get to Yes: before the report goes out, Luis checks the sources and his boss says go ahead. One of a Foundation gate's answers is "wait for a person."10 In this course, a report with a source still unopened gets that answer. This story also assumes Luis is allowed to use this AI tool and to put this purchasing data into it. Lessons 100.7 and 100.9 cover those questions.
- Stay at Yes: next time, he checks again. One good check does not cover every answer after it.
- Recover to Yes: if a made-up fact gets out, stop, own it, fix it for the people who relied on it, and change the rule. Then check that the fix worked.
Keep notes, not secrets. Luis writes down which sources he opened and what he found. He keeps that in his team's own files. The Foundation's receipt is different. It never carries "prompts, outputs, payloads, personal data".8 So the AI's answer and the purchasing data do not go in it.
Words to know: token, hallucination, confabulation, source.