
What a model is, and isn't · Standard gear
You will be able to: explain what a generative AI model is and why it does not look answers up; explain why a fluent, confident answer is not evidence that the answer is true; and sort everyday work tasks into ones AI can help with when a person checks the result, and ones it should not be trusted with alone.
Reviewed and published, version 1.0.2. It is not a credential. Facts about particular models, studies and cases are dated in the text. This module is as of November 2026.
1. A model learned patterns. It predicts; it does not look things up
Figure data as text
Predicting or looking up?
| Part | What it does | Lesson sources |
|---|---|---|
| Text model | Uses learned patterns and the text you give it to predict a likely continuation. | s1, s2 |
| Separate lookup tool | Searches for relevant documents and can pass their text to the model. | s5 |
A generative AI model is a computer system that makes new content, such as text, pictures, sound or video.1 The US National Institute of Standards and Technology (NIST) describes what all of these have in common: generative models produce outputs that approximate the statistical patterns of the data they were trained on.1 That is the big idea of this module. A model learned patterns.
This module looks closely at one kind of generative model: the language model, which works with text. GPT-3, described by its builders in 2020, is one well-known example. They call it "an autoregressive language model with 175 billion parameters".2 In this course, a large language model (LLM) means a language model built at that kind of scale, with a very large number of parameters.
The important word is autoregressive. A model like this builds its answer one small piece at a time. The Transformer, a model design published in 2017, generates its output "one element at a time", and at each step it uses the pieces it has already produced as input for choosing the next one.3 NIST gives the same example: large language models "predict the next token or word in a sentence or phrase".1 This is how the text models in this module work. It is not a claim about how every image, sound or video model builds its output, and this module does not cover those.
So when you ask a question, a model like this is not finding a stored answer in a table, the way a search box finds a record in a database. It is producing a likely continuation of your words, based on patterns it learned from a very large number of examples. Think of the autocomplete on your phone. It suggests the next word. A language model does something like that for whole answers.
What the model learned in training is held inside the model itself, in its parameters. Researchers call this parametric memory.4 An answer can also use what you put in front of the model right now. GPT-3's builders gave it tasks and worked examples as plain text, "without any gradient updates or fine-tuning": the model's parameters did not change, yet it used what was in the text.2 So what is in the input can shape the answer, even though the model was never retrained on it.
Some AI tools add a separate step. Before answering, they search a set of documents and hand the relevant passages to the model. In one 2020 design, that search covered an index of Wikipedia, and adding it made the answers more specific and more factual.4 That is a tool built around a model, not the model itself. Module 100.6 covers these open-book tools. A plain model has no such step. It writes from the patterns it learned and the text in front of it, not from a lookup.
The misconception this corrects: "The model looks the answer up somewhere." It does not. It predicts what a good answer would probably say.
2. Sounding sure is not the same as being right
Because the model is built to produce likely text, it is very good at producing text that sounds right. In 2020, people were asked to tell real news articles from articles written by GPT-3. For the largest model, their accuracy was about 52%, where 50% is what you would get by guessing.2 That study tested whether people could tell who wrote an article, not whether the articles were true. It shows how hard it is to spot model writing by how it reads.
Fluent writing can also be false. That is a separate point. NIST names the failure. It calls it confabulation: "the production of confidently stated but erroneous or false content", known in everyday speech as "hallucinations".1 So fluent, confident writing does not show that a text is true. Three points from NIST's description matter for every learner:1
- Confabulation can happen even when the model is working the way it was designed to. NIST calls it "a natural result of the way generative models are designed", because they produce outputs that fit the patterns of their training data. This is about why it happens, not how often.
- Confidence is part of the danger. Harm can follow "when users believe false content – often due to the confident nature of the response".
- A model can also make up the support for its answer, including "confabulated logic or citations", which can make people trust a wrong answer even more.
A model can also be wrong in ways that are not obvious from one sentence. The GPT-3 authors reported that its longer writing could repeat itself, lose coherence and contradict itself.2
A real case: the company corrects its own chatbot. On 21 March 2023, Google published a post inviting people to try its chatbot, Bard, and share feedback. The post warned that tools like this "can provide inaccurate, misleading or false information while presenting it confidently". It then showed one of Bard's own answers about plants, and pointed out the mistake: "Bard got some things wrong, like the scientific name for the ZZ plant — actually Zamioculcas zamiifolia, not Zamioculcas zamioculcas."5 The two names look alike. Reading the answer would not catch the error. Checking the name against another source would. This is one error the company itself reported. It does not tell you how often Bard, or any model, gets things wrong.
A 2021 test showed that being bigger did not make the models of that time more truthful. TruthfulQA asked language models 817 questions across 38 topics, including health, law, finance and politics. The best model tested was truthful on 58% of the questions. The study's human baseline was 94%, but that was one person, who answered 250 of the questions with access to the internet, not a large group taking the whole test. The largest models were generally the least truthful.6 These are results for the models tested in 2021, not a score for the tools you use as of November 2026.
The lesson that lasts is the method. The only way to know whether an answer is true is to check it against something other than the answer itself: the original document, a measurement, or a person who knows. Asking the same model again does not count. A Foundation editorial summary by Bob Rapp puts it this way: "a second question to the same model is not a check".7 And if you could not check it, say so. "Not checked" is an honest label. The Foundation's Practice site promises: "'I don't know yet' is always an answer, and it always pairs you with someone."8 Then find that someone.
The misconception this corrects: "If it sounds sure, it is right." Tone tells you how the text was written, not whether it is true.
3. Help when checked, or never alone
The same tool can be good AI or bad AI. What decides it is the job you give it, and whether a person checks the result. NIST's guidance on generative AI points the same way. It warns about people over-relying on these systems and about automation bias, which means trusting automated output too readily.1 It says false content matters most where AI feeds "consequential decision making". Its example is a made-up summary of patient records that could lead doctors to a wrong diagnosis.1 Among its suggested actions, it lists checking AI output with human oversight and fact-checking techniques.1 It also notes that organisations using generative AI may need additional human review.1
This course uses a simple sort. Ask two questions about any task. If the answer is wrong, how bad is it? Can a person check it before anyone relies on it?
| AI can help, if a person checks the result | AI should not be trusted with this alone |
|---|---|
| A first draft of an email or report that you will read and correct | A safety setting on a machine or process |
| A summary of a long document that you will also read | A number that goes into the books, a price or a quote to a customer |
| Ideas for a meeting, a name or a slogan | A promise, a contract term or legal advice |
| Rewording something you wrote, in simpler words | A decision about a person: hiring, pay, discipline |
| Explaining a term or a fault code, as a starting point to check | A fact told to customers or the public that nobody checked |
The left column is not "safe". It is work where the person checking still owns the result. The right column is not "AI can never be used". AI can still help a person prepare. But a person, not the model, makes and owns the decision. Being able to write anything does not make a model able to be trusted with anything. Even the GPT-3 authors reported tasks where their model still struggled.2
The Foundation calls the point where that decision is made a gate. A gate gives one of four answers: "go ahead · only this far · wait for a person · no".9 For the right-hand column, the answer to "may the model do this alone?" is wait for a person or no. Name that person. "A human in the loop" is not enough if nobody knows who can say go ahead.
A real case: citations nobody opened. In Mata v. Avianca, a US federal court found in 2023 that lawyers and their firm "abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question."10 The court also found that before one lawyer signed a filing, that lawyer "did not review any judicial authorities cited in his affirmation."10 A court filing is right-hand-column work. The missing check was plain: open each cited case in the court's own records. This case shows what happens when plausible output is filed unchecked. It does not show how often models invent citations, or why this one did.
A Foundation editorial summary by Bob Rapp about the case, under the heading "The gate that would have caught it", states this rule: "every consequential AI output is independently verified against the system of record before it is used or filed."7 That is the Foundation's proposed rule, not the court's ruling, and the post is not evidence of the case facts; those come from the court's opinion.
At Big Widget Company (a made-up company; the people are made up too). In both stories, assume the AI tool is one the company has approved for this kind of work.
A robot in Maya's paint shop stops with a fault code.
- Good AI: Maya asks the approved AI tool what the code usually means. She treats the answer as a starting point and checks it against the robot's manual and with the maintenance team.
- Bad AI: Maya follows the AI's confident answer straight away and changes a safety setting. Nobody checked whether the answer was true. At Big Widget, only the maintenance lead, a person, may say go ahead on a safety setting.
Luis, a finance analyst, asks the approved AI tool to draft the note that explains this month's numbers to his manager. That is good AI if the company allows that tool to see these figures and Luis checks every figure against the ledger before sending it. Checking the figures does not make it good AI if the tool was not allowed to see the ledger in the first place. It is bad AI if the tool's numbers go into the books unchecked.
Keep the evidence, not the secrets. If you keep a record that a check happened, keep private material out of anything you share. The Foundation's receipt carries "hashes, decision, tier, policy hash, key id". It never carries "prompts, outputs, payloads, personal data".8 Luis's ledger figures and the AI's draft stay in his company's own records, if his workplace allows him to keep them.
The three yeses: where this module fits. The Foundation names three moves: Get to Yes, Stay at Yes, Recover to Yes.9 This module gives you only the first questions of each. Module 100.7 covers which tools and information are allowed, and module 100.9 covers all three yeses in full.
- Get to Yes, before you use it. Is this a job AI is good at? Is the tool approved, and is the information safe to share? Who checks the result, and who owns it? Who says go ahead? A sure tone answers none of these.
- Stay at Yes, while you use it. Check each output that matters, each time. One good answer does not make the next one right.
- Recover to Yes, when a wrong answer gets through. Stop relying on it, name the owner, fix it, and change the rule so it is caught next time. Call it fixed only when you have the evidence.
The misconception this corrects: "A tool that can write anything can be trusted with anything."
Words to know
- Model: the trained system that produces the output.
- Generative AI: AI that makes new content, such as text, pictures or code.
- Language model: a generative model that works with text.
- Large language model (LLM): a language model built at very large scale, with a very large number of parameters. The ones in this module, like GPT-3, write text one piece at a time.2
- Parameters: the part of a model where what it learned in training is stored.4
- Token: a small piece of text. The autoregressive language models in this module, like GPT-3, predict the next token, one after another.1 2 Not every generative model works this way.
- Prompt: what you type to the AI.
- Confabulation (hallucination): confidently stated but false content.1
- Gate: the point where a decision is made to go ahead, go only this far, wait for a person, or say no.9
- Not checked: the honest label for a claim nobody has checked yet.
Further learning (not sources)
Free public course materials, for learners who want more. Some assume programming or university maths. They are links only, not citations, and nothing here is copied from them.
- Stanford University, CS324: Large Language Models: <https://stanford-cs324.github.io/winter2022/>
- Harvard University, CS50's Introduction to Artificial Intelligence with Python: <https://cs50.harvard.edu/ai/>