
What a model is, and isn't · Plain gear
You will be able to: say what an AI model is; say why sounding sure is not the same as being right; sort tasks into "AI can help if I check" and "AI should not do this alone."
Reviewed and published, version 1.0.2.
It learned patterns. It does not look answers up
Figure data as text
Predicting or looking up?
| Part | What it does | Lesson sources |
|---|---|---|
| Text model | Uses learned patterns and the text you give it to predict a likely continuation. | s1, s2 |
| Separate lookup tool | Searches for relevant documents and can pass their text to the model. | s5 |
A generative AI model is a computer program that makes new things. It can make words, pictures, sound or video. It learned patterns from a huge number of examples, and what it makes follows those patterns.1
This lesson is about models that work with words. They are called language models. In this course, a very big one, built with a huge number of parts, is called a large language model.
A language model does not keep a list of answers. When you ask it something, it writes a new answer, one small piece at a time. Each new piece is a likely next piece, given the pieces before it.1 That is how the word models in this lesson work. This lesson does not cover how models that make pictures or sound build what they make.
Think of the autocomplete on your phone, which suggests the next word as you type. A language model does something like that, but for whole answers. It is not looking anything up. It is writing what usually comes next.
What it learned is stored inside the model. But it can also use the words you give it right now. In 2020, the people who built one big model gave it jobs and examples just by typing them in. They did not change the model at all, and it still used them.2
Some AI tools also search a set of documents before they answer. That is an extra step built around the model. Lesson 100.6 is about those tools. The model on its own does not search. It writes from the patterns it learned and the words you give it.2 3
So remember: the model does not look the answer up. It guesses what a good answer would say.
Sounding sure is not being right
A model is very good at writing things that sound right. In one test from 2020, people tried to spot news stories written by an AI. They got it right about half the time. That is only a little better than tossing a coin.2 That test was about who wrote the story, not whether the story was true.
Sounding right and being right are different things. A model can write something false and say it in a calm, sure voice. The US standards agency, NIST, has a name for this: confabulation. Most people call it a "hallucination".1
NIST says three things about it:1
- It can happen even when the model is working the way it was built to work. It comes from the way these models are built.
- A sure voice is part of the danger. People often believe a wrong answer because it sounds so sure.
- A model can even make up the proof, like a fake source or a fake reason.
A true story: the wrong plant name. In March 2023, Google wrote about its chatbot, Bard. Google warned that tools like this can give false information "while presenting it confidently". Then it showed one of Bard's own answers about plants. Bard got the science name of the ZZ plant wrong. The right name is Zamioculcas zamiifolia. Bard said Zamioculcas zamioculcas.4 The two names look alike. You would not spot the mistake by reading. You would spot it by looking the name up somewhere else. This is one mistake. It does not tell you how often chatbots make mistakes.
In a 2021 test, AI models answered 817 tricky questions about things like health, law and money. The best model told the truth on 58 of every 100 questions. One person, who answered 250 of the questions and could use the internet, told the truth on 94 of every 100. The biggest models were often the least truthful.5 Today's tools are newer than those.
But the lesson still holds. The only way to know an answer is true is to check it somewhere else. Asking the same AI again does not count. In an editorial summary for the AiGovOps Foundation, Bob Rapp writes: "a second question to the same model is not a check".6
And if you could not check it, say so. "Not checked" is an honest answer. So is "I don't know yet".7 Then ask someone who can help.
So remember: a sure voice tells you how it was written, not whether it is true.
Help if I check, or never alone
The same AI tool can be good AI or bad AI. It depends on the job you give it, and on whether a person checks the work.
NIST warns that people can trust a machine too much, just because it is a machine. It says mistakes matter most when AI is used for big decisions. Its example: a made-up summary of a patient's records could lead a doctor to the wrong answer.1
Ask two questions about any task:
- If the answer is wrong, how bad is that?
- Can a person check it before anyone relies on it?
| AI can help, if I check it | AI should not do this alone |
|---|---|
| A first draft of an email that I read and fix | A safety setting on a machine |
| A short summary of a long report that I also read | A number for the company's books, or a price for a customer |
| Ideas for a name or a slogan | A promise or a contract |
| Putting my own words more simply | A decision about a person, like hiring |
| Explaining a word or a fault code, which I then check | A fact for customers that nobody checked |
A tool that can write anything still cannot be trusted with everything. The people who built one big model in 2020 said there were jobs it still found hard.2
For the right-hand side, a person decides. The Foundation calls this a gate. A gate has four answers: "go ahead · only this far · wait for a person · no".8 For these jobs, the AI never says go ahead on its own. Know who the person is.
A true story: cases that did not exist. In 2023, a US court looked at a case called Mata v. Avianca. Some lawyers had sent the court papers that pointed to earlier court cases. Those cases did not exist. ChatGPT had made them up, with fake quotes. The lawyers kept standing by them, even after the court asked if they were real.9 The court also found that one lawyer did not read any of the cases before signing.9 The check was simple: look each case up in the court's records. This story does not tell you how often AI makes up cases. It shows what can happen when nobody checks.
At Big Widget Company (a made-up company with made-up people). A robot in Maya's paint shop stops. It shows a fault code. Maya uses an AI tool that her company says she may use for this kind of work.
- Good AI: Maya asks the AI what the code usually means. She treats the answer as a first guess. She checks it in the robot's manual and with the maintenance team.
- Bad AI: Maya does what the AI says straight away and changes a safety setting. Nobody checked whether the answer was true. Only the maintenance boss may say go ahead on a safety setting.
Keep secrets out of what you share. You may keep a note that you checked. The Foundation's receipt, its record of a decision, never holds "prompts, outputs, payloads, personal data".7 So what you typed, what the AI wrote, and anyone's private details stay out of it.
The three yeses. The Foundation has three steps: Get to Yes, Stay at Yes, Recover to Yes.8 Here are the first questions for each.
- Get to Yes (before): Is this a job AI is good at? Am I allowed to use this tool, and to share this information with it? Who checks it? Who owns the result?
- Stay at Yes (while): Check each answer that matters, every time.
- Recover to Yes (after a mistake): Stop. Say who owns it. Fix it. Change the rule so it does not happen again.
A sure voice answers none of these. Checking the work also does not make it OK to use a tool you are not allowed to use. Lesson 100.7 is about that, and lesson 100.9 covers all three yeses.
Words to know
- Model: the trained program that makes the answer.
- Generative AI: AI that makes new things, like words or pictures.
- Language model: a model that works with words.
- Large language model: a very big language model, built with a huge number of parts.
- Prompt: what you type to the AI.
- Hallucination: a false answer that sounds sure.1
- Gate: where a person decides: go ahead, only this far, wait for a person, or no.8
- Not checked: what you say when nobody has checked yet.