Level 100: How AI works: AI at work and in life · Module 100.4

Try each exercise first, then open its model answer.

Why it works, and where it breaks · Workbook

Three exercises, one for each thing this module teaches. Use your own work if you can, or the made-up Big Widget Company from the lesson. Do not paste real customer or private details into an AI tool, or into your answers here, unless your workplace allows it. Describe the task instead.

Exercise 1 (100.4-a)

Pick two tasks from your own work (or from Sam's support team at Big Widget Company).

  1. Name one task you would guess AI is good at. Say which study in the lesson it is most like, and what that study found.
  2. Name one task you would guess AI is bad at. Say which study it is most like, and what that study found.
  3. In one sentence, say why these are guesses to test, not rules.
Model answer

A good answer names two real tasks and ties each one to a study the lesson taught. For example:

  1. Good at: drafting a first reply to a common customer question for a new team member. This is like the help-desk study, where an AI helper raised problems solved per hour on average, and newer, less skilled workers gained most. It is also like the shoe tasks in the consulting study (ideas, drafts, writing), where people with AI did more, faster and better.
  2. Bad at: deciding a warranty dispute where the answer hangs on one line in a customer's old email. This is like the brand task in the consulting study: the right answer was hidden in the interview notes, and people with AI got it right less often than people without it.
  3. These come from a few studies, with the tools and people of their time. They tell me where to start testing, not what will happen on my work.

Watch for: an answer that says AI is good at everything, or useless, misses the point. So does an answer that calls "expert work" always bad for AI. That rests on one dated trial, and in the consulting study the top performers also gained on the shoe tasks.

Exercise 2 (100.4-b)

A colleague says: "AI did a brilliant job on a hard report last week. This week's report is similar, so we can skip checking it."

Write a short reply (four to six sentences) that:

Model answer

A good reply covers these points, in the colleague's language:

  • The frontier is jagged. In one study at a big consulting firm, AI helped consultants with idea, writing and analysis tasks about a new shoe. It made a different group worse on a brand recommendation, where the answer was hidden in interview notes. Both were realistic consulting work. Tasks that look alike can land on opposite sides of the edge. The study did not rate the tasks as equally hard; the point is that how a task looks does not tell you which side it is on.
  • Reason 1: failures are hard to predict. The researchers say these models fail in some situations in ways that are hard to predict in advance.
  • Reason 2: wrong answers can sound right. On the brand task, people with AI wrote more coherent and persuasive recommendations whether they were right or wrong. Only checking against the interviews showed which were wrong.
  • What to do instead. Check this report against its source material. If we use AI for this kind of report often, test it on a set of real past examples where we know the right answer.

Watch for: "it sounded confident, so it is fine" is the trap. Not every wrong answer sounds right, but how an answer reads is no evidence that it is right.

Exercise 3 (100.4-c)

Your team (or Sam's support team) feels that a new AI helper "saves loads of time". Write a one-page plan to find out whether it really does. Your plan should say:

  1. Which task you will measure, and why you will not just ask people how it feels.
  2. What you will compare, and how you will decide who or what gets the AI, so you can tell whether a change comes from the AI and not from something else.
  3. What you will count (at least one measure of speed and one of correctness, each out of how many), and who checks the answers.
  4. How long you will run it, and when you will measure again.
  5. Who looks at the result and decides.
  6. What goes in the team's work records, and what never goes into a receipt.
Model answer

A good plan covers each point. For example, for Sam's team:

  1. Task: answering one kind of customer chat. Feelings are not enough: in the 2025 developer trial, experienced programmers believed AI had made them about 20% faster, but timed, tasks took 19% longer. (METR now marks those early-2025 results out of date; the lesson is about feeling versus measuring, not about today's tools.)
  2. Comparison: for the same weeks, some of this kind of chat is handled with the helper and some without. A random draw decides which chats (or which agents) get the helper. Nobody picks, and nobody volunteers. Agents differ a lot in skill and experience, and the lesson showed that newer staff gained most in the help-desk study. So if the keen or the newest agents choose the helper, the groups differ before the AI does anything. If a random draw is not possible, match the two groups on experience and workload, and say plainly that the result is a strong hint, not proof that the AI caused the change. A plain before-and-after count is not enough on its own either, because the mix of questions, staffing or the season can change too.
  3. Counts: problems solved per hour (speed). For correctness, the checker takes answers from both groups by the same rule, for example a random sample of the same size from each, and records how many were right out of how many checked. Compare the share right, not the raw number right: the faster group answers more chats, so it can have more right answers without being right more often. Rework or reopened chats are worth counting too, the same way. A named person checks the answers, and they are not told which group each answer came from.
  4. Time: a few weeks, long enough to see a pattern. Then keep measuring while it is in use, and test again when the tool, the kinds of questions or the staff change. NIST's AI Risk Management Framework asks for testing before deployment and regularly while in operation, in conditions like real use.
  5. Decision: Sam, who owns the team's work, reads the results and decides. The AI tool does not approve itself. A good result can support "go ahead" for this one kind of chat. It does not settle whether customer data is safe to share with the tool, or what must wait for a person; modules 100.7 and 100.9 cover those.
  6. Records: customer emails, the AI's answers and the checker's marks stay in the team's controlled work records. None of that goes into a receipt; the Foundation's rule is that a gate's receipt holds no private content.

Watch for: a plan that ends with "ask the team if it feels faster", that credits a before-and-after change wholly to AI, or that tests once and never again. Also mark down a plan that lets people choose whether they use the helper and then credits the whole difference to AI, or that counts right answers without saying out of how many were checked in each group. If something has not been measured yet, the honest answer is "not checked", with a next step to test it.

Next module: 100.5 Talking to AI well