Level 300: Building with gen AI · Module 300.5

Two coworkers at a café table sort blank answer cards into a green-edged pile and an orange-edged pile; one holds a pencil over a blank notepad. The chalkboard reads “Out of how many?”
Out of how many?

Evaluating what you built · Plain gear

You will be able to: test each part of your AI build on its own, and count each part's results out of the right number; check an AI marker against a person before you let it mark your work; and say the honest number, even when the honest answer is "we can't count that yet".

This module is as of November 2026. Reviewed and published, version 1.0.0. It does not earn you a credential. Big Widget Company is made up, and so are all its numbers here.

1. Test each part on its own

The shopper and the cook

Think of a kitchen with two jobs. A shopper fetches the ingredients. Then a cook makes the dish.

If dinner tastes wrong, whose fault is it? Maybe the shopper brought the wrong things. Maybe the shopper got it right and the cook made a mess. If you only taste the dinner, you can't tell.

An AI assistant that looks things up works the same way. First it finds passages in a set of documents. Then the AI writes an answer from them. An AI agent adds one more job: it asks to use tools, like opening a ticket, and the system decides yes, wait for a person, or no.

The people who made a well-known way to test these assistants, called RAGAs, say the testing is hard because there are several things to check. Does the search find the right passages? Does the AI use them faithfully? Is the writing good?1 So they check each one on its own.1

Six things to count

In Level 200 you made a small test: real-looking questions, each with a known good answer. Now you keep that test, but you look at each step, not just the final answer.

  1. Finding. Did the search bring back the passage with the answer?
  2. Sticking to it. Did the answer only say things the passages back up?
  3. After a miss. When the search missed, did the AI say "I didn't find it", or answer anyway?
  4. Saying "I don't know". When the answer isn't in the documents at all, does it say so?
  5. Waiting. When the rules say a person must approve, does the agent wait?
  6. Stopping. When someone hits stop, does it really stop?

Each one gets its own "out of how many".

Here is the tricky one. You judge "sticking to it" only for questions where the search found the right passage. That way a search mistake isn't counted as a writing mistake.

But that doesn't let the cook off the hook. If the shopper brings the wrong things, a good cook says "I can't make this dish." A bad cook serves something else and calls it dinner. So check that too: out of the times the search missed, how often did the AI answer anyway instead of saying so?

Answering anyway is not always making things up. Say someone asks the price of part A, but the search only found the page about part B. The AI says "Part B costs \$5". That is true, and the page backs it up. But it doesn't answer the question, so it still fails. If you also want to know how often the AI made things up after a miss, check those answers against the pages it found. Count that on its own.

One question can go wrong in two places. You don't have to pick just one to blame. Each check has its own count.

You still want the final score: how many final answers were right and useful? But you get it by marking each final answer, the way you did in Level 200. You can't get it by taking away the other counts. An answer can stick to its passages and still not answer the question.

A worked example

Big Widget's dealer team built an assistant for warranty rules. It can also open a service ticket. They test it with 40 questions. Every answer is somewhere in their documents. They also run 12 "must wait" tests and 4 "stop" tests.

The 29 out of 40 tells you something is wrong. It doesn't tell you what. The step counts do. Six misses mean: fix the search or the documents. Made-up answers, after a miss and when the passage was there, mean: fix how the AI writes, so it says when it hasn't found the answer. And the skipped approval and the missed stop must be fixed before it goes live, however good the answers are.

If the search never misses in a test, the "after a miss" check has nothing to count. Zero out of zero is not a perfect score. It means you haven't tested that yet. Add questions the search is likely to miss.

The US standards agency NIST says the same. Write down your tests and how you scored them.2 Test in conditions like the real job.2 Check the sources the AI points to.3

2. Check the AI marker before it marks your work

Step 1, SET · COURSE STEP: Write the rule first. Set the false-accept limit and minimum sample before grading; include hard cases. Step 2, GRADE [s3, s4]: An independent person grades. Use the rubric without seeing the model grader’s score; then run the grader. Step 3, COMPARE · COURSE STEP: Count agreements and errors. A false accept passes an answer the person failed; count false rejects too.
Course calibration procedure. NIST supports independent assessment and feedback (s3, s4), not a particular sample size or release threshold. Both graders use the same answers.
Figure data as text

Calibrate before grading

  1. SET · course step · Write the rule first — set the false-accept limit and minimum sample before grading; include hard cases.
  2. GRADE [s3, s4] · An independent person grades — use the rubric without seeing the model grader’s score; then run the grader.
  3. COMPARE · course step · Count agreements and errors — a false accept passes an answer the person failed; count false rejects too.

Objective: 300.5-b. Text equivalent of figures/calibrate-before-grading.svg.

A helper who marks homework

Marking hundreds of answers by hand is slow. So teams often use an AI to mark the answers. It's like a teacher getting a helper to mark homework.

A big study in 2023 found that strong AI markers agreed with people more than 8 times in 10. That is about as often as people agree with each other.4 Good news. But that was on the study's own questions.4 It doesn't tell you how well your AI marker marks your answers.

The same study found AI markers have habits:4

One fix from the study: ask twice, swapping the order of the two answers. Only count a win if the same answer wins both times.4

The RAGAs makers tested their own AI marker against people too. It matched people more often on some kinds of check than others. They said one kind was "the hardest" to mark.1 Even the inventors' own marker was uneven.

Check the helper against the teacher

Before you trust the helper, check it.

  1. Write your rule first. For example: "out of every 8 answers a person says are wrong, the AI marker may let through at most 1." Write it before you look, so nobody can move the line later.
  2. A person marks a pile of answers. They don't peek at the AI's marks. It should be someone who didn't build the system. NIST says checks by people who weren't the builders help.23
  3. The AI marks the same pile.
  4. Count the misses that matter. The worst one is when the person says "wrong" and the AI says "fine". That's a wrong answer that would slip through. Call it a false accept.

A worked example

The warranty team takes 30 answers. The person marks 8 of them wrong. The AI marker agrees about most answers. But it says "fine" to 2 of those 8 wrong ones.

The rule allowed 1 out of 8. The AI let through 2. So the AI marker can't be the one that says the assistant is ready. A person marks it instead, or the team fixes the AI marker and checks it again.

Who decides that? The person in charge of the launch, using the rule they wrote first. The AI marker doesn't get to decide whether it can be trusted.

Watch out for one thing. The AI and the person agreed on 26 of the 30 answers. That sounds great. But it hides the 2 wrong answers that would have slipped through. Always count the false accepts on their own.

And if your pile has too few answers the person marked wrong, you can't judge the marker yet. Mark more.

The person has to really look

A check against a person only works if the person really reads. In 2023 two people insured by Cigna, a health insurer, sued it. Their court papers claim that Cigna's doctors turned down more than 300,000 payment requests in two months in 2022. They claim the doctors spent 1.2 seconds on each, on average, and signed them off in batches.5 These are claims in a lawsuit, not proven facts. The numbers come from news reporting that the papers point to.5 We have not checked how the case ended.

The lesson still holds. Clicking "fine" in a second is not checking. Write down who marked each answer, and when, in the team's own records. That shows someone was there. It doesn't prove they read it.

3. Say the honest number

Four questions before you share a number

In Level 100 you learned to ask "out of how many?" about other people's claims. Now ask it about your own. Before you share a number, say:

  1. What did you count? Questions? Answers? Tool uses?
  2. Out of how many? And did you leave any out?
  3. Who marked them? A person, or an AI marker that passed its check?
  4. When, and which version? AI tools and documents change.

NIST says to test before launch and keep testing after.2 It also warns not to judge a system from a few quick tries, and says lab tests can miss what happens in real life.3

"We can't count that yet"

Sometimes you can't count something. Maybe the live assistant's record keeps the question and the answer, but not which passages it found. Then you can't count how often the search missed in real use.

Don't guess. Say "we can't count that yet". Say why, and say what you'd need to record to count it. NIST says the same: write down what you can't measure, and why.23

A worked example: "95% accurate"

A slide says: "Warranty assistant: 95% accurate." Behind it is a made-up record of 200 questions. Dealers gave 190 of the answers a thumbs-up.

The Cigna numbers are the same kind of puzzle. "More than 300,000 turned down" is a claimed count.5 On its own it doesn't say how many were wrong, or out of how many requests. So the honest answer is "can't count that from these numbers".

The honest report

Tested on 18 Nov 2026. Search missed 6 out of 40. After those misses, the AI answered anyway 6 out of 6 times, and each time it made something up. Answers said unbacked things 5 out of 34. A person marked the final answers: 29 out of 40 were right and useful. The agent skipped an approval 1 out of 12, and missed a stop 1 out of 4: both must be fixed before launch. The AI marker let through 2 out of 8 wrong answers, more than our rule allows, so a person marked this test. We can't count real-use misses yet, because the record doesn't keep them. Next step: keep the passages it finds. The team lead owns that.

It's less shiny than "95% accurate". But it's true. And it tells the person in charge what to fix.

"We can't count that yet" is fine to say, as long as you say who will find out. The Foundation puts it this way: "I don't know yet" is always an answer, "and it always pairs you with someone".6

Keep private things out of the receipt

Testing makes two kinds of record. The test log has the questions, the passages, the answers and the marks. It stays in the team's own records. A receipt is the record a gate leaves when it makes a decision. The Foundation's rule is that it "holds no private content".7 It never carries "prompts, outputs, payloads, personal data".6 So no dealer questions and no answers go in it.

The honest report is a third thing. It has counts and "out of how many", but no questions or answers.

A receipt shows a gate made a decision. It doesn't show the test numbers were right. Only the test log, checked by someone who didn't build the system, can show that.

The three yeses

The Foundation names three moves: Get to Yes before anything runs or ships, Stay at Yes while it runs, and Recover to Yes after something goes wrong.7 Here is the testing part of each.

Sources

  1. s2 Shahul Es, Jithin James, Luis Espinosa-Anke, Steven Schockaert (2024). RAGAs: Automated Evaluation of Retrieval Augmented Generation. Paper. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩
  2. s4 National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. Official doc. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩ ↩ ↩
  3. s3 National Institute of Standards and Technology (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). Official doc. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩ ↩
  4. s1 Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Paper. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩ ↩
  5. s5 Plaintiffs' counsel, filed in the United States District Court for the Eastern District of California (2023). Kisting-Leung v. Cigna Corp., No. 2:23-cv-01477, Class Action Complaint (Document 1). Record. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩
  6. s7 AiGovOps Foundation (n.d.). AiGovOps Foundation Practice (home page). Official doc. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩
  7. s6 AiGovOps Foundation (n.d.). AiGovOps Foundation Manifesto. Official doc. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩

Next: the quiz