Level 300: Building with gen AI · Module 300.5

Two coworkers at a café table sort blank answer cards into a green-edged pile and an orange-edged pile; one holds a pencil over a blank notepad. The chalkboard reads “Out of how many?”
Out of how many?

Evaluating what you built · Standard gear

You will be able to: score each stage of a retrieval or agent build separately, each with its own denominator; check a model grader against a small sample graded by a person, and decide, against a rule you wrote down first, whether the grader may gate a release; and report the honest number, including "cannot count yet" when your logs do not record what you need.

This module is as of November 2026. Reviewed and published, version 1.0.0. It is not a credential. Big Widget Company is made up. Every Big Widget number in this module is invented for practice and says nothing about any real system.

In Level 200 (module 200.4) you built a small test set for one task: realistic questions with known good answers, scored against a written rubric. That told you whether the tool's answers were right and usable. In this level you have built more than a tool. You have a retrieval path (300.1, 300.2) and an agent with limits (300.3, 300.4). This module extends your test set so it tells you where a build fails, whether you can trust the thing doing the grading, and what you can honestly say about the result.

1. Score each stage, each with its own denominator

Why one pass rate is not enough

A retrieval assistant does two things. It finds passages in a chosen collection, and then a model writes an answer from them. An agent adds a third: it proposes tool calls, and the system decides whether each one runs, waits for a person, or is refused. A wrong final answer can come from any of these stages. One end-to-end pass rate adds them all together and hides which one broke.

The authors of RAGAs, a published method for grading retrieval systems, make the same point. Evaluating a retrieval system "is challenging because there are several dimensions to consider": how well retrieval finds relevant, focused passages, how faithfully the model uses them, and the quality of what it writes.1 They score those dimensions separately. One of their measures, faithfulness, asks whether "the claims that are made in the answer can be inferred from the context", that is, from the passages the system retrieved.1

The stage scoreboard

Extend your 200.4 test set so every test records what each stage did, not only the final answer. You can only do this if your build's trace shows which passages were retrieved and which tool calls were proposed, approved, refused or stopped (you read these traces in 300.2 and 300.4).

StageA test fails whenOut of how many (the denominator)
RetrievalThe passage that holds the answer is not among the passages retrievedEvery test question whose answer is in the collection
SupportThe answer says something the retrieved passages do not supportOnly the questions where retrieval did find the right passage
After a missRetrieval missed, and the system answers anyway instead of saying it did not find the answerOnly the questions where retrieval missed
No answerNothing relevant is in the collection, and the system answers anyway instead of saying soEvery no-answer test
End to endThe final answer fails your 200.4 rubric (right and usable)Every test question whose answer is in the collection
ApprovalA call that the authority table says must wait for a named person runs without that approvalEvery approval test
StopA stop is triggered, and the system keeps actingEvery stop test

The support row is a conditional rate. It asks: when the model had the right passage, did it stick to it? Counting only those questions keeps a search failure from being charged to the writing step.

It does not make the writing step blameless when search misses. Given the wrong passages, a model can still do the safe thing and say it did not find the answer, or it can answer anyway. The "after a miss" row counts the second case, out of the misses. The no-answer tests do not cover it. They ask questions whose answer is not in the collection at all. Here the answer is in the collection, and search failed to bring it back.

Be exact about what that row counts. It counts answering instead of saying "not found". That is not the same as making something up. Say a dealer asks the price of part A, and search brings back only the page for part B. "Part B costs \$5" is supported by the passage the model was given. It is not made up. It still fails, because it answers anyway and does not answer the question. If you also want to know how many answers after a miss made unsupported claims, check each one against its retrieved passages, as the support row does, and report that as a separate count with the same denominator.

One question can fail at more than one stage. You do not have to pick one guilty stage for each failed question. Each row asks its own question, with its own denominator, and the rows are not meant to add up.

The end-to-end row is graded directly: a person (or a grader that has passed the check in section 2) marks each final answer against your 200.4 rubric. You cannot get it by subtracting the other rows. An answer can stick to its passages and still miss the question, leave out the condition that was asked about, or quote a true sentence that does not answer it. RAGAs scores that separately, as answer relevance, alongside faithfulness.1

The last two rows are about the agent, not the answers. Each approval test is a call your authority table (300.4) says must wait, run on purpose in the sandbox to see whether it waits. Each stop test triggers a stop and checks from the trace that nothing else happened after it. These rows keep their own denominators and are never folded into an answer pass rate. In this course, one approval skipped or one stop ignored is a defect to fix before launch, not a percentage to weigh against good answers.

Worked example: the warranty-policy assistant

Big Widget's dealer-support team has built a retrieval assistant over its warranty policies, with an agent that can open a service ticket. They run 40 test questions, every one with its answer somewhere in the collection, plus 12 approval tests and 4 stop tests.

The end-to-end figure, 72.5%, cannot tell the team what to fix. The stage scores can. Six misses point at the collection or the search. Six made-up answers after those misses, and five unsupported answers when the passage was there, point at the instructions, the model or how passages are passed to it: it should say when it has not found the answer. The approval and stop failures block launch on their own, whatever the answer scores say.

One more caution. If a run has no retrieval misses, the "after a miss" row has nothing to count. Zero out of zero is not a perfect score. Report "no misses in this run, so after-a-miss behaviour was not tested", and add test questions that are likely to miss.

NIST's AI Risk Management Framework asks for the same discipline: "Test sets, metrics, and details about the tools used during TEVV [test, evaluation, verification and validation] are documented", and performance is "demonstrated for conditions similar to deployment setting(s)".2 NIST's generative AI profile adds that organisations should "Review and verify sources and citations in GAI system outputs" before deployment and while the system runs.3 Writing down each stage's test, count and denominator is how you do that.

2. Check the grader before it grades your build

Step 1, SET · COURSE STEP: Write the rule first. Set the false-accept limit and minimum sample before grading; include hard cases. Step 2, GRADE [s3, s4]: An independent person grades. Use the rubric without seeing the model grader’s score; then run the grader. Step 3, COMPARE · COURSE STEP: Count agreements and errors. A false accept passes an answer the person failed; count false rejects too.
Course calibration procedure. NIST supports independent assessment and feedback (s3, s4), not a particular sample size or release threshold. Both graders use the same answers.
Figure data as text

Calibrate before grading

  1. SET · course step · Write the rule first — set the false-accept limit and minimum sample before grading; include hard cases.
  2. GRADE [s3, s4] · An independent person grades — use the rubric without seeing the model grader’s score; then run the grader.
  3. COMPARE · course step · Count agreements and errors — a false accept passes an answer the person failed; count false rejects too.

Objective: 300.5-b. Text equivalent of figures/calibrate-before-grading.svg.

Model graders are useful, and they can be wrong

Grading hundreds of answers by hand is slow, so many teams use a model as the grader. RAGAs itself works this way: its measures are computed "by prompting an LLM", and they are "reference-free", meaning they do not need a known good answer for each question.1

The best-known study of model graders is Zheng and colleagues' 2023 paper, Judging LLM-as-a-Judge. It found that strong model judges like GPT-4 "can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans."4 That is a real result, and it is about their tasks: agreement is measured on choices between two answers to the paper's own benchmark questions.4 It does not tell you how well your grader, with your rubric, grades your warranty answers. Only a check on your own data can tell you that.

The same paper names the ways model judges go wrong:4

The paper also gives a fix for position bias: "call a judge twice by swapping the order of two answers and only declare a win when an answer is preferred in both orders."4

The RAGAs authors checked their own grader against people, too. On their test set, built from 50 Wikipedia pages, its choices between pairs of answers (or, for context relevance, pairs of retrieved passages) matched human annotators' with an accuracy of 0.95 on faithfulness, 0.78 on answer relevance and 0.70 on context relevance, and they "found context relevance to be the hardest quality dimension to evaluate".1 Even the grader's designers found it more reliable on some kinds of check than others, on their own data. Yours will differ.

NIST's generative AI profile suggests assessing output "by comparing it to a set of known ground truth data and by using a variety of evaluation methods (e.g., human oversight and automated evaluation ...)".3 A calibration sample does exactly that.

Calibrate the grader on a sample a person grades

  1. Write the rule first. Before anyone grades anything, write down the most false accepts you will allow, out of how many answers the person fails, and the smallest sample you will accept. Writing it first stops anyone moving the line after seeing the result.
  2. Pick a sample of answers from your test runs, including hard ones.
  3. A person grades each answer against the rubric, without seeing the grader's score. That person should not be the one who built the system. NIST's framework calls for "Internal experts who did not serve as front-line developers for the system and/or independent assessors"2, and its generative AI profile asks that people giving structured feedback are "not directly involved in system development tasks for the same GAI model".3
  4. Run the grader on the same answers.
  5. Count the four boxes: both pass, both fail, the grader passed what the person failed (a false accept), and the grader failed what the person passed (a false reject).

The false accepts matter most. A false accept is a wrong answer the grader would wave through to release.

Worked example: may the grader gate the release?

The warranty team's rule, written before the test: at most 1 false accept for every 8 answers the person fails, and at least 8 human fails in the sample.

They take 30 answers. The person passes 22 and fails 8. The grader agrees on most of them, but it passes 2 of the 8 answers the person failed, and fails 2 of the 22 the person passed.

Who makes that call matters. The person who owns the release applies the rule written in step 1. The grader does not get a say in whether it is trusted, and a good agreement figure does not make the decision for anyone.

An agreement of 87% sounds good. It hides the number that matters: a quarter of the answers a person judged wrong would have passed. Always report false accepts as their own count, out of the human fails.

Small samples are rough. With only 8 human fails, one answer either way moves the false-accept rate by 12.5 points. If a sample has fewer human fails than your rule needs, the honest answer is not "pass"; it is "cannot decide yet: grade more".

The person has to actually grade

A calibration sample is only as good as the human grading behind it. A real case shows what the opposite looks like. In July 2023 a class action, Kisting-Leung v. Cigna Corp., was filed in federal court in California. It alleges that "over a period of two months in 2022, Cigna doctors denied over 300,000 requests for payments using this method, spending an average of just 1.2 seconds “reviewing” each request", and that doctors signed off on denials "in batches".5 These are allegations in a complaint, not findings; the complaint takes the figures from ProPublica's reporting, which it cites, and how the case turned out is not checked here.5

Whatever the case decides, the lesson for your grader check holds: a person clicking "pass" in a second or two is not a human grade. Record who graded each answer and when, in the team's own work record, so a reviewer can see the grading was real. A timestamp and a name show that someone was there. They do not prove the person read the answer or got it right. Module 300.8 returns to this case as a rubber-stamp review.

3. Count honestly: what, out of how many, or "cannot count yet"

Four questions before any number leaves the team

Module 100.8 taught you to ask "out of how many?", "measured how?" and "over how long?" about other people's claims. Now ask them of your own. Before you report a number, write down:

  1. What is counted. Questions, answers, claims inside answers, tool calls, or users? "95% accurate" means nothing until you know which.
  2. Out of how many. The denominator, and anything left out of it (for example, questions the system declined to answer).
  3. Graded by whom. A person, or a model grader that passed its calibration check?
  4. When, and on which version. The model version, the collection's date and the test date.

NIST's framework says AI systems "should be tested before their deployment and regularly while in operation".2 Its generative AI profile warns: "Avoid extrapolating GAI system performance or capabilities from narrow, non-systematic, and anecdotal assessments", and notes that "Measurement gaps can arise from mismatches between laboratory and real-world settings."3 So a test-set number is a number about the test set, on the day it ran. Say so.

When the log cannot answer: say "cannot count yet"

Sometimes the honest answer is that you cannot count it. Suppose the live assistant's log records each question and final answer but not which passages were retrieved. Then you cannot count live retrieval misses at all. You can still count test-set misses, because your test runs keep traces.

Do not estimate a number the log cannot give. Report "cannot count yet", say why, and say what logging would make it countable. NIST's framework asks for exactly this: "The risks or trustworthiness characteristics that will not – or cannot – be measured are properly documented."2 The generative AI profile adds: document risks "that cannot be measured quantitatively, including explanations as to why some risks cannot be measured".3 The course's hold prompts use the same rule for the coverage line you will write in 300.P: how many actions the gate saw, out of how many, or "cannot count yet". The Foundation's practitioner's checklist puts it in the first person: "I know my system's coverage number, or I know that I do not."6

Worked example: the "95% accurate" slide

A slide at Big Widget's monthly review says: "Warranty assistant: 95% accurate." Behind it is a synthetic log of 200 live questions, each with the final answer and a thumbs-up or thumbs-down from the dealer who asked. 190 got a thumbs-up.

Counts without denominators: the Cigna figures

The Cigna complaint's alleged figures, over 300,000 denials in two months at an average of 1.2 seconds each,5 are a useful drill in the same habit. They are counts of acts. On their own they do not say how many of those denials were wrong, out of how many claims, or how the time was spread across reviewers: an average hides the fast and the slow. From the alleged totals alone, the answer to "how many were wrong?" is "cannot count". Module 300.8 works with a synthetic hourly log to set budgets, rather than dividing these totals.

The honest report

Put it together in a few plain lines. For the warranty assistant:

Tested 18 Nov 2026 on the sandbox build, model and collection versions recorded in the test log. Retrieval misses: 6 of 40 questions. Unsupported answers: 5 of the 34 with the right passage. After a miss: 6 of 6 answered anyway instead of saying the answer was not found, and all 6 made unsupported claims. End to end, graded by a person against the 200.4 rubric: 29 of 40 passed. Approval tests: 1 of 12 ran without approval (blocks launch). Stop tests: 1 of 4 failed to stop (blocks launch). Model grader: 2 false accepts in 8 human fails, over the preset limit of 1, so a person graded this run. Not counted: live retrieval misses and live accuracy; the live log does not record passages or grades. Cannot count yet. Next step: log retrieved passage ids and grade a live sample; owner: the warranty team's lead.

That report is less impressive than "95% accurate". It is also true, and it tells the holder of the system (300.8) exactly what to fix. "Cannot count yet" is an honest answer, and it comes with a next step and someone who owns it. The Foundation puts it this way: "I don't know yet" is always an answer, "and it always pairs you with someone".7

Keep private things out of the receipt

An evaluation makes two kinds of record, and they are not the same thing. The test log holds the questions, the retrieved passages, the answers, and who graded each one and when. It stays in the team's controlled work records. A receipt of a gate decision is different. The Foundation's rule is that a gate leaves "a signed receipt that holds no private content".6 A receipt carries "hashes, decision, tier, policy hash, key id", and never "prompts, outputs, payloads, personal data".7 So no dealer questions, no answers and no warranty passages go into it. The honest report above is a third thing: counts, denominators and versions, with no questions or answers in it.

A receipt shows that a gate made a decision under a given policy. It does not show that the numbers behind the decision were right. Only the test log, checked by someone who did not build the system, can show that.

The three yeses: the evaluating part

The Foundation names three moves. Get to Yes comes "before a single action runs", and also "before a rule or a system ships". Stay at Yes is "while it runs, in real time". Recover to Yes is "after — on demand, on incident".6 This module covers the evaluating part of each. Module 300.8 covers holding the system once it runs.

Sources

  1. s2 Shahul Es, Jithin James, Luis Espinosa-Anke, Steven Schockaert (2024). RAGAs: Automated Evaluation of Retrieval Augmented Generation. Paper. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩ ↩ ↩
  2. s4 National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. Official doc. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩ ↩
  3. s3 National Institute of Standards and Technology (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). Official doc. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩ ↩ ↩
  4. s1 Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Paper. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩ ↩
  5. s5 Plaintiffs' counsel, filed in the United States District Court for the Eastern District of California (2023). Kisting-Leung v. Cigna Corp., No. 2:23-cv-01477, Class Action Complaint (Document 1). Record. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩
  6. s6 AiGovOps Foundation (n.d.). AiGovOps Foundation Manifesto. Official doc. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩ ↩
  7. s7 AiGovOps Foundation (n.d.). AiGovOps Foundation Practice (home page). Official doc. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩

Next: the quiz