
Evaluating what you built · Standard gear
You will be able to: score each stage of a retrieval or agent build separately, each with its own denominator; check a model grader against a small sample graded by a person, and decide, against a rule you wrote down first, whether the grader may gate a release; and report the honest number, including "cannot count yet" when your logs do not record what you need.
This module is as of November 2026. Reviewed and published, version 1.0.0. It is not a credential. Big Widget Company is made up. Every Big Widget number in this module is invented for practice and says nothing about any real system.
In Level 200 (module 200.4) you built a small test set for one task: realistic questions with known good answers, scored against a written rubric. That told you whether the tool's answers were right and usable. In this level you have built more than a tool. You have a retrieval path (300.1, 300.2) and an agent with limits (300.3, 300.4). This module extends your test set so it tells you where a build fails, whether you can trust the thing doing the grading, and what you can honestly say about the result.
1. Score each stage, each with its own denominator
Why one pass rate is not enough
A retrieval assistant does two things. It finds passages in a chosen collection, and then a model writes an answer from them. An agent adds a third: it proposes tool calls, and the system decides whether each one runs, waits for a person, or is refused. A wrong final answer can come from any of these stages. One end-to-end pass rate adds them all together and hides which one broke.
The authors of RAGAs, a published method for grading retrieval systems, make the same point. Evaluating a retrieval system "is challenging because there are several dimensions to consider": how well retrieval finds relevant, focused passages, how faithfully the model uses them, and the quality of what it writes.1 They score those dimensions separately. One of their measures, faithfulness, asks whether "the claims that are made in the answer can be inferred from the context", that is, from the passages the system retrieved.1
The stage scoreboard
Extend your 200.4 test set so every test records what each stage did, not only the final answer. You can only do this if your build's trace shows which passages were retrieved and which tool calls were proposed, approved, refused or stopped (you read these traces in 300.2 and 300.4).
| Stage | A test fails when | Out of how many (the denominator) |
|---|---|---|
| Retrieval | The passage that holds the answer is not among the passages retrieved | Every test question whose answer is in the collection |
| Support | The answer says something the retrieved passages do not support | Only the questions where retrieval did find the right passage |
| After a miss | Retrieval missed, and the system answers anyway instead of saying it did not find the answer | Only the questions where retrieval missed |
| No answer | Nothing relevant is in the collection, and the system answers anyway instead of saying so | Every no-answer test |
| End to end | The final answer fails your 200.4 rubric (right and usable) | Every test question whose answer is in the collection |
| Approval | A call that the authority table says must wait for a named person runs without that approval | Every approval test |
| Stop | A stop is triggered, and the system keeps acting | Every stop test |
The support row is a conditional rate. It asks: when the model had the right passage, did it stick to it? Counting only those questions keeps a search failure from being charged to the writing step.
It does not make the writing step blameless when search misses. Given the wrong passages, a model can still do the safe thing and say it did not find the answer, or it can answer anyway. The "after a miss" row counts the second case, out of the misses. The no-answer tests do not cover it. They ask questions whose answer is not in the collection at all. Here the answer is in the collection, and search failed to bring it back.
Be exact about what that row counts. It counts answering instead of saying "not found". That is not the same as making something up. Say a dealer asks the price of part A, and search brings back only the page for part B. "Part B costs \$5" is supported by the passage the model was given. It is not made up. It still fails, because it answers anyway and does not answer the question. If you also want to know how many answers after a miss made unsupported claims, check each one against its retrieved passages, as the support row does, and report that as a separate count with the same denominator.
One question can fail at more than one stage. You do not have to pick one guilty stage for each failed question. Each row asks its own question, with its own denominator, and the rows are not meant to add up.
The end-to-end row is graded directly: a person (or a grader that has passed the check in section 2) marks each final answer against your 200.4 rubric. You cannot get it by subtracting the other rows. An answer can stick to its passages and still miss the question, leave out the condition that was asked about, or quote a true sentence that does not answer it. RAGAs scores that separately, as answer relevance, alongside faithfulness.1
The last two rows are about the agent, not the answers. Each approval test is a call your authority table (300.4) says must wait, run on purpose in the sandbox to see whether it waits. Each stop test triggers a stop and checks from the trace that nothing else happened after it. These rows keep their own denominators and are never folded into an answer pass rate. In this course, one approval skipped or one stop ignored is a defect to fix before launch, not a percentage to weigh against good answers.
Worked example: the warranty-policy assistant
Big Widget's dealer-support team has built a retrieval assistant over its warranty policies, with an agent that can open a service ticket. They run 40 test questions, every one with its answer somewhere in the collection, plus 12 approval tests and 4 stop tests.
- Retrieval. In 6 of the 40 questions the right passage was not retrieved. Retrieval misses: 6 of 40 (15%).
- Support. That leaves 34 questions where the right passage was retrieved. In 5 of them the answer made a claim the passages did not support. Unsupported answers, given the right passage: 5 of 34 (about 15%).
- After a miss. In all 6 misses, the system answered anyway instead of saying it had not found the answer: 6 of 6 answered anyway. The team also checked those 6 answers against the passages that were retrieved. Each one stated a warranty term those passages did not support: 6 of 6 unsupported.
- End to end. A person graded every final answer against the 200.4 rubric: 29 of 40 passed (72.5%). In this run the 29 happen to be exactly the answers with neither a miss nor an unsupported claim. The team found that by grading, not by subtraction.
- Approval. One of the 12 calls that should have waited for a person ran without approval: 1 of 12 failed.
- Stop. In one of the 4 stop tests the agent opened a ticket after the stop: 1 of 4 failed.
The end-to-end figure, 72.5%, cannot tell the team what to fix. The stage scores can. Six misses point at the collection or the search. Six made-up answers after those misses, and five unsupported answers when the passage was there, point at the instructions, the model or how passages are passed to it: it should say when it has not found the answer. The approval and stop failures block launch on their own, whatever the answer scores say.
One more caution. If a run has no retrieval misses, the "after a miss" row has nothing to count. Zero out of zero is not a perfect score. Report "no misses in this run, so after-a-miss behaviour was not tested", and add test questions that are likely to miss.
NIST's AI Risk Management Framework asks for the same discipline: "Test sets, metrics, and details about the tools used during TEVV [test, evaluation, verification and validation] are documented", and performance is "demonstrated for conditions similar to deployment setting(s)".2 NIST's generative AI profile adds that organisations should "Review and verify sources and citations in GAI system outputs" before deployment and while the system runs.3 Writing down each stage's test, count and denominator is how you do that.
2. Check the grader before it grades your build
Figure data as text
Calibrate before grading
- SET · course step · Write the rule first — set the false-accept limit and minimum sample before grading; include hard cases.
- GRADE [s3, s4] · An independent person grades — use the rubric without seeing the model grader’s score; then run the grader.
- COMPARE · course step · Count agreements and errors — a false accept passes an answer the person failed; count false rejects too.
Objective: 300.5-b. Text equivalent of figures/calibrate-before-grading.svg.
Model graders are useful, and they can be wrong
Grading hundreds of answers by hand is slow, so many teams use a model as the grader. RAGAs itself works this way: its measures are computed "by prompting an LLM", and they are "reference-free", meaning they do not need a known good answer for each question.1
The best-known study of model graders is Zheng and colleagues' 2023 paper, Judging LLM-as-a-Judge. It found that strong model judges like GPT-4 "can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans."4 That is a real result, and it is about their tasks: agreement is measured on choices between two answers to the paper's own benchmark questions.4 It does not tell you how well your grader, with your rubric, grades your warranty answers. Only a check on your own data can tell you that.
The same paper names the ways model judges go wrong:4
- Position bias: "a propensity to favor certain positions over others", such as whichever answer comes first.
- Verbosity bias: favouring "longer, verbose responses, even if they are not as clear".
- Self-enhancement bias: judges "may favor the answers generated by themselves".
- Weak at maths and reasoning: the paper reports "Limited capability in grading math and reasoning questions". That matters at Big Widget, where answers carry torque values, part numbers and dates.
The paper also gives a fix for position bias: "call a judge twice by swapping the order of two answers and only declare a win when an answer is preferred in both orders."4
The RAGAs authors checked their own grader against people, too. On their test set, built from 50 Wikipedia pages, its choices between pairs of answers (or, for context relevance, pairs of retrieved passages) matched human annotators' with an accuracy of 0.95 on faithfulness, 0.78 on answer relevance and 0.70 on context relevance, and they "found context relevance to be the hardest quality dimension to evaluate".1 Even the grader's designers found it more reliable on some kinds of check than others, on their own data. Yours will differ.
NIST's generative AI profile suggests assessing output "by comparing it to a set of known ground truth data and by using a variety of evaluation methods (e.g., human oversight and automated evaluation ...)".3 A calibration sample does exactly that.
Calibrate the grader on a sample a person grades
- Write the rule first. Before anyone grades anything, write down the most false accepts you will allow, out of how many answers the person fails, and the smallest sample you will accept. Writing it first stops anyone moving the line after seeing the result.
- Pick a sample of answers from your test runs, including hard ones.
- A person grades each answer against the rubric, without seeing the grader's score. That person should not be the one who built the system. NIST's framework calls for "Internal experts who did not serve as front-line developers for the system and/or independent assessors"2, and its generative AI profile asks that people giving structured feedback are "not directly involved in system development tasks for the same GAI model".3
- Run the grader on the same answers.
- Count the four boxes: both pass, both fail, the grader passed what the person failed (a false accept), and the grader failed what the person passed (a false reject).
The false accepts matter most. A false accept is a wrong answer the grader would wave through to release.
Worked example: may the grader gate the release?
The warranty team's rule, written before the test: at most 1 false accept for every 8 answers the person fails, and at least 8 human fails in the sample.
They take 30 answers. The person passes 22 and fails 8. The grader agrees on most of them, but it passes 2 of the 8 answers the person failed, and fails 2 of the 22 the person passed.
- Agreement: 30 − 2 − 2 = 26, so 26 of 30 (about 87%).
- False accepts: 2 of 8 human fails (25%).
- Decision: the rule allows 1 in 8; the grader made 2. The grader may not gate the release. A person grades the release run, or the team fixes the grader and calibrates it again on a fresh sample.
Who makes that call matters. The person who owns the release applies the rule written in step 1. The grader does not get a say in whether it is trusted, and a good agreement figure does not make the decision for anyone.
An agreement of 87% sounds good. It hides the number that matters: a quarter of the answers a person judged wrong would have passed. Always report false accepts as their own count, out of the human fails.
Small samples are rough. With only 8 human fails, one answer either way moves the false-accept rate by 12.5 points. If a sample has fewer human fails than your rule needs, the honest answer is not "pass"; it is "cannot decide yet: grade more".
The person has to actually grade
A calibration sample is only as good as the human grading behind it. A real case shows what the opposite looks like. In July 2023 a class action, Kisting-Leung v. Cigna Corp., was filed in federal court in California. It alleges that "over a period of two months in 2022, Cigna doctors denied over 300,000 requests for payments using this method, spending an average of just 1.2 seconds “reviewing” each request", and that doctors signed off on denials "in batches".5 These are allegations in a complaint, not findings; the complaint takes the figures from ProPublica's reporting, which it cites, and how the case turned out is not checked here.5
Whatever the case decides, the lesson for your grader check holds: a person clicking "pass" in a second or two is not a human grade. Record who graded each answer and when, in the team's own work record, so a reviewer can see the grading was real. A timestamp and a name show that someone was there. They do not prove the person read the answer or got it right. Module 300.8 returns to this case as a rubber-stamp review.
3. Count honestly: what, out of how many, or "cannot count yet"
Four questions before any number leaves the team
Module 100.8 taught you to ask "out of how many?", "measured how?" and "over how long?" about other people's claims. Now ask them of your own. Before you report a number, write down:
- What is counted. Questions, answers, claims inside answers, tool calls, or users? "95% accurate" means nothing until you know which.
- Out of how many. The denominator, and anything left out of it (for example, questions the system declined to answer).
- Graded by whom. A person, or a model grader that passed its calibration check?
- When, and on which version. The model version, the collection's date and the test date.
NIST's framework says AI systems "should be tested before their deployment and regularly while in operation".2 Its generative AI profile warns: "Avoid extrapolating GAI system performance or capabilities from narrow, non-systematic, and anecdotal assessments", and notes that "Measurement gaps can arise from mismatches between laboratory and real-world settings."3 So a test-set number is a number about the test set, on the day it ran. Say so.
When the log cannot answer: say "cannot count yet"
Sometimes the honest answer is that you cannot count it. Suppose the live assistant's log records each question and final answer but not which passages were retrieved. Then you cannot count live retrieval misses at all. You can still count test-set misses, because your test runs keep traces.
Do not estimate a number the log cannot give. Report "cannot count yet", say why, and say what logging would make it countable. NIST's framework asks for exactly this: "The risks or trustworthiness characteristics that will not – or cannot – be measured are properly documented."2 The generative AI profile adds: document risks "that cannot be measured quantitatively, including explanations as to why some risks cannot be measured".3 The course's hold prompts use the same rule for the coverage line you will write in 300.P: how many actions the gate saw, out of how many, or "cannot count yet". The Foundation's practitioner's checklist puts it in the first person: "I know my system's coverage number, or I know that I do not."6
Worked example: the "95% accurate" slide
A slide at Big Widget's monthly review says: "Warranty assistant: 95% accurate." Behind it is a synthetic log of 200 live questions, each with the final answer and a thumbs-up or thumbs-down from the dealer who asked. 190 got a thumbs-up.
- The question to ask: 95% of what, out of how many, graded by whom? Here it is 190 thumbs-up out of 200 dealer ratings. That is dealer satisfaction, not accuracy checked against the policy.
- What you can count from this log: thumbs-up, 190 of 200.
- What you cannot count: how many answers were actually right against the policy, and how many were retrieval misses, because the log records neither a grade against the policy nor the passages retrieved. For both: "cannot count yet". To make them countable, log the retrieved passages and have a person, or a calibrated grader, grade a sample.
Counts without denominators: the Cigna figures
The Cigna complaint's alleged figures, over 300,000 denials in two months at an average of 1.2 seconds each,5 are a useful drill in the same habit. They are counts of acts. On their own they do not say how many of those denials were wrong, out of how many claims, or how the time was spread across reviewers: an average hides the fast and the slow. From the alleged totals alone, the answer to "how many were wrong?" is "cannot count". Module 300.8 works with a synthetic hourly log to set budgets, rather than dividing these totals.
The honest report
Put it together in a few plain lines. For the warranty assistant:
Tested 18 Nov 2026 on the sandbox build, model and collection versions recorded in the test log. Retrieval misses: 6 of 40 questions. Unsupported answers: 5 of the 34 with the right passage. After a miss: 6 of 6 answered anyway instead of saying the answer was not found, and all 6 made unsupported claims. End to end, graded by a person against the 200.4 rubric: 29 of 40 passed. Approval tests: 1 of 12 ran without approval (blocks launch). Stop tests: 1 of 4 failed to stop (blocks launch). Model grader: 2 false accepts in 8 human fails, over the preset limit of 1, so a person graded this run. Not counted: live retrieval misses and live accuracy; the live log does not record passages or grades. Cannot count yet. Next step: log retrieved passage ids and grade a live sample; owner: the warranty team's lead.
That report is less impressive than "95% accurate". It is also true, and it tells the holder of the system (300.8) exactly what to fix. "Cannot count yet" is an honest answer, and it comes with a next step and someone who owns it. The Foundation puts it this way: "I don't know yet" is always an answer, "and it always pairs you with someone".7
Keep private things out of the receipt
An evaluation makes two kinds of record, and they are not the same thing. The test log holds the questions, the retrieved passages, the answers, and who graded each one and when. It stays in the team's controlled work records. A receipt of a gate decision is different. The Foundation's rule is that a gate leaves "a signed receipt that holds no private content".6 A receipt carries "hashes, decision, tier, policy hash, key id", and never "prompts, outputs, payloads, personal data".7 So no dealer questions, no answers and no warranty passages go into it. The honest report above is a third thing: counts, denominators and versions, with no questions or answers in it.
A receipt shows that a gate made a decision under a given policy. It does not show that the numbers behind the decision were right. Only the test log, checked by someone who did not build the system, can show that.
The three yeses: the evaluating part
The Foundation names three moves. Get to Yes comes "before a single action runs", and also "before a rule or a system ships". Stay at Yes is "while it runs, in real time". Recover to Yes is "after — on demand, on incident".6 This module covers the evaluating part of each. Module 300.8 covers holding the system once it runs.
- Get to Yes. The stage scores and the grader check go to the person who owns the release, here the warranty team's lead. They decide; the test does not, and the grader does not. A gate gives one of four answers: "go ahead · only this far · wait for a person · no".6 Here, with an approval skipped and a stop ignored, the answer is no until both are fixed and re-tested. Good answer scores cannot buy that back.
- Stay at Yes. A yes given once is not a yes for good. Re-run the stage scoreboard, and check the grader again on a fresh sample, whenever the model, the collection, the rubric or the grader changes, and on a schedule while it runs.
- Recover to Yes. Say a made-up warranty term reaches a dealer. Stop relying on the assistant for that kind of question, name the owner, and add the failing case to the test set. Stopping is not recovering. The team can call it recovered only when a re-run shows the fix holds, with the counts and denominators to prove it.