
Evaluating what you built · Technical gear
You will be able to: decompose an evaluation of a retrieval-augmented or agentic build into per-stage metrics, each with an explicit denominator; calibrate a model grader against a human-graded sample, report agreement and the false-accept rate with their denominators, and apply a pre-registered release threshold; and report each metric with its unit, denominator, grader and version, or as "cannot count yet" when the telemetry does not support it.
This module is as of November 2026. Reviewed and published, version 1.0.0. It is not a credential. Primary sources are listed in sources.yml. All Big Widget figures are synthetic.
1. Per-stage evaluation
Why the end-to-end metric is not diagnostic
A retrieval-augmented system composes a retriever and a generator; an agent adds a policy layer that decides whether each proposed tool call executes, waits for approval or is refused (300.3, 300.4). An end-to-end pass rate is a function of all of them, so a change in it cannot be attributed to a stage.
Es et al. frame the problem the same way: evaluating RAG "is challenging because there are several dimensions to consider", namely the retriever's ability to identify relevant and focused passages, the generator's ability to use them faithfully, and the quality of the generation.1 RAGAs scores faithfulness, answer relevance and context relevance separately; an answer is faithful when "the claims that are made in the answer can be inferred from the context".1
Metrics and denominators
Let the test set contain $N$ answerable questions (gold passage present in the collection), $N_0$ no-answer questions, $A$ approval tests and $S$ stop tests. From the trace of each run, let:
- $M$ = answerable questions where no gold passage is in the retrieved set (retrieval misses);
- $U$ = questions with the gold passage retrieved, whose answer contains at least one claim not entailed by the retrieved passages (unsupported answers);
- $F_M$ = retrieval misses where the system answered instead of abstaining (non-abstention after a miss);
- $U_M$ = retrieval misses whose answer contains at least one claim not entailed by the retrieved passages (unsupported answers after a miss). $U_M \le F_M$, since an abstention makes no claim; computing $U_M$ needs support labels on the miss answers, as for $U$;
- $P$ = answerable questions whose final answer passes the inherited 200.4 rubric, graded directly by a person or a calibrated grader;
- $F_0$ = no-answer questions where the system answered instead of declining;
- $F_A$ = approval tests where the call executed without the required approval event;
- $F_S$ = stop tests where any action is logged after the stop event.
| Metric | Definition | Denominator |
|---|---|---|
| Retrieval miss rate | $M / N$ | $N$ |
| Unsupported-answer rate, given correct retrieval | $U / (N - M)$ | $N - M$ |
| Non-abstention rate, given a retrieval miss | $F_M / M$ | $M$ |
| Unsupported-answer rate, given a retrieval miss | $U_M / M$ | $M$ |
| No-answer failure rate | $F_0 / N_0$ | $N_0$ |
| End-to-end rubric pass rate | $P / N$ | $N$ |
| Approval failures | $F_A$ of $A$ | $A$ |
| Stop failures | $F_S$ of $S$ | $S$ |
$U/(N-M)$ is explicitly conditional: it estimates generation behaviour given adequate context, so a retrieval failure is not charged to the generator. It does not imply the generator is blameless on a miss. With the same insufficient context, one generator abstains and another confabulates; RAGAs defines faithfulness against the context actually retrieved, $c(q)$, and needs no gold passage to do so.1 $F_M/M$ and $U_M/M$ measure that behaviour. The $N_0$ tests do not: there the answer is absent from the collection, whereas a miss has the answer present but not retrieved.
$F_M$ and $U_M$ are different numerators over the same denominator; do not report one under the other's name. $F_M - U_M$ counts non-abstaining answers whose claims are all entailed by the retrieved context. Example: the question asks the price of part A; the collection holds it, but retrieval returns only part B's page; the output "Part B costs \$5" is faithful to $c(q)$, so it adds to $F_M$ but not to $U_M$. It still fails the rubric. If $M = 0$, both rates are undefined: report "not estimable in this run", not 0, and add cases likely to miss.
Stage failures are not mutually exclusive, and the rows are not a partition of failed questions; a question can fail at retrieval and at generation, and the rates need not sum to anything. $P$ is not derivable from $M$, $U$, $F_M$ and $U_M$. A fully supported answer can still be irrelevant to the question or incomplete, which RAGAs scores separately as answer relevance.1 Grade $P$ against the rubric.
Approval and stop results are reported as counts, not blended into any answer metric; in this course a single failure in either is launch-blocking.
Instrumentation precondition. Per-stage scoring needs a trace per test case recording: the retrieved passage ids, the final answer, each proposed tool call, the policy decision and approval event id (if any), and the stop event and any subsequent actions. Without these fields the metric is not computable from the run.
Worked example (synthetic)
Big Widget's warranty-policy assistant, with a ticket-opening tool: $N = 40$, $A = 12$, $S = 4$.
- $M = 6$: retrieval miss rate $6/40 = 15\%$.
- $N - M = 34$; $U = 5$: unsupported-answer rate given correct retrieval $5/34 \approx 14.7\%$.
- $F_M = 6$: no miss produced an abstention, $6/6$. Support labels on those 6 answers: each asserted a warranty term not entailed by its retrieved passages, so $U_M = 6$, $6/6$.
- $P = 29$ by direct human grading against the 200.4 rubric: $29/40 = 72.5\%$. In this run the passing set coincides with the 29 questions having neither a miss nor an unsupported answer; that coincidence is an observed fact of the grading, not an identity.
- $F_A = 1$ of 12; $F_S = 1$ of 4. Both block launch.
The 72.5% does not localise the fault; $M$ points to collection coverage or the retriever, $U$, $F_M$ and $U_M$ to the prompt, the model or context assembly (in particular, no abstention behaviour when context is insufficient).
Mapping to NIST
AI RMF 1.0 MEASURE 2.1: "Test sets, metrics, and details about the tools used during TEVV are documented." MEASURE 2.3: performance criteria are "demonstrated for conditions similar to deployment setting(s)".2 The generative AI profile, MS-2.5-003: "Review and verify sources and citations in GAI system outputs during pre-deployment risk measurement and ongoing monitoring activities."3 The per-stage table, with its traces, is the documentation those subcategories ask for.
2. Calibrating a model grader
Figure data as text
Calibrate before grading
- SET · course step · Write the rule first — set the false-accept limit and minimum sample before grading; include hard cases.
- GRADE [s3, s4] · An independent person grades — use the rubric without seeing the model grader’s score; then run the grader.
- COMPARE · course step · Count agreements and errors — a false accept passes an answer the person failed; count false rejects too.
Objective: 300.5-b. Text equivalent of figures/calibrate-before-grading.svg.
What the literature does and does not establish
RAGAs computes its metrics "by prompting an LLM" (gpt-3.5-turbo-16k in the paper) and is "reference-free", requiring no gold answers.1 Its own validation on WikiEval (50 Wikipedia pages) reports accuracy against human annotators in pairwise comparisons of 0.95 (faithfulness), 0.78 (answer relevance) and 0.70 (context relevance); the authors "found context relevance to be the hardest quality dimension to evaluate".1
Zheng et al. report that strong judges like GPT-4 "can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans".4 They define agreement as the probability that randomly selected individuals of each judge type agree on a randomly selected question, computed on pairwise comparisons (single-answer grades are converted to pairwise results).4 They document three biases, position ("a propensity to favor certain positions over others"), verbosity (favouring "longer, verbose responses, even if they are not as clear") and self-enhancement (judges "may favor the answers generated by themselves"), and "Limited capability in grading math and reasoning questions".4 Their conservative mitigation for position bias is to "call a judge twice by swapping the order of two answers and only declare a win when an answer is preferred in both orders", treating inconsistent results as a tie.4
Neither result transfers to a different grader, rubric, task or label scheme. Agreement on pairwise preference is not the same quantity as pass/fail accuracy on a rubric. The defensible inference is the course's: measure your grader on your task.
NIST AI 600-1, MP-2.3-001, suggests assessing GAI output "by comparing it to a set of known ground truth data and by using a variety of evaluation methods (e.g., human oversight and automated evaluation ...)".3
Protocol
- Pre-register the decision rule before any grading: the maximum false accepts $k$ per $h$ human fails, and the minimum $h$ required for a decision.
- Sample $n$ outputs from test runs, stratified to include hard cases.
- Human grading, blind to grader output, by someone other than the builder. AI RMF MEASURE 1.3 calls for "Internal experts who did not serve as front-line developers for the system and/or independent assessors"2; AI 600-1 MS-1.3-003 asks to verify that those conducting structured human feedback exercises are "not directly involved in system development tasks for the same GAI model".3
- Grader run on the same $n$ outputs, with order randomised or swapped where the grader compares.
- Tabulate with human labels as reference:
| Human pass | Human fail | |
|---|---|---|
| Grader pass | $a$ | $b$ (false accepts) |
| Grader fail | $c$ (false rejects) | $d$ |
- Agreement $= (a + d)/n$.
- False-accept rate $= b/(b + d)$, the share of human fails the grader passed. The denominator is the number of human fails, $h = b + d$.
- False-reject rate $= c/(a + c)$.
Report $b$ and $h$, not only the rate. The release gate cares about $b$: each false accept is a failing output the grader would pass. Share the results with whoever approves release (AI 600-1 MS-2.3-003: results go to those "with system release approval authority").3
Worked example (synthetic)
Pre-registered rule: $b \le 1$ with $h \ge 8$. Sample $n = 30$; human: 22 pass, 8 fail. Grader: $b = 2$, $c = 2$, so $a = 20$, $d = 6$.
- Agreement $= 26/30 \approx 86.7\%$.
- False-accept rate $= 2/8 = 25\%$; $h = 8$ meets the minimum.
- $b = 2 > 1$: the grader fails the rule and may not gate the release. A person grades the release run, or the grader is revised and recalibrated on a fresh sample (not the one used to tune it).
The decision is applied by the named release owner against the pre-registered rule; the grader's own output never determines whether the grader is admitted.
High agreement co-exists with an unacceptable false-accept rate because agreement is dominated by the larger class. With $h = 8$, each unit of $b$ moves the rate by 12.5 points; small $h$ gives coarse estimates, and $h$ below the pre-registered minimum yields "cannot decide yet", not a pass.
Human labels must be real labels
Calibration inherits the quality of its reference labels. Kisting-Leung v. Cigna Corp. (E.D. Cal., No. 2:23-cv-01477, complaint filed 24 July 2023) alleges at ¶2 that "over a period of two months in 2022, Cigna doctors denied over 300,000 requests for payments using this method, spending an average of just 1.2 seconds “reviewing” each request", and at ¶5 that doctors were allowed "to sign off on the denials in batches".5 These are pleadings, not findings; the figures are attributed in footnote 3 to ProPublica's March 2023 reporting, so they are not independent of it; the outcome is not checked here.5 The operational point does not depend on the case: record grader identity and timestamps per label, in the controlled work record, so that review can detect labels produced without review. Identity and timing are necessary audit evidence, not sufficient: they do not show that the grader read the output or graded it correctly. Module 300.8 treats the case as rubber-stamp review.
3. Reporting: unit, denominator, grader, version
The reporting tuple
Every reported metric carries: the unit counted (question, answer, claim, tool call, user); the denominator and any exclusions (e.g. declined questions); the grader (human, or a model grader with its calibration result); and the version and date (model version, collection snapshot, test date).
AI RMF 1.0 §5.3: "AI systems should be tested before their deployment and regularly while in operation."2 AI 600-1 MS-2.5-001: "Avoid extrapolating GAI system performance or capabilities from narrow, non-systematic, and anecdotal assessments."3 Appendix A.1.4: "Measurement gaps can arise from mismatches between laboratory and real-world settings."3 MS-4.2-002 asks for evaluation in real-world scenarios to reveal issues that "might not surface in controlled and optimized testing environments".3 A test-set metric is therefore scoped to the test set and the version tested, and the report says so.
"Cannot count yet"
If the production log stores question and answer but not retrieved passage ids, the production retrieval miss rate is not computable, whatever the test-set rate. Report it as "cannot count yet", state the missing field, and state the instrumentation change that would make it countable. Do not impute it from the test set.
This is the documentation AI RMF MEASURE 1.1 asks for: "The risks or trustworthiness characteristics that will not – or cannot – be measured are properly documented."2 AI 600-1 MS-1.1-009 adds: "Track and document ... GAI risks that cannot be measured quantitatively, including explanations as to why some risks cannot be measured".3 The course applies the same rule to the 300.P coverage line (actions seen by the gate, out of how many, or "cannot count yet"; levels/300/practice/hold-prompts.md). The Foundation's practitioner's checklist states it in the first person: "I know my system's coverage number, or I know that I do not."6
Worked example: "95% accurate" (synthetic)
Slide: "Warranty assistant: 95% accurate." Log: 200 production questions, final answer, dealer thumbs rating; 190 thumbs-up.
- Unit and grader: dealer satisfaction rating, not rubric correctness. Countable: 190/200 thumbs-up.
- Not countable: rubric correctness (no grade against the policy) and production retrieval misses (no passage ids). Both: "cannot count yet". Fix: log retrieved passage ids; human-grade a sample, or use a grader that has passed calibration.
The Cigna complaint's totals (over 300,000 denials over two months; mean 1.2 seconds)5 are the same shape of evidence: a numerator and a mean, with no error count, no claim-volume denominator and no distribution across reviewers or time. The error rate is not computable from them. Module 300.8 sets budgets on a synthetic time-bucketed log instead of dividing these totals.
Report template
Test date 2026-11-18; sandbox build; model and collection versions as recorded in the run log. Retrieval misses 6/40. Unsupported answers 5/34 (given correct retrieval). Given a miss: answered instead of abstaining 6/6; unsupported claims 6/6. End-to-end rubric pass 29/40, human-graded. Approval failures 1/12 and stop failures 1/4 (launch-blocking). Model grader: b = 2 false accepts in h = 8 human fails, above the pre-registered limit of 1; human grading used for this run. Production retrieval misses and production correctness: cannot count yet (log lacks passage ids and grades). Next step: log retrieved passage ids and human-grade a production sample; owner: the warranty team's lead.
An unknown is reported with its owner and next check. The Foundation's phrasing: "I don't know yet" is always an answer, "and it always pairs you with someone".7
Evidence boundaries: test log, report, receipt
Three artefacts, three audiences. The test log (questions, retrieved passage ids, outputs, labels, labeller identity and timestamps) is the evidence for the metrics and stays in the controlled work record. The report above carries units, counts, denominators, grader and versions, and no question or output text. A signed receipt of a gate decision is narrower again: per the Foundation, a gate "leaves a signed receipt that holds no private content",6 carrying "hashes, decision, tier, policy hash, key id" and never "prompts, outputs, payloads, personal data".7 A receipt shows that a decision was made under a given policy; it does not show that the metrics behind it were correct.
The three yeses: the evaluation part
The Foundation maps its gates to three movements: Get to Yes ("before a single action runs"; "before a rule or a system ships"), Stay at Yes ("while it runs, in real time") and Recover to Yes ("after — on demand, on incident").6 The evaluation obligations in each, as this course applies them:
- Get to Yes. Stage metrics and the calibration table go to the named release owner (here the warranty team's lead), who decides; neither the test nor the grader does; the gate's answers are "go ahead · only this far · wait for a person · no".6 With $F_A > 0$ and $F_S > 0$ the answer is no until both are fixed and re-tested, regardless of $P$.
- Stay at Yes. Re-run the stage metrics and recalibrate the grader on fresh samples whenever the model, collection snapshot, rubric or grader changes, and on a schedule in operation. A prior calibration result is scoped to the grader version and data it was measured on.
- Recover to Yes. On an incident (an unsupported claim reaching a dealer, an approval bypassed), stop relying on the affected path, name the owner, add the failing case to the test set and re-run. Recovery is claimed only on re-run evidence with counts and denominators, not on the stop or an announced fix.
Further learning (not copied; background only)
- Carnegie Mellon University, Machine Learning in Production, https://mlip-cmu.github.io/ : evaluating and operating ML systems. The book is CC BY-NC-ND 4.0 (https://mlip-cmu.github.io/book/); licences for slides, assignments and recordings are not verified. Link only.
- University of Toronto, CSC2541: Large Models (Winter 2025), https://www.cs.toronto.edu/~cmaddis/courses/csc2541_w25/ : public readings only; auditing is restricted to U of T students. No content licence was found; link only.