Level 300: Building with gen AI · Module 300.5

Two coworkers at a café table sort blank answer cards into a green-edged pile and an orange-edged pile; one holds a pencil over a blank notepad. The chalkboard reads “Out of how many?”
Out of how many?

Evaluating what you built · Technical gear

You will be able to: decompose an evaluation of a retrieval-augmented or agentic build into per-stage metrics, each with an explicit denominator; calibrate a model grader against a human-graded sample, report agreement and the false-accept rate with their denominators, and apply a pre-registered release threshold; and report each metric with its unit, denominator, grader and version, or as "cannot count yet" when the telemetry does not support it.

This module is as of November 2026. Reviewed and published, version 1.0.0. It is not a credential. Primary sources are listed in sources.yml. All Big Widget figures are synthetic.

1. Per-stage evaluation

Why the end-to-end metric is not diagnostic

A retrieval-augmented system composes a retriever and a generator; an agent adds a policy layer that decides whether each proposed tool call executes, waits for approval or is refused (300.3, 300.4). An end-to-end pass rate is a function of all of them, so a change in it cannot be attributed to a stage.

Es et al. frame the problem the same way: evaluating RAG "is challenging because there are several dimensions to consider", namely the retriever's ability to identify relevant and focused passages, the generator's ability to use them faithfully, and the quality of the generation.1 RAGAs scores faithfulness, answer relevance and context relevance separately; an answer is faithful when "the claims that are made in the answer can be inferred from the context".1

Metrics and denominators

Let the test set contain $N$ answerable questions (gold passage present in the collection), $N_0$ no-answer questions, $A$ approval tests and $S$ stop tests. From the trace of each run, let:

MetricDefinitionDenominator
Retrieval miss rate$M / N$$N$
Unsupported-answer rate, given correct retrieval$U / (N - M)$$N - M$
Non-abstention rate, given a retrieval miss$F_M / M$$M$
Unsupported-answer rate, given a retrieval miss$U_M / M$$M$
No-answer failure rate$F_0 / N_0$$N_0$
End-to-end rubric pass rate$P / N$$N$
Approval failures$F_A$ of $A$$A$
Stop failures$F_S$ of $S$$S$

$U/(N-M)$ is explicitly conditional: it estimates generation behaviour given adequate context, so a retrieval failure is not charged to the generator. It does not imply the generator is blameless on a miss. With the same insufficient context, one generator abstains and another confabulates; RAGAs defines faithfulness against the context actually retrieved, $c(q)$, and needs no gold passage to do so.1 $F_M/M$ and $U_M/M$ measure that behaviour. The $N_0$ tests do not: there the answer is absent from the collection, whereas a miss has the answer present but not retrieved.

$F_M$ and $U_M$ are different numerators over the same denominator; do not report one under the other's name. $F_M - U_M$ counts non-abstaining answers whose claims are all entailed by the retrieved context. Example: the question asks the price of part A; the collection holds it, but retrieval returns only part B's page; the output "Part B costs \$5" is faithful to $c(q)$, so it adds to $F_M$ but not to $U_M$. It still fails the rubric. If $M = 0$, both rates are undefined: report "not estimable in this run", not 0, and add cases likely to miss.

Stage failures are not mutually exclusive, and the rows are not a partition of failed questions; a question can fail at retrieval and at generation, and the rates need not sum to anything. $P$ is not derivable from $M$, $U$, $F_M$ and $U_M$. A fully supported answer can still be irrelevant to the question or incomplete, which RAGAs scores separately as answer relevance.1 Grade $P$ against the rubric.

Approval and stop results are reported as counts, not blended into any answer metric; in this course a single failure in either is launch-blocking.

Instrumentation precondition. Per-stage scoring needs a trace per test case recording: the retrieved passage ids, the final answer, each proposed tool call, the policy decision and approval event id (if any), and the stop event and any subsequent actions. Without these fields the metric is not computable from the run.

Worked example (synthetic)

Big Widget's warranty-policy assistant, with a ticket-opening tool: $N = 40$, $A = 12$, $S = 4$.

The 72.5% does not localise the fault; $M$ points to collection coverage or the retriever, $U$, $F_M$ and $U_M$ to the prompt, the model or context assembly (in particular, no abstention behaviour when context is insufficient).

Mapping to NIST

AI RMF 1.0 MEASURE 2.1: "Test sets, metrics, and details about the tools used during TEVV are documented." MEASURE 2.3: performance criteria are "demonstrated for conditions similar to deployment setting(s)".2 The generative AI profile, MS-2.5-003: "Review and verify sources and citations in GAI system outputs during pre-deployment risk measurement and ongoing monitoring activities."3 The per-stage table, with its traces, is the documentation those subcategories ask for.

2. Calibrating a model grader

Step 1, SET · COURSE STEP: Write the rule first. Set the false-accept limit and minimum sample before grading; include hard cases. Step 2, GRADE [s3, s4]: An independent person grades. Use the rubric without seeing the model grader’s score; then run the grader. Step 3, COMPARE · COURSE STEP: Count agreements and errors. A false accept passes an answer the person failed; count false rejects too.
Course calibration procedure. NIST supports independent assessment and feedback (s3, s4), not a particular sample size or release threshold. Both graders use the same answers.
Figure data as text

Calibrate before grading

  1. SET · course step · Write the rule first — set the false-accept limit and minimum sample before grading; include hard cases.
  2. GRADE [s3, s4] · An independent person grades — use the rubric without seeing the model grader’s score; then run the grader.
  3. COMPARE · course step · Count agreements and errors — a false accept passes an answer the person failed; count false rejects too.

Objective: 300.5-b. Text equivalent of figures/calibrate-before-grading.svg.

What the literature does and does not establish

RAGAs computes its metrics "by prompting an LLM" (gpt-3.5-turbo-16k in the paper) and is "reference-free", requiring no gold answers.1 Its own validation on WikiEval (50 Wikipedia pages) reports accuracy against human annotators in pairwise comparisons of 0.95 (faithfulness), 0.78 (answer relevance) and 0.70 (context relevance); the authors "found context relevance to be the hardest quality dimension to evaluate".1

Zheng et al. report that strong judges like GPT-4 "can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans".4 They define agreement as the probability that randomly selected individuals of each judge type agree on a randomly selected question, computed on pairwise comparisons (single-answer grades are converted to pairwise results).4 They document three biases, position ("a propensity to favor certain positions over others"), verbosity (favouring "longer, verbose responses, even if they are not as clear") and self-enhancement (judges "may favor the answers generated by themselves"), and "Limited capability in grading math and reasoning questions".4 Their conservative mitigation for position bias is to "call a judge twice by swapping the order of two answers and only declare a win when an answer is preferred in both orders", treating inconsistent results as a tie.4

Neither result transfers to a different grader, rubric, task or label scheme. Agreement on pairwise preference is not the same quantity as pass/fail accuracy on a rubric. The defensible inference is the course's: measure your grader on your task.

NIST AI 600-1, MP-2.3-001, suggests assessing GAI output "by comparing it to a set of known ground truth data and by using a variety of evaluation methods (e.g., human oversight and automated evaluation ...)".3

Protocol

  1. Pre-register the decision rule before any grading: the maximum false accepts $k$ per $h$ human fails, and the minimum $h$ required for a decision.
  2. Sample $n$ outputs from test runs, stratified to include hard cases.
  3. Human grading, blind to grader output, by someone other than the builder. AI RMF MEASURE 1.3 calls for "Internal experts who did not serve as front-line developers for the system and/or independent assessors"2; AI 600-1 MS-1.3-003 asks to verify that those conducting structured human feedback exercises are "not directly involved in system development tasks for the same GAI model".3
  4. Grader run on the same $n$ outputs, with order randomised or swapped where the grader compares.
  5. Tabulate with human labels as reference:
Human passHuman fail
Grader pass$a$$b$ (false accepts)
Grader fail$c$ (false rejects)$d$

Report $b$ and $h$, not only the rate. The release gate cares about $b$: each false accept is a failing output the grader would pass. Share the results with whoever approves release (AI 600-1 MS-2.3-003: results go to those "with system release approval authority").3

Worked example (synthetic)

Pre-registered rule: $b \le 1$ with $h \ge 8$. Sample $n = 30$; human: 22 pass, 8 fail. Grader: $b = 2$, $c = 2$, so $a = 20$, $d = 6$.

The decision is applied by the named release owner against the pre-registered rule; the grader's own output never determines whether the grader is admitted.

High agreement co-exists with an unacceptable false-accept rate because agreement is dominated by the larger class. With $h = 8$, each unit of $b$ moves the rate by 12.5 points; small $h$ gives coarse estimates, and $h$ below the pre-registered minimum yields "cannot decide yet", not a pass.

Human labels must be real labels

Calibration inherits the quality of its reference labels. Kisting-Leung v. Cigna Corp. (E.D. Cal., No. 2:23-cv-01477, complaint filed 24 July 2023) alleges at ¶2 that "over a period of two months in 2022, Cigna doctors denied over 300,000 requests for payments using this method, spending an average of just 1.2 seconds “reviewing” each request", and at ¶5 that doctors were allowed "to sign off on the denials in batches".5 These are pleadings, not findings; the figures are attributed in footnote 3 to ProPublica's March 2023 reporting, so they are not independent of it; the outcome is not checked here.5 The operational point does not depend on the case: record grader identity and timestamps per label, in the controlled work record, so that review can detect labels produced without review. Identity and timing are necessary audit evidence, not sufficient: they do not show that the grader read the output or graded it correctly. Module 300.8 treats the case as rubber-stamp review.

3. Reporting: unit, denominator, grader, version

The reporting tuple

Every reported metric carries: the unit counted (question, answer, claim, tool call, user); the denominator and any exclusions (e.g. declined questions); the grader (human, or a model grader with its calibration result); and the version and date (model version, collection snapshot, test date).

AI RMF 1.0 §5.3: "AI systems should be tested before their deployment and regularly while in operation."2 AI 600-1 MS-2.5-001: "Avoid extrapolating GAI system performance or capabilities from narrow, non-systematic, and anecdotal assessments."3 Appendix A.1.4: "Measurement gaps can arise from mismatches between laboratory and real-world settings."3 MS-4.2-002 asks for evaluation in real-world scenarios to reveal issues that "might not surface in controlled and optimized testing environments".3 A test-set metric is therefore scoped to the test set and the version tested, and the report says so.

"Cannot count yet"

If the production log stores question and answer but not retrieved passage ids, the production retrieval miss rate is not computable, whatever the test-set rate. Report it as "cannot count yet", state the missing field, and state the instrumentation change that would make it countable. Do not impute it from the test set.

This is the documentation AI RMF MEASURE 1.1 asks for: "The risks or trustworthiness characteristics that will not – or cannot – be measured are properly documented."2 AI 600-1 MS-1.1-009 adds: "Track and document ... GAI risks that cannot be measured quantitatively, including explanations as to why some risks cannot be measured".3 The course applies the same rule to the 300.P coverage line (actions seen by the gate, out of how many, or "cannot count yet"; levels/300/practice/hold-prompts.md). The Foundation's practitioner's checklist states it in the first person: "I know my system's coverage number, or I know that I do not."6

Worked example: "95% accurate" (synthetic)

Slide: "Warranty assistant: 95% accurate." Log: 200 production questions, final answer, dealer thumbs rating; 190 thumbs-up.

The Cigna complaint's totals (over 300,000 denials over two months; mean 1.2 seconds)5 are the same shape of evidence: a numerator and a mean, with no error count, no claim-volume denominator and no distribution across reviewers or time. The error rate is not computable from them. Module 300.8 sets budgets on a synthetic time-bucketed log instead of dividing these totals.

Report template

Test date 2026-11-18; sandbox build; model and collection versions as recorded in the run log. Retrieval misses 6/40. Unsupported answers 5/34 (given correct retrieval). Given a miss: answered instead of abstaining 6/6; unsupported claims 6/6. End-to-end rubric pass 29/40, human-graded. Approval failures 1/12 and stop failures 1/4 (launch-blocking). Model grader: b = 2 false accepts in h = 8 human fails, above the pre-registered limit of 1; human grading used for this run. Production retrieval misses and production correctness: cannot count yet (log lacks passage ids and grades). Next step: log retrieved passage ids and human-grade a production sample; owner: the warranty team's lead.

An unknown is reported with its owner and next check. The Foundation's phrasing: "I don't know yet" is always an answer, "and it always pairs you with someone".7

Evidence boundaries: test log, report, receipt

Three artefacts, three audiences. The test log (questions, retrieved passage ids, outputs, labels, labeller identity and timestamps) is the evidence for the metrics and stays in the controlled work record. The report above carries units, counts, denominators, grader and versions, and no question or output text. A signed receipt of a gate decision is narrower again: per the Foundation, a gate "leaves a signed receipt that holds no private content",6 carrying "hashes, decision, tier, policy hash, key id" and never "prompts, outputs, payloads, personal data".7 A receipt shows that a decision was made under a given policy; it does not show that the metrics behind it were correct.

The three yeses: the evaluation part

The Foundation maps its gates to three movements: Get to Yes ("before a single action runs"; "before a rule or a system ships"), Stay at Yes ("while it runs, in real time") and Recover to Yes ("after — on demand, on incident").6 The evaluation obligations in each, as this course applies them:

Further learning (not copied; background only)

Sources

  1. s2 Shahul Es, Jithin James, Luis Espinosa-Anke, Steven Schockaert (2024). RAGAs: Automated Evaluation of Retrieval Augmented Generation. Paper. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩ ↩ ↩ ↩
  2. s4 National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. Official doc. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩ ↩
  3. s3 National Institute of Standards and Technology (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). Official doc. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  4. s1 Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Paper. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩ ↩
  5. s5 Plaintiffs' counsel, filed in the United States District Court for the Eastern District of California (2023). Kisting-Leung v. Cigna Corp., No. 2:23-cv-01477, Class Action Complaint (Document 1). Record. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩
  6. s6 AiGovOps Foundation (n.d.). AiGovOps Foundation Manifesto. Official doc. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩ ↩ ↩
  7. s7 AiGovOps Foundation (n.d.). AiGovOps Foundation Practice (home page). Official doc. Checked 2026-11-19 by source checker, round 3 re-check (separate Claude session 48b5b024, Claude Opus 5.5). ↩ ↩

Next: the quiz