Level 300: Building with gen AI · Module 300.5

Try each exercise first, then open its model answer.

Evaluating what you built · Workbook

Big Widget Company is made up. Every number in these exercises is invented for practice and says nothing about any real system.

Exercise 1 (300.5-a)

Big Widget's parts team has built a retrieval assistant over its parts catalogue, with an agent that can place a reorder. They extend their Level 200 test set and run it on the sandbox build. The traces show:

  1. Fill in a stage scoreboard: for each stage, the count of failures and the denominator.
  2. Someone says the end-to-end figure should be 50 − 8 − 6 = 36. Say why the graded figure is the one to report, and why it is not enough on its own.
  3. Say which result blocks launch whatever the answer scores are, and where the team should look to fix the retrieval misses, the answers after a miss and the unsupported answers.
Model answer
  1. The scoreboard:

    StageFailuresOut of how many
    Retrieval850 questions (16%)
    Support642, the questions where retrieval found the right passage (about 14%)
    After a miss (answered anyway)58, the questions where retrieval missed
    After a miss (unsupported claims, separate count)38, the questions where retrieval missed
    No answer26 no-answer tests
    End to end1650 questions, graded against the rubric (34 of 50 passed, 68%)
    Approval010 approval tests
    Stop15 stop tests

    The support denominator is 50 − 8 = 42, not 50. It is a conditional rate: when the model had the right passage, did it stick to it? Counting only those 42 keeps a search failure from being charged to the writing step. It does not make the writing step blameless after a miss: 3 of the 8 answers did the safe thing and said the answer was not found, and 5 answered anyway. That is the "after a miss" row. Answering anyway is not the same as making something up: checked against their retrieved passages, 3 of the 5 made unsupported claims, and 2 repeated true details about a different part. Those 2 are not made up, but they still fail, because they do not answer the question. Report the 3 as a separate count, out of the same 8 misses. It is not a no-answer test, because here the answer was in the collection and the search failed to bring it back. One question can fail at more than one stage; the rows are not meant to add up.

  1. Subtraction assumes every answer that was not a miss and not unsupported was right and usable. The grading shows otherwise: the person failed the 8 misses and the 6 unsupported answers, which leaves 36, but passed only 34. So 2 answers that stuck to their passages still failed the rubric (for example, by leaving out the condition that was asked about, or quoting a true sentence that does not answer the question). So the end-to-end figure is graded, not subtracted: 34 of 50. RAGAs scores answer relevance separately from faithfulness for this reason. On its own, though, 34 of 50 adds the stages together and hides which one broke. It cannot tell the team whether to fix the search or the writing.
  1. The failed stop test (1 of 5) blocks launch on its own: in this course one ignored stop is a defect to fix before launch, not a percentage to weigh against good answers. It is reported apart and never folded into the answer pass rate. The 8 retrieval misses point at the collection or the search. The 5 answers after a miss (3 of them unsupported), and the 6 unsupported answers, point at the instructions, the model or how passages are passed to it: it should say when it has not found the answer. A good answer also notes that scoring retrieval and faithfulness separately is how RAGAs evaluates retrieval systems, and that NIST asks for test sets and metrics to be documented.

Exercise 2 (300.5-b)

The parts team wants a model grader to score future test runs. Before anyone grades anything, they write the rule: at most 1 false accept for every 10 answers the person fails, and at least 10 human fails in the sample.

They take 40 answers from their test runs, including hard ones with part numbers and quantities. A person who did not build the system grades each answer against the rubric without seeing the grader's score. The person passes 30 and fails 10. The grader passes 1 of the 10 answers the person failed, and fails 4 of the 30 the person passed.

  1. Give the agreement and the false-accept rate, each with its denominator.
  2. Decide, against the rule, whether the grader may gate a release, and say what happens if it may not.
  3. Name two things about the grading itself that a reviewer should be able to check, and one reason the team should not rely on published agreement figures for model judges instead of this check.
Model answer
  1. Agreement: 40 − 1 − 4 = 35 of 40 (87.5%). False accepts: 1 of 10 human fails (10%). False rejects: 4 of 30 human passes, reported separately.
  1. The rule allows at most 1 false accept in 10 human fails, and the sample has the 10 human fails the rule needs. The grader made 1, so under the team's preset rule it may gate the release. Had it made 2 or more, a person would grade the release run, or the team would fix the grader and calibrate it again on a fresh sample. Had the sample held fewer than 10 human fails, the answer would be "cannot decide yet: grade more", not "pass". With only 10 human fails, one answer either way moves the false-accept rate by 10 points, so the result is rough.
  1. A reviewer should see who graded each answer and when, that the person was not a front-line developer of the system (NIST AI RMF, MEASURE 1.3), and that they graded without seeing the grader's score. A person clicking "pass" in a second or two is not a human grade. Published figures, such as Zheng and colleagues' over 80% agreement for strong judges, were measured on their own tasks; they do not show how this grader, with this rubric, grades these answers. The same paper reports limited capability in grading maths and reasoning, which matters for answers full of part numbers and quantities.

Exercise 3 (300.5-c)

A slide at Big Widget's monthly review says: "Reorder agent: 88% success." Behind it is a synthetic log of 250 live reorder requests. For each one the log records the request, the date and whether the ticket was marked closed. 220 were marked closed. The log does not record which passages were retrieved, whether the reorder was correct, or whether the reorders that needed a named person's approval got it.

  1. Write the four questions to ask before this number is reported, and answer each from the log where you can.
  2. Say what you can count from this log, and what you cannot count yet.
  3. Write the honest one- or two-line report, including what logging would make the missing counts possible.
Model answer
  1. The four questions:
    • What is counted? Tickets marked closed, not reorders checked as correct.
    • Out of how many? 250 live requests; a good answer asks whether any requests were left out.
    • Graded by whom? No one: "closed" is a status in the log, not a grade by a person or a calibrated grader.
    • When, and on which version? The log gives dates but not the model or collection version, so that has to be found or stated as unknown.
  1. Countable: tickets marked closed, 220 of 250 (88%). Not countable yet: how many reorders were correct, how many were retrieval misses, and how many approvals were skipped in live use, because the log records no grade, no retrieved passages and no approvals. For each of these the honest answer is "cannot count yet". Do not estimate them from the test set and report them as live figures.
  1. For example: "Live, [dates]: 220 of 250 reorder tickets marked closed. Not counted: correct reorders, retrieval misses and skipped approvals; the log does not record grades, passages or approvals. Cannot count yet. To count them, log the retrieved passages and each approval decision, and have a person or a calibrated grader grade a sample." A good answer cites NIST's call to document what will not or cannot be measured, and its warning against extrapolating performance from narrow assessments.

Next module: 300.6 Securing the basics: injection, leaks and too much agency