Try each exercise first, then open its model answer.
Engineering AI in your role (role tracks) · Workbook
Reviewed and published, version 1.0.0. Big Widget Company, its people, models, thresholds and records are made up. The model answers are examples of what a good answer contains, not the only right answers. Two 400s or a host mark your work at 400.P. An AI does not.
What not to write here. Do not paste real model outputs, production data, customer or employee details, keys or other secret information into these exercises. Use a made-up model at Big Widget, or describe your real one in general terms. Keep real records in a system your workplace controls.
Exercise 1 (400.R-a): three questions for one model
Take the track for your own role from the lesson's table, or write a new row for your role at Big Widget.
- Pick one model in that role. Write its intended use in one sentence, including what it must never do on its own. If your track owns a component (a pipeline, a serving platform or an evaluation harness), name the model it supports, the tests you own for the component, and the team that owns that model's evaluation.
- Does it work? (evaluation) Name:
- the held-out data, how it was built and from which period, and why it looks like real use;
- the baseline it must beat, scored on the same data;
- the metric tied to the harm, and which direction is better (higher or lower);
- the smallest improvement over the baseline worth acting on;
- the slices you report on their own, with intervals, and each slice's floor;
- the decision threshold, with the reason for it;
- what you do if an interval is too wide to tell.
- Does it still work? (monitoring) For at least two monitors, name the signal, the threshold set before launch, who is told, and what happens next (hold or roll back). Say what is held while that person checks, and how you would know the fallback is safe under current conditions. Say when the rollback is practised. If the true answers (labels) arrive late, say how late and what you watch meanwhile.
- Could a stranger check it? (evidence) List the records and their versions, who records the release decision (a named person: not you, not anyone who built the model, and never an AI tool), what stays out of the bundle, and how a stranger would verify the bundle without trusting your team.
- Whose job is it? In two or three sentences, say which of the choices above only you could write down, and why they cannot be written down later by someone else.
Model answer
An example for the service and warranty track (made up):
- Model and intended use: a model that routes warranty claims that may be safety-related to engineers. It flags claims for an engineer to review; it never closes or rejects a claim on its own.
- Does it work?
- Held-out data: claims from a later quarter than any used in building, collected the way the claims system collects them. The procedure that built the set is written down, because even a test set rebuilt the same way can give different scores, so one test set is not the truth about the task.
- Baseline: today's keyword rule, scored on the same claims.
- Metric and direction: the share of real safety claims the model routes, because a missed safety claim is the harm. Higher is better.
- Pass rule, written before the run: the lower end of the interval for "model minus keyword rule" sits above the smallest improvement worth acting on. If the interval is too wide to tell, the answer is "insufficient evidence": hold and re-check, not pass.
- Slices: results by product line, with intervals, so a good average cannot hide a product line where the model misses safety claims. The lower end of each product line's interval must be at or above that line's floor.
- Decision threshold: the score at which the model routes a claim, and the reason for it, written down before the run. It is not the same thing as the slice floors.
- Does it still work?
- Missed claims: each week a person labels a sample of claims the model did not route. If the miss rate on that sample crosses the threshold set before launch, the named owner (Sam, the service engineering lead, made up) is told at once and the model is held: engineers review every incoming claim by hand. The team rolls back to the last good version only once Sam has checked that it works on today's claims.
- Inputs: an alert when incoming claims stop matching what the model was tested on, for example a new product line or a change in the claim form, with the data engineering owner notified when an upstream source changes.
- Rollback: practised on a normal day, and the drill is recorded. A rollback can contain harm that a new release caused, if the old version is shown to be safe on today's claims. It does not undo a change in the world. A new claim form might make the old version a little worse but still within its floors, or it might hide safety claims from both versions, and then switching back recovers nothing. Knowing the cause does not tell Sam which; a check on today's claims does. Containment is not recovery either. The team finds the cause, re-runs the evaluation and goes back through the release decision. Until then the status is "held", not "fixed".
- Could a stranger check it?
- A model card for this version: intended use, evaluation data, metric and its direction, threshold, results by product line with intervals, caveats.
- Versions of the model, the test set and the configuration; the monitor thresholds and owners; the rollback drill record.
- The release decision (allow, constrain, hold or deny), recorded by a named validator who did not build the model. An AI tool may help draft these records; it never records the decision.
- All of it in a signed bundle a stranger verifies offline against a key and a checkpoint received separately (400.8).
- What stays out: the claims themselves and any customer details. They stay in Big Widget's controlled work record; the bundle carries their hash.
- Whose job: I picked the test claims, set the routing threshold and chose the monitors. If I do not record those choices and the reasons when I make them, nobody can reconstruct them after launch. Governance runs through the work; it is not a write-up at the end. My job is to produce the evidence; the validator's job is to check it.
A good answer:
- uses held-out data like real use, a baseline on the same data, a metric tied to the harm with its direction stated, and slices with intervals and floors;
- writes the pass rule for that direction (for an error, lower is better), and holds when an interval is too wide to decide;
- on a component track, names the supported model and keeps component tests apart from model-level evidence;
- gives every monitor a signal, a threshold fixed before launch, a named person and a next step, and says the rollback is practised, not only planned;
- holds the affected action first, and does not assume switching back stops every harm: it checks the fallback works under today's conditions, and calls the status "held", not "fixed", until the cause is found and the evaluation re-run;
- does not treat the model card as proof on its own: its accuracy relies on whoever wrote it, so it goes into a versioned, signed bundle that someone outside the team can check;
- names a person who did not build the model to record the release decision, and never an AI tool;
- treats a confident team, or a confident model, as no substitute for records.
Exercise 2 (400.R-b): your rail, and what a certificate does not show
- Your rail. Name the cert-prep rail your role draws on (ML engineering, data, AI security or AI audit). Use the lesson's table. If your role touches more than one, pick the one you will use in 400.P and say why.
- Two objectives. Name two of the course objectives that rail draws on, and say in one line each how you will use it in 400.P.
- What it does not show. Finish this sentence for the model from Exercise 1: "A certification held by anyone on my team would not show that my model ..."
- Person or system. For each item below, say whether it is evidence about a person or about the system, and why:
- (a) an engineer's exam pass certificate;
- (b) a record of the rollback drill for version 3.2.0;
- (c) a model card for version 3.2.0 with results by slice;
- (d) a manager's statement that the team is highly qualified.
Model answer
For the service and warranty engineer in Exercise 1 (made up):
- Rail: data. The course usually maps the service and warranty track to the data rail, which covers pipelines, data quality and evaluation. No vendor or exam is named: each reads "not cleared" until Bob clears it.
- Two objectives: 400.6-a, to design the router's evaluation on later claims against the keyword rule, with a metric tied to missed safety claims and its direction written down; and 400.7-a, to measure its error rates by product line and show that the overall average does not hide a product line where it fails. (Any two from the data rail's list in the lesson are fine if the use is clear.)
- What it does not show: "A certification held by anyone on my team would not show that my model routes enough real safety claims, still does so after launch, or was released by someone who checked its records."
- Person or system:
- (a) Person. It shows the engineer met the exam's bar on the date they sat it. It contains no measurement of any model.
- (b) System. It records that rollback of this version was practised.
- (c) System, but self-reported. It is only as reliable as the people who wrote it, so it counts as evidence once it sits in a versioned, signed bundle that someone outside the team can check.
- (d) Neither is shown. It is a claim about people, and confidence is not evidence.
A good answer:
- names a rail from the course's map and objectives that rail actually draws on;
- states plainly that a certification is about a person, on a date, and contains no measurement of the model;
- does not say a certificate replaces the validator, the tests or the bundle;
- notes that a system-level score such as the ML Test Score gives no points for who ran a test: half a point for a test run by hand with its results documented and shared, and a full point for a test a system runs automatically and repeatedly;
- treats a certificate as a reason to trust that someone can produce good evidence, never as the evidence.