Level 400: Engineering AI systems · Module 400.R

Five coworkers from different jobs compare notebooks with different coloured covers.
Does it still work?

Engineering AI in your role (role tracks) · Standard gear

As of 19 November 2026. Reviewed and published, version 1.0.0. Big Widget Company, its people, its models and its numbers are made up for teaching. Every real framework, paper and case below is real and cited. The role tracks, the three questions, the track rows, the release rule and the rail mapping are this course's own practice, not a law or a standard. Where we do not know something, we say so, and we say who checks next.

What this module does

Every role track asks of one model: does it work (evaluation, Get to Yes), does it still work (monitoring, Stay at Yes), and could a stranger check it (evidence, Recover to Yes).
The three questions are the course's track structure. Monitoring of functionality and behaviour follows the AI RMF (s1); short records that travel with a model follow model cards (s2).
Figure data as text

Three questions for any role

StageWhat it shows
GET TO YES (COURSE)Does it work? Held-out data like real use, a baseline, uncertainty, by slice.
STAY AT YES [s1]Does it still work? Which signal, which threshold, who is told, and hold or roll back.
RECOVER TO YES [s2]Could a stranger check it? Which records, which versions, decided by whom, checkable without trust.

Modules 400.1 to 400.9 taught how models are trained, adapted, fed with data, operated, secured, evaluated, tested for fairness, recorded and audited. This module puts that work into your own job at Big Widget Company, the made-up 100,000-person maker of complex, expensive, car-like widgets used across this course.

You take the track for your role. Big Widget has eleven engineering tracks, listed below. Then you answer two things:

Two beliefs this module corrects:

Three questions for one model

Every track asks the same three questions about one model the learner builds, adapts, runs or supports. They sit on the three yeses from Level 100.

QuestionWhat it asksWhere in the three yesesWhere the course taught it
Does it work? (evaluation)On held-out data that looks like real use, against a baseline, with uncertainty, broken down by the slices that matterGet to Yes: before the release decision400.6, 400.7
Does it still work? (monitoring)Which signal, which threshold, who is told, and what happens next (hold, roll back)Stay at Yes: while it runs400.4
Could a stranger check it? (evidence)Which records, with which versions, decided by whom, verifiable without trusting the teamRecover to Yes: after, on demand or on incident400.8, 400.9

The release decision itself is one of four answers: allow (go ahead), constrain (only this far), hold (wait for a person) or deny (no). A hold is not a slow yes. A named person releases it, or it stays held. An AI tool may help draft the evidence, but it never records the release decision. A person does.

These questions are not new ideas invented for this course. They line up with outcomes that a published framework and two papers already name.

Does it work? The NIST AI Risk Management Framework (AI RMF 1.0, January 2023) lists, under its MEASURE function, that "Test sets, metrics, and details about the tools used during TEVV are documented" (TEVV is test, evaluation, verification and validation).1 It also asks that the system to be deployed is "demonstrated to be valid and reliable", and that "Limitations of the generalizability beyond the conditions under which the technology was developed are documented".1 Breck and colleagues' ML Test Score, a rubric of 28 actionable tests for production machine-learning systems, includes "A simpler model is not better" (regularly test against a very simple baseline model) and "Model quality is sufficient on all important data slices".2 Slicing, they write, "avoids having fine-grained quality issues masked by a global summary metric".2

Does it still work? The AI RMF asks that "The functionality and behavior of the AI system and its components ... are monitored when in production".1 Its MANAGE function asks that post-deployment monitoring plans are implemented, including mechanisms for "appeal and override, decommissioning, incident response, recovery, and change management".1 The ML Test Score explains why a launch score is not enough: "Validation data will always be older than real serving input data", so quality measured before serving "is only an estimate".2 Its monitoring tests include checking that data invariants hold for inputs, that models are not too stale, and that prediction quality has not regressed; for quality it says thresholds must be set and "a responder should be notified immediately if quality drifts outside that threshold".2 Its infrastructure tests include that serving models can be rolled back, and it says operators should practise rolling back "normally, when not in emergency conditions".2

Could a stranger check it? Mitchell and colleagues proposed model cards: "short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions".3 A model card also discloses "the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information".3 Its proposed sections are Model Details, Intended Use, Factors, Metrics, Evaluation Data, Training Data, Quantitative Analyses, Ethical Considerations, and Caveats and Recommendations.3 The authors say quantitative analyses "should be disaggregated, that is, broken down by the chosen factors", with confidence intervals when possible.3

But the same authors also say a card is not proof on its own: "the usefulness and accuracy of a model card relies on the integrity of the creator(s) of the card itself".3 They call model cards "one transparency tool among many", alongside, for example, algorithmic auditing by third parties.3 That is why the third question asks what a stranger would need, not what the team would show.

Signing the card does not close that gap. A signature shows who made a record and that nobody changed it afterwards. It does not show that the numbers in it are right, or that the monitors and the release gate actually ran. So a stranger needs the underlying records and a method they can repeat, not just a signed summary. The signed, versioned, tamper-evident bundle a stranger verifies offline, against a key and a checkpoint received separately, is what 400.8 and 400.9 taught.

What the bundle holds, and what it never holds. The shareable record carries hashes, versions, the decision and who made it, and the policy it was made under. It never carries prompts, model outputs, payloads or personal data: no customer claims, no employee details, no raw test images. Those stay in a separately controlled work record at Big Widget that the bundle points to by hash. A hash is a pointer, not permission to share what it points to.

Why evidence is the engineer's job

The AI RMF treats governance as a function that runs through all the others, not as a separate team's step at the end. It says governance "is designed to be a cross-cutting function to inform and be infused throughout the other three functions".1 One of its GOVERN outcomes is that "Roles and responsibilities and lines of communication related to mapping, measuring, and managing AI risks are documented and are clear to individuals and teams throughout the organization".1

It also separates building from checking. Its figure of the people involved across the AI lifecycle notes that those "building and using the models" are separated, as a best practice, from those "verifying and validating the models".1 And it asks that "Internal experts who did not serve as front-line developers for the system and/or independent assessors are involved in regular assessments and updates".1

Put those together and the engineer has two duties, not zero:

  1. Produce the evidence while doing the work: the test set and its version, the baseline, the slices, the metric and which direction is better, the threshold and why it was chosen, the monitor and who it alerts, the rollback that was practised.
  2. Do not be the one who decides the release. The RMF asks for the separation. This course goes one step further and has an independent validator record the release decision (400.9). The engineer's job is to make that check possible. Not being on the build team is the minimum here; 400.9 also keeps the validator separate from the person who holds and operates the system.

What happens when nobody can check. In Bates v Post Office Ltd (No 6), judgment of 16 December 2019, Mr Justice Fraser found that the Post Office's Legacy Horizon accounting system "was not robust", and that although Horizon Online in its HNG-X form was better, its "robustness was questionable, and did not justify the confidence placed in it by the Post Office in terms of its accuracy".4 In the same paragraph he said the later HNG-A "is a different matter", which the experts agreed is "far more robust than Horizon in earlier times".4 Horizon was an accounting system, not a machine-learning model, and the finding is about named versions. It is here for one lesson that 400.8 and 400.9 build on: confidence in a system's output is not evidence that the output is right.

A worked example: the paint-line inspection model

Made up for teaching. Big Widget's paint line has cameras. A vision model flags panels with paint defects for a person to inspect. The team wants to replace an older rule-based check with it.

Does it work? (evaluation, Get to Yes)

Say which way is better, then write the pass rule. Some metrics improve as they go up: the share of defects caught, or the share of safety claims routed. Others improve as they go down: forecast error, simulation error. The course's pass rule, written before the run, has three parts:

  1. A worthwhile improvement. The whole interval for the difference from the baseline sits on the better side by at least the smallest improvement worth acting on. For a higher-is-better metric, that means the lower end of "model minus baseline" is above it. For an error, it means the upper end of "model error minus baseline error" is below minus that amount.
  2. No bad slice. Every slice clears its own floor. For a higher-is-better metric, the lower end of each slice's interval is at or above the floor. For an error, the upper end of each slice's error interval is at or below the most error that slice can tolerate.
  3. Undecided means hold. If an interval is too wide to tell, the answer is "insufficient evidence": hold and re-check, not pass.

Made-up numbers to show the direction matters. A recall improvement with a difference interval of [0.08, 0.12] and a smallest worthwhile improvement of 0.02 passes part 1. A forecast whose average error drops from 10 to 7, with an error-difference interval of [−4, −2] and a worthwhile improvement of 1, also passes: the whole interval is below −1. An error that rises from 10 to 12, interval [1, 3], fails. Apply the higher-is-better rule to the raw errors, with a smallest improvement of zero, and it goes backwards: the real improvement (lower end −4) fails, and the regression (lower end 1) passes. An interval of [−3, 1] is too wide to decide: hold.

The floors for slices are not the decision threshold. The decision threshold is the score at which the model flags a panel. The floors are the least acceptable result for each slice. Write both down, with reasons, before the run.

Does it still work? (monitoring, Stay at Yes)

Recover to Yes. A rollback can contain harm that a new release caused, if the fallback is compatible and safe under current conditions. It does not undo a change in the world. A change in live inputs may degrade the old version a little, leaving it within its bounds, or it may make both versions unsafe. Suppose the camera software changes and the new images hide defects from both the new model and the old one: switching back recovers nothing. The cause alone does not tell you which case you are in; a check of the fallback on current images does. So the first step is to hold the affected action, not to assume the old version is safe. On the paint line that means people inspect every panel by hand until the model's named owner has checked that the fallback works on today's images. Containment is not recovery either. The team finds the cause, re-runs the evaluation on the repaired version, and goes back through the release decision before claiming it is at yes again. Until then, the honest status is "held", not "fixed".

Could a stranger check it? (evidence)

The eleven tracks

Each row is one track. The example models and the answers in the row are made-up Big Widget examples, written as course guidance. Where a real case sits behind a row, the row cites it; otherwise it names the module that teaches it. The last column is the cert-prep rail the track most often draws on (next section). Many roles touch more than one rail.

TrackExample model (made up)Does it work?Does it still work?What a stranger needsCase or moduleRail
ML engineeringA language model fine-tuned on service manualsTask score against the base model, and the safety evaluation re-run after every fine-tuneRefusal and error rates on a fixed probe set, per releaseBase model, data and fine-tune versions; both evaluation runsFine-tuning can degrade safety6; 400.2, 400.LML engineering
Data scienceA warranty-claim triage modelHeld-out claims from a later period, against the current rule, with intervals; higher recall is betterLive quality from a labelled sample; drift in claim mixModel card with sliced results3400.6, 400.7Data
Data engineeringThe warranty-claim triage model, through the pipeline that feeds itRole-owned: schema and invariant tests on every load. Model-level: the data science team's evaluation, re-run on each new dataset versionInvariants on live inputs; alerts when upstream sources change2Dataset versions and lineage, linked to the model version each one trained400.3Data
MLOps and platformThe paint-line inspection model, through the serving platformRole-owned: canary and validation before serving.2 Model-level: the inspection team's evaluation, which the gate checks is present and passingModel age, latency and quality alerts with named owners2A practised rollback record restoring model, data and configuration together400.4ML engineering
Security engineeringThe service-manual assistantTests for extraction of training data and for injectionLogs of unusual query patternsAttack test results, key custody and signed logsTraining-data extraction7; 400.5AI security
Evaluation and qualityThe warranty-claim triage model, through Big Widget's evaluation harnessModel-level: fresh test sets built the documented way; baselines; intervals. Role-owned: the harness re-scores a fixed, known case and gets the recorded answerRe-evaluation on a schedule and after every changeTest sets, metrics and tool details documented1New test sets lowered scores5; 400.6Data
Manufacturing and quality inspectionThe paint-line vision model (above)Held-out line images, sliced by colour, camera and shiftLabelled sample of passed panels; input checksModel card, versions, signed releaseNew test sets lowered scores5; 400.6ML engineering
Maintenance and reliabilityA model predicting bearing failure from vibration dataBack-test on held-out machines, against the fixed service scheduleMissed failures counted from the work-order record, once failures are known; the label delay is statedWhich machines, sensors and periods it was tested on400.4, 400.6ML engineering
Forecasting and supply chainA parts-demand forecastError against a naive forecast on later periods, with ranges; lower error is betterError tracked each period once actual demand is in; alert when it leaves its bandForecast versions and the data each one used400.6Data
Service and warrantyA model routing safety-related claims to engineersShare of real safety claims it routes, by product lineLabelled sample of claims it did not routeSliced results; who reviews unrouted claims400.6, 400.7Data
Engineering and simulationA surrogate model that speeds up a stress simulationError against the full simulation on held-out designs, inside the tested range; lower error is betterSpot checks against the full simulation; alert outside the tested rangeThe tested range and its limits, written down1400.6ML engineering

Component tracks. Three tracks own a component rather than a model: the pipeline, the serving platform and the evaluation harness. They still answer 400.R-a for one model, the one their component supports. They test their own component, and they name the model-level evidence and the team that supplies it. Component evidence does not stand in for model evidence. A schema check cannot show the triage model's recall. A canary cannot show the inspection model is accurate. A harness cannot validate its own measurements just by existing.

Two course rules run through every row. First, the team that built the model does not decide its own release. Second, any threshold, slice, metric direction or rollback that is not written down before launch does not count as evidence after it.

The cases behind the rows

ML engineering: re-check after every fine-tune. Qi and colleagues (ICLR 2024) report that, in their tests, fine-tuning GPT-3.5 Turbo on only 10 adversarially designed examples jailbroke its safety guardrails, at a cost of less than USD 0.20 through OpenAI's APIs.6 They also report that fine-tuning on benign, commonly used datasets can degrade safety alignment, though to a lesser extent.6 This is a measured result on 2023 models and APIs, not proof that every model loses every guardrail. The lesson for the track: a fine-tuned model is a new model, and its safety evaluation is re-run, not inherited.

Security engineering: test what the model gives away. Carlini and colleagues (USENIX Security 2021) extracted hundreds of verbatim text sequences from GPT-2's training data by querying the model.7 The examples included public personal information such as names, phone numbers and email addresses, and some appeared in only one document in the training data.7 The lesson for the track: an extraction test belongs in the evaluation, and its result belongs in the evidence. The result goes in as a count and a pass or fail, not as the extracted text.

Evaluation and quality, and inspection: one test set is not the task. The Recht study above.5

Every track: confidence is not evidence. The Horizon judgment above.4

Cert-prep rails: what a certification does and does not prove

Level 400 maps four cert-prep rails to the course objectives they draw on. These are this course's map (levels/400/objectives.yml), not a vendor's syllabus. The course prepares people for vendor-proctored exams; it never sits them and never shares live exam content. No certification vendor or exam is named until Bob clears it, so every rail's vendor and exam read "not cleared" (as of 19 November 2026).

RailWhat it coversDraws onTracks that usually take it
ML engineeringA cloud machine-learning engineering certification: build, deploy and monitor models400.2-a, 400.2-c, 400.L-a, 400.4-a, 400.4-b, 400.6-aML engineering; MLOps and platform; inspection; maintenance; simulation
DataA data engineering or data science certification: pipelines, data quality, evaluation400.3-a, 400.3-b, 400.6-a, 400.6-c, 400.7-aData science; data engineering; evaluation and quality; forecasting; service and warranty
AI securityAn AI or cloud security certification: attacks on ML systems, keys, logs400.5-a, 400.5-b, 400.5-c, 400.8-bSecurity engineering
AI auditAn AI governance or AI management-system audit certification400.8-a, 400.8-c, 400.9-a, 400.9-b, 400.9-cAnyone who will validate or audit other teams' models

What a passed exam shows, and what it does not. This is course guidance, from what an exam and an evidence bundle are each about.

A passed certification examAn evidence bundle
It is aboutA personOne system, at named versions
It showsThat person met the exam's bar, on the date they sat itWhat was tested, monitored and decided for that system, and by whom
It does not showWhether any model that person built works, still works or is safeWhether the people involved hold any credential
Who can check itThe certifying body's recordsA stranger with the bundle, and a key and checkpoint received separately (400.8)

The ML Test Score makes the same split from the other side. Its score is meant to measure "how ready for production a given machine learning system is".2 Each test earns half a point if it is run manually "with the results documented and distributed", and a full point if a system runs it automatically on a repeated basis; the final score is the minimum of the four section scores.2 The score belongs to the system and its records. A test that is never run earns nothing, however certified the engineer. A test run once by hand with no written, shared results earns nothing either. This course also asks for automated test results to be in the bundle; that is the course's evidence rule, not part of the paper's score.

So a certification is a reason to trust that someone can produce good evidence. It is never a substitute for the evidence. In 400.P, the stranger verifies the bundle, not the team's credentials.

Your track exercise

Bring this to the project (400.P).

  1. Pick your track from the table, or write a new row for your role at Big Widget.
  2. Pick one model in that role. Write its intended use in one sentence, including what it must never do on its own. If your track owns a component, name the model it supports and the team that owns that model's evaluation.
  3. Does it work? Name the held-out data and how it was built, the baseline, the metric tied to the harm and which direction is better, the slices and their floors, the decision threshold and why, and the smallest improvement worth acting on.
  4. Does it still work? Name the signal, the threshold, who is told, and what happens next. Say what is held while that person checks, and how you would know the fallback is safe under current conditions. If labels arrive late, say how late and what you watch meanwhile.
  5. Could a stranger check it? List the records, their versions, who decides the release (not you), what stays out of the bundle, and how a stranger would verify it without trusting your team.
  6. Your rail. Name the cert-prep rail your role draws on and two of its objectives you will use in 400.P, with one sentence each on how it applies to your model. Then write one sentence on what that certification would not show about your model.

Where you cannot answer yet, write "not known yet" and name who will find out. That is an honest answer; a guess is not.

What this module does not claim

Words to know

Further learning (not cited, never copied)

Free university courses listed in levels/400/universities.md for this module: MIT 6.S191 Introduction to Deep Learning; Carnegie Mellon 10-414/714 Deep Learning Systems; and MIT 6.5940 TinyML and Efficient AI Computing (Fall 2026, partial public materials). They are further learning only; this module does not rely on them for any claim.

Sources

  1. s1 Elham Tabassi (National Institute of Standards and Technology) (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. Standard. Checked 2026-11-19 by source-checker agent (Claude, fresh session separate from the writer, reviser, item writer and earlier checkers), 400.R round 3 re-check, 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  2. s3 Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. Paper. Checked 2026-11-20 by source-checker agent (Claude, fresh session separate from the writer, reviser, item writer and earlier checkers), 400.R round 4 re-check, 20 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s2 Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, et al. (2018). Model Cards for Model Reporting. Paper. Checked 2026-11-19 by source-checker agent (Claude, fresh session separate from the writer, reviser, item writer and earlier checkers), 400.R round 3 re-check, 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  4. s7 Mr Justice Fraser, High Court of Justice, Queen's Bench Division (2019). Bates v Post Office Ltd (No 6) "Horizon Issues" [2019] EWHC 3408 (QB). Record. Checked 2026-11-19 by source-checker agent (Claude, fresh session separate from the writer, reviser, item writer and earlier checkers), 400.R round 3 re-check, 19 Nov 2026. ↩ ↩ ↩ ↩ ↩
  5. s4 Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar (2019). Do ImageNet Classifiers Generalize to ImageNet?. Paper. Checked 2026-11-19 by source-checker agent (Claude, fresh session separate from the writer, reviser, item writer and earlier checkers), 400.R round 3 re-check, 19 Nov 2026. ↩ ↩ ↩ ↩ ↩
  6. s5 Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, et al. (2023). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!. Paper. Checked 2026-11-19 by source-checker agent (Claude, fresh session separate from the writer, reviser, item writer and earlier checkers), 400.R round 3 re-check, 19 Nov 2026. ↩ ↩ ↩ ↩
  7. s6 Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, et al. (2021). Extracting Training Data from Large Language Models. Paper. Checked 2026-11-19 by source-checker agent (Claude, fresh session separate from the writer, reviser, item writer and earlier checkers), 400.R round 3 re-check, 19 Nov 2026. ↩ ↩ ↩

Next: the quiz