Level 400: Engineering AI systems · Module 400.R

Five coworkers from different jobs compare notebooks with different coloured covers.
Does it still work?

Engineering AI in your role (role tracks) · Technical gear

As of 19 November 2026. Reviewed and published, version 1.0.0. Big Widget Company, its models, thresholds, intervals and records are made up for teaching. The notation for the three questions, the pass and release predicates and the rail mapping are this course's own practice, not a standard. Every external claim is cited to a primary source. Where something is not known or not checked, we say so.

1. Scope and claims

Every role track asks of one model: does it work (evaluation, Get to Yes), does it still work (monitoring, Stay at Yes), and could a stranger check it (evidence, Recover to Yes).
The three questions are the course's track structure. Monitoring of functionality and behaviour follows the AI RMF (s1); short records that travel with a model follow model cards (s2).
Figure data as text

Three questions for any role

StageWhat it shows
GET TO YES (COURSE)Does it work? Held-out data like real use, a baseline, uncertainty, by slice.
STAY AT YES [s1]Does it still work? Which signal, which threshold, who is told, and hold or roll back.
RECOVER TO YES [s2]Could a stranger check it? Which records, which versions, decided by whom, checkable without trust.

For a model $m$ at version $v$ in role $r$, this module asks the learner to specify three artefacts and one mapping:

The misconceptions addressed are that $B$ is someone else's job (400.R-a), and that a credential held by the engineer is evidence about $m$ (400.R-b).

The artefacts sit on the course's three yeses. $E$ and the release decision are Get to Yes: before the model acts. $M$ is Stay at Yes: while it runs. Recover to Yes is containment (hold, a rollback where it is safe, or a manual fallback), diagnosis, re-evaluation and a fresh release decision, with a $B$ a stranger can verify on demand: after an incident, or whenever someone asks. The release decision takes one of four values, allow, constrain, hold or deny; a hold is released only by a named person and is not a deferred allow. An AI tool may draft parts of $B$; it never records the release decision.

If $r$ owns a component (a pipeline, a serving platform, an evaluation harness) rather than a model, $m$ is the model that component supports; section 6.1 gives the contract.

2. Alignment with the framework and the literature

The three questions map onto outcomes in the NIST AI RMF 1.0 (NIST AI 100-1, January 2023), whose Core has four functions, GOVERN, MAP, MEASURE and MANAGE, with GOVERN "designed to be a cross-cutting function to inform and be infused throughout the other three functions".1

ArtefactAI RMF outcome (verbatim)Engineering rubric
$E$MEASURE 2.1: "Test sets, metrics, and details about the tools used during TEVV are documented."1 MEASURE 2.5: "The AI system to be deployed is demonstrated to be valid and reliable. Limitations of the generalizability beyond the conditions under which the technology was developed are documented."1 MEASURE 2.11: "Fairness and bias ... are evaluated and results are documented."1ML Test Score Model 5, "A simpler model is not better"; Model 6, "Model quality is sufficient on all important data slices"2
$M$MEASURE 2.4: "The functionality and behavior of the AI system and its components ... are monitored when in production."1 MANAGE 4.1: post-deployment monitoring plans are implemented, including mechanisms for "appeal and override, decommissioning, incident response, recovery, and change management".1Monitoring tests, including "Data invariants hold for inputs", "Models are not too stale" and "Prediction quality has not regressed"; Infra 7, "Serving models can be rolled back"2
$B$ and the releaseGOVERN 2.1: roles and responsibilities "are documented and are clear to individuals and teams throughout the organization".1 MEASURE 1.3: "Internal experts who did not serve as front-line developers for the system and/or independent assessors are involved in regular assessments and updates."1Model cards: Model Details, Intended Use, Factors, Metrics, Evaluation Data, Training Data, Quantitative Analyses, Ethical Considerations, Caveats and Recommendations3

The RMF's figure of lifecycle actors notes that actors in the AI Model dimension are separated "as a best practice, with those building and using the models separated from those verifying and validating the models".1 The framework is "intended to be voluntary".1 It is used here as a vocabulary, not as a compliance claim. It asks for separation and independent assessment; it does not require a signed release decision. The signed decision is this course's practice (400.9).

Two qualifications from the sources themselves shape the design of $B$:

  1. A launch measurement is an estimate. "Validation data will always be older than real serving input data, so measuring a model's quality on that validation data before pushing it to serving is only an estimate of quality metrics on actual live serving inputs."2 Hence $M$ is not optional.
  2. Self-reported documentation is not self-verifying. Mitchell and colleagues: "the usefulness and accuracy of a model card relies on the integrity of the creator(s) of the card itself", and model cards are "one transparency tool among many", alongside, for example, "algorithmic auditing by third-parties".3 Hence $B$ is checked by someone outside the team (400.8, 400.9).

A signature over the card does not remove the second qualification. It binds the card to a key holder and makes later changes evident; it says nothing about whether the reported numbers are true or whether the monitors and the gate actually ran. "Could a stranger check it?" therefore requires $B$ to carry, or point by hash to, the underlying records and a method the stranger can re-run, not only a signed summary.

3. Specifying $E$: evaluation

A course-standard specification of $E(m, v)$ has seven fields:

FieldContentBasis
Intended useThe decision the output feeds, and what $m$ must never do unsupervisedModel card, Intended Use3
Test distribution $\mathcal{D}_{test}$How the held-out set was built, from which period and sources, and its content hashMEASURE 2.11
Baseline $b$The incumbent process or a deliberately simple model, scored on the same $\mathcal{D}_{test}$ML Test Score Model 52
Metric $\mu$ and orientation $\sigma$A metric tied to the harm, and whether higher ($\sigma = +1$) or lower ($\sigma = -1$) is betterCourse practice
Decision threshold $\tau$The score at which $m$'s output triggers its action, and why it was chosenModel card, Metrics: "If decision thresholds are used, what are they, and why were those decision thresholds chosen?"3
Slices $S = \{s_1, \dots, s_k\}$ and floors $\varphi_s$Factors along which $\mu$ is reported separately, with intervals, and the least acceptable value of $\mu$ on each"Quantitative analyses should be disaggregated, that is, broken down by the chosen factors", "providing confidence interval values when possible"3; Model 62
Validity envelopeThe conditions under which $E$ was run, and what is out of rangeMEASURE 2.51

$\tau$ and $\varphi_s$ are different quantities. $\tau$ is a threshold on the model's score that turns it into a decision. $\varphi_s$ is a threshold on the evaluation metric that decides whether the model may be released.

The pass condition, with orientation made explicit. Define the oriented utility $u = \sigma \mu$, so larger $u$ is always better. Let $[L, U]$ be the chosen interval (400.6) for the raw difference $\mu(m) - \mu(b)$, and $[L_s, U_s]$ the interval for $\mu_s(m)$. Then the interval for the utility difference $\Delta = u(m) - u(b)$ is $[L, U]$ when $\sigma = +1$ and $[-U, -L]$ when $\sigma = -1$: negation swaps the endpoints. Write $\underline{\Delta}, \overline{\Delta}$ for its lower and upper ends, and likewise $\underline{u_s}, \overline{u_s}$ for the slice utility. The course uses a three-valued outcome:

E(m, v) = \begin{cases}
\text{pass} & \text{if } \underline{\Delta} > \delta \;\wedge\; \forall s \in S:\ \underline{u_s} \ge \sigma \varphi_s, \\
\text{fail} & \text{if } \overline{\Delta} \le \delta \;\vee\; \exists s \in S:\ \overline{u_s} < \sigma \varphi_s, \\
\text{insufficient evidence} & \text{otherwise,}
\end{cases}

where $\delta \ge 0$ (the smallest improvement worth acting on), $\sigma$ and every $\varphi_s$ are written down before the run. For $\sigma = +1$ the pass line reads: the lower bound of $\mu(m) - \mu(b)$ exceeds $\delta$, and every slice's lower bound is at least its floor. For $\sigma = -1$ (an error or loss) it reads: the upper bound of $\mu(m) - \mu(b)$ is below $-\delta$, and every slice's upper error bound is at most its maximum tolerable error $\varphi_s$. "Insufficient evidence" means hold and re-check, never pass. The three cases are mutually exclusive, because a lower bound never exceeds its upper bound.

Synthetic checks (made-up numbers, no real model).

Case$\sigma$Raw interval $[L, U]$$\delta$$[\underline{\Delta}, \overline{\Delta}]$Outcome
Recall improves$+1$$[0.08, 0.12]$0.02$[0.08, 0.12]$pass
MAE improves, 10 to 7$-1$$[-4, -2]$1$[2, 4]$pass
MAE regresses, 10 to 12$-1$$[1, 3]$1$[-3, -1]$fail
MAE unclear$-1$$[-3, 1]$1$[-1, 3]$insufficient evidence: hold

Applying the unoriented rule $L > \delta$ to raw error with $\delta = 0$ would fail the real improvement ($L = -4$) and pass the regression ($L = 1$). That reversal is the reason $\sigma$ is a required field. Slice outcomes are assumed to clear their floors in this table.

Why one test set is not the task. Recht, Roelofs, Schmidt and Shankar built new test sets for CIFAR-10 and ImageNet "by closely following the original dataset creation processes".4 A broad range of models lost 3 to 15 percentage points of accuracy on the new CIFAR-10 set and 11 to 14 on the new ImageNet set.4 The authors report that accuracy gains on the original test sets translated to larger gains on the new ones, and write that their results suggest the drops come from the models' inability to generalise to slightly "harder" images rather than from adaptivity.4 Engineering consequence: record the construction procedure of $\mathcal{D}_{test}$ in $B$, and state the validity envelope rather than a single number.

Why $E$ is re-run after adaptation. Qi and colleagues (ICLR 2024) report that, in their tests, fine-tuning GPT-3.5 Turbo on 10 adversarially designed examples jailbroke its safety guardrails, at a cost of less than USD 0.20 through OpenAI's APIs, and that fine-tuning on benign, commonly used datasets can also degrade safety alignment, to a lesser extent.5 For any fine-tuned $m'$ derived from $m$, $E(m', v')$ includes the safety evaluation again; it is not inherited from $E(m, v)$.

Why $E$ includes an extraction test for generative models. Carlini and colleagues (USENIX Security 2021) extracted hundreds of verbatim sequences from GPT-2's training data by querying it, including public names, phone numbers and email addresses, some present in only one training document.6 The test result enters $B$ as counts and an outcome, never as extracted text.

4. Specifying $M$: monitoring

$M(m, v)$ is a set of monitors. Each monitor is a tuple

(\text{signal } x_t,\; \text{window } w,\; \text{label delay } \lambda,\; \text{threshold } \theta,\; \text{owner},\; \text{action} \in \{\text{notify},\, \text{hold},\, \text{rollback}\}).

The ML Test Score's monitoring section supplies the usual signals:2

Two design limits on the quality monitor are the learner's to state, not the source's to solve.

The action column must be executable. The ML Test Score treats rollback of the learned model as an ML-specific incident response, and recommends that operators practise it "normally, when not in emergency conditions".2 At this level the rollback restores model, data and configuration versions together (400.4). A rollback to $v_{prev}$ can contain a regression introduced by $v$, provided $v_{prev}$ is compatible with, and safe under, current conditions. A change in live inputs or dependencies can also degrade $v_{prev}$; degraded is not the same as unsafe, and the two versions need not degrade equally. Rollback remains a valid containment option whenever $v_{prev}$, checked on current inputs, still meets its pre-set bounds. It fails when the change makes both versions unsafe. Synthetic example: a camera firmware change makes some defects invisible in the images, so both $v$ and $v_{prev}$ miss them; restoring $v_{prev}$ recovers no information. The cause category alone does not separate these cases; a check of $v_{prev}$ under current conditions does. The first action on a quality alarm is therefore hold: route the affected decisions to an independently safe fallback (on the paint line, manual inspection of every panel) until $\text{owner}(m)$ has checked $v_{prev}$ against current inputs. Containment is not evidence of recovery either. Return to allow requires a repaired $v'$ that passes $E$ and the release predicate again; until then the recorded status is hold, not fixed.

5. Specifying $B$: the bundle and the release predicate

A minimal manifest, as used in 400.8 and 400.P (field names are course practice; every value here is made up):

model: paint-defect-detector
version: 3.2.0
intended_use: flag panels for human inspection; never auto-reject
test_set: {built_by: procedure-v4, period: 2026-Q3 weeks 9-12, sha256: "<hash>"}  # images stay in the work record
baseline: rule-check-v7
metric: {name: defect recall, direction: higher_is_better, min_improvement: "<delta>"}
decision_threshold: model_card.md#metrics           # tau and its rationale
slices: {factors: [paint_colour, camera_id, shift], floors: slice_floors.yml}
evaluation_result: pass | fail | insufficient_evidence
monitors: monitors.yml        # signal, window, label delay, threshold, owner, action
rollback_drill: drills/2026-11-12.log
model_card: model_card.md
release: {decision: allow | constrain | hold | deny, approver: "<named validator, not on the build team>"}
signature: detached; verified offline against a key and checkpoint received separately
# never in the bundle: prompts, model outputs, payloads, personal data

The manifest is tamper-evident, not tamper-proof: the signature shows a change, it does not prevent one. Hashes point into Big Widget's controlled work record; a hash is not permission to disclose what it points to.

The release predicate the course uses:

\text{allow}(m, v) \implies E(m, v) = \text{pass} \;\wedge\; M \ne \varnothing \;\wedge\; \text{rollback drilled} \;\wedge\; \text{approver} \notin \text{builders}(m) .

It is a necessary condition, not a sufficient one: the named approver still decides, and may constrain. $E = \text{fail}$ makes the gate answer deny for this version; any other missing conjunct, including $E = \text{insufficient evidence}$, makes it hold. The last conjunct follows the RMF's separation of building from verifying and validating.1 It is the minimum. Being outside $\text{builders}(m)$ does not show independence from the people who operate $m$; 400.9's premise gate keeps a separate boundary between the Level 300 holder and the validator, and nothing here relaxes it.

Why the bundle has to stand on its own. In Bates v Post Office Ltd (No 6), judgment of 16 December 2019, Mr Justice Fraser held at para 936 that Legacy Horizon "was not robust", and that although Horizon Online in its HNG-X form was better, its "robustness was questionable, and did not justify the confidence placed in it by the Post Office in terms of its accuracy".7 The same paragraph says that HNG-A "is a different matter", which the experts agreed is "far more robust than Horizon in earlier times".7 The finding is version-specific. Horizon was an accounting system, not a learned model; it is used here for the general point that institutional confidence in a system's outputs is not evidence about them. 400.8 and 400.9 develop it.

6. The eleven tracks as specifications

Rows are course guidance with made-up Big Widget models. $\rho$ is the rail (section 7). $\sigma$ is noted where the metric is an error.

Track$m$ (made up)$E$: key element$M$: key monitor$B$: key recordGrounding$\rho$
ML engineeringLLM fine-tuned on service manualsTask metric vs. base model; safety suite re-run on $m'$Refusal/error rate on a fixed probe set per releaseBase, data and adapter versions; both evaluation runs5; 400.2, 400.LML eng.
Data scienceWarranty-claim triage classifierLater-period holdout vs. incumbent rule, sliced, with intervals; $\sigma = +1$ (recall)Labelled-sample recall; claim-mix driftModel card with disaggregated results3400.6, 400.7Data
Data engineeringTriage classifier, via its feature pipeline $c$$E_c$: schema and invariant tests per load. $E(m)$: data science's evaluation, re-run per dataset versionInput invariants; dependency notifications2Dataset versions and lineage, linked to each $m$ version trained400.3Data
MLOps and platformPaint-line vision model, via the serving platform $c$$E_c$: validation and canary before serving.2 $E(m)$: inspection team's evaluation, checked present and passing by the gateStaleness, latency, quality with owners2Rollback drill log (model, data, config)400.4ML eng.
Security engineeringService-manual assistantExtraction and injection testsQuery-pattern anomaliesAttack results; key custody; signed logs6; 400.5AI security
Evaluation and qualityTriage classifier, via the evaluation harness $c$$E(m)$: fresh test sets by documented procedure; baselines; intervals. $E_c$: harness reproduces a pinned reference resultScheduled and change-triggered re-evaluationTest sets, metrics and tool details14; 400.6Data
Manufacturing and quality inspectionPaint-line vision modelLine images sliced by colour, camera, shift; $\sigma = +1$ (recall)Labelled sample of passed panels; input checksModel card; versions; signed release4; 400.6ML eng.
Maintenance and reliabilityBearing-failure predictorBack-test on held-out machines vs. fixed scheduleMissed failures from work orders, $\lambda$ statedMachines, sensors and periods tested400.4, 400.6ML eng.
Forecasting and supply chainParts-demand forecastError vs. naive forecast on later periods, with ranges; $\sigma = -1$Per-period error band, after actuals arriveForecast versions and input data400.6Data
Service and warrantySafety-claim routerRecall of true safety claims, by product line; $\sigma = +1$Labelled sample of unrouted claims (escape prevalence)Sliced results; reviewer of unrouted claims400.6, 400.7Data
Engineering and simulationSurrogate for a stress simulationError vs. full simulation on held-out designs, inside range; $\sigma = -1$Spot checks; out-of-range alertValidity envelope1400.6ML eng.

6.1 Component tracks

For a role that owns a component $c$ supporting model $m$, the contract is:

$E_c$ does not imply $E(m, v) = \text{pass}$. A schema check carries no information about the downstream model's recall; a canary does not measure model quality on the full distribution; and a harness cannot certify its own measurements merely by existing. No role is asked to train a model it does not own; it is asked to name the model-level evidence and who supplies it.

7. Rails, and what a certification establishes

The Level 400 map (levels/400/objectives.yml) defines four rails. Certification vendor and exam are "not cleared" for each (as of 19 November 2026); the course prepares learners for vendor-proctored exams and never sits them or shares live exam content.

RailDraws on
ML engineering400.2-a, 400.2-c, 400.L-a, 400.4-a, 400.4-b, 400.6-a
Data400.3-a, 400.3-b, 400.6-a, 400.6-c, 400.7-a
AI security400.5-a, 400.5-b, 400.5-c, 400.8-b
AI audit400.8-a, 400.8-c, 400.9-a, 400.9-b, 400.9-c

Subject of the claim. Write $C(p, t)$ for "person $p$ passed exam $X$ at time $t$" and $B(m, v)$ for the verified bundle. They are claims about different subjects. $C(p, t)$ is evidence about $p$'s demonstrated knowledge at $t$; it contains no measurement of any $m$. $B(m, v)$ is evidence about one system at one version; it says nothing about $p$'s credentials. In particular,

C(p, t) \;\wedge\; p \in \text{builders}(m) \;\;\not\Rightarrow\;\; E(m, v) = \text{pass} .

The same holds for a course level or badge: it describes a person's practice, not a system's assurance.

A system-level score as a contrast. The ML Test Score is designed "to measure how ready for production a given machine learning system is".2 For each of its 28 tests, half a point is awarded for executing the test manually "with the results documented and distributed", and a full point if a system runs it automatically on a repeated basis; section scores are summed, and the final score is the minimum over the four sections (features and data, model development, ML infrastructure, monitoring).2 Formally, with $t_{ij} \in \{0, \tfrac12, 1\}$ for test $j$ in section $i$,

\text{Score} = \min_{i \in \{1,\dots,4\}} \sum_{j} t_{ij} .

The authors chose the minimum so that a system "must consider all" four sections to raise its score.2 Nothing in the formula refers to who ran the tests or what credentials they hold. A test that is never run, or run once by hand without documented and distributed results, has $t_{ij} = 0$ however qualified its author. The paper states no documentation condition for the automated full point; this course additionally asks for automated results in $B$, as its own evidence rule. That is the distinction 400.R-b asks the learner to state.

8. Exercise

For your track: write $E$ (the seven fields of section 3, with $\sigma$, $\delta$ and every $\varphi_s$ fixed in advance, and $\tau$ kept distinct from $\varphi_s$); if you own a component, write $E_c$ and name $E(m, v)$ and its owner (section 6.1). Write $M$ (at least one monitor per row of section 4 that applies, each with a stated label delay, an owner and an executable action, and say what is held, and how the fallback is shown safe under current conditions, before any rollback). Write the manifest for $B$ (section 5) with a named approver who is not a builder, and confirm it holds no prompts, outputs, payloads or personal data. Give $\rho$ with two objectives you will use in 400.P and one sentence each on how they apply to $m$. Finish with one sentence of the form: "A certification held by anyone on my team would not establish that ...". Where a field cannot be filled yet, write "not known yet" and the owner who will fill it. Carry all of it into 400.P.

9. Limits

Further learning (not cited, never copied)

MIT 6.S191 Introduction to Deep Learning; Carnegie Mellon 10-414/714 Deep Learning Systems; MIT 6.5940 TinyML and Efficient AI Computing (Fall 2026; some materials public). Listed in levels/400/universities.md.

Sources

  1. s1 Elham Tabassi (National Institute of Standards and Technology) (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. Standard. Checked 2026-11-19 by source-checker agent (Claude, fresh session separate from the writer, reviser, item writer and earlier checkers), 400.R round 3 re-check, 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  2. s3 Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. Paper. Checked 2026-11-20 by source-checker agent (Claude, fresh session separate from the writer, reviser, item writer and earlier checkers), 400.R round 4 re-check, 20 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s2 Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, et al. (2018). Model Cards for Model Reporting. Paper. Checked 2026-11-19 by source-checker agent (Claude, fresh session separate from the writer, reviser, item writer and earlier checkers), 400.R round 3 re-check, 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩
  4. s4 Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar (2019). Do ImageNet Classifiers Generalize to ImageNet?. Paper. Checked 2026-11-19 by source-checker agent (Claude, fresh session separate from the writer, reviser, item writer and earlier checkers), 400.R round 3 re-check, 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩
  5. s5 Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, et al. (2023). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!. Paper. Checked 2026-11-19 by source-checker agent (Claude, fresh session separate from the writer, reviser, item writer and earlier checkers), 400.R round 3 re-check, 19 Nov 2026. ↩ ↩ ↩
  6. s6 Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, et al. (2021). Extracting Training Data from Large Language Models. Paper. Checked 2026-11-19 by source-checker agent (Claude, fresh session separate from the writer, reviser, item writer and earlier checkers), 400.R round 3 re-check, 19 Nov 2026. ↩ ↩
  7. s7 Mr Justice Fraser, High Court of Justice, Queen's Bench Division (2019). Bates v Post Office Ltd (No 6) "Horizon Issues" [2019] EWHC 3408 (QB). Record. Checked 2026-11-19 by source-checker agent (Claude, fresh session separate from the writer, reviser, item writer and earlier checkers), 400.R round 3 re-check, 19 Nov 2026. ↩ ↩ ↩ ↩

Next: the quiz