Level 400 objectives

Objectives map signed: Ken Johnston, 2026-11-18 (by proxy: Ken gave his proxy for all levels to Bob Rapp, who approved and recorded it at 20:35 PST). Level 400 lessons are published.

level
400
name
Engineering AI systems
for
Engineers and data scientists who fine-tune, evaluate and operate models.
assumes
completed
Level 300, or placement at 300 and its project (a hold card and an audit receipt for one running system)
skills
  • Python
    read and edit a short script, run a prepared notebook, install a package
  • Probability and statistics
    mean, variance, sampling, a proportion and its confidence interval
  • Linear algebra
    vectors, matrices, the dot product, at the level needed to read a model's shapes
  • Machine learning
    train, validation and test splits; a loss; overfitting; a baseline
entry check
Planned, not yet built. Bob decided on 18 Nov 2026 (decision 6) that Level 400 will build it. A short no-stakes diagnostic, one item per skill above, taken before 400.1; a learner who misses an item is pointed to the bridge, and may still start 400. Placement stays capped at 300.
bridge
Planned, not yet built, and optional. Bob decided on 18 Nov 2026 (decision 6) that Level 400 will build it. Each missed diagnostic item points to the matching bridge part (Python, statistics, linear algebra or ML basics). An optional bridge using a small tabular Big Widget dataset, a runnable baseline and a worked uncertainty example, drawing on the Foundation's free Bachelor's-level program list. Synthetic data only (decision 10).
plain gear
A plain-gear learner who misses the Python or linear-algebra items can still take the whole level, because the plain gear needs no code or equations; the bridge's statistics and ML-basics parts are recommended for them. Plain-gear learners interpret prepared, scaffolded outputs (tables, curves, receipts) without equations or writing code; standard and technical gears run the code.
carries forward
from 300
The learner's Level 300 hold card and audit receipt (300.9-a, 300.9-c, 300.P) are the starting record for 400.4 and 400.9. Each carried item keeps a provenance label (made at 300, content-free, and not authenticated at source unless it was signed when made); signing the 400 bundle proves integrity only from packaging onward, never what happened before.
adds at 400
Versioned monitoring and a rollback plan (400.4), an independent validator distinct from the 300 holder (400.9-b), and an evidence bundle a stranger verifies offline against a key and checkpoint received through a channel independent of the bundle (400.8, 400.9-c, 400.P).
to 500
The verified bundle and the stranger's verification statement are inputs to 500.6 (One Receipt, where assurance levels are interpreted) and 500.8 (designing a governance regime).
reading level
Plain gear at US grade 8, as Level 100 (Bob, 18 Nov 2026, decision 3), without equations; standard and technical gears use light statistics and equations where they help
setting
Big Widget Company, a made-up 100,000-person maker of complex, expensive, car-like widgets
signed by
Ken Johnston, 2026-11-18 (by proxy: Ken gave his proxy for all levels to Bob Rapp, who approved and recorded it at 20:35 PST)
prep rails
  • id
    400.rail-ml-engineering
    rail
    A cloud machine-learning engineering certification (build, deploy and monitor models)
    vendor
    not cleared
    exam
    not cleared
    draws on
    • 400.2-a
    • 400.2-c
    • 400.L-a
    • 400.4-a
    • 400.4-b
    • 400.6-a
  • id
    400.rail-data
    rail
    A data engineering or data science certification (pipelines, data quality, evaluation)
    vendor
    not cleared
    exam
    not cleared
    draws on
    • 400.3-a
    • 400.3-b
    • 400.6-a
    • 400.6-c
    • 400.7-a
  • id
    400.rail-ai-security
    rail
    An AI or cloud security certification (attacks on ML systems, keys, logs)
    vendor
    not cleared
    exam
    not cleared
    draws on
    • 400.5-a
    • 400.5-b
    • 400.5-c
    • 400.8-b
  • id
    400.rail-ai-audit
    rail
    An AI governance or AI management-system audit certification
    vendor
    not cleared
    exam
    not cleared
    draws on
    • 400.8-a
    • 400.8-c
    • 400.9-a
    • 400.9-b
    • 400.9-c
modules
  • id
    400.1
    slug
    how-training-scales
    title
    How training scales: loss, compute and tokens
    strand
    how-it-works
    minutes
    45
    anchor cases
    • chinchilla-2022
    objectives
    • id
      400.1-a
      text
      Explain what next-token cross-entropy loss measures during pre-training, and what a lower loss does and does not say about a downstream Big Widget task.
      misconception
      A lower training loss means the model is better at the job I need.
      quiz item
      Two checkpoints, one with lower validation loss and one with a higher task score on the plant's defect-report summaries; the learner picks which claim each number supports.
    • id
      400.1-b
      text
      Given a published scaling study's stated conditions and a concrete baseline allocation, decide whether the study points to more parameters or more tokens for the same compute, and name one condition under which the result would not transfer (for example inference cost, limited data, or a budget outside the tested range).
      misconception
      A scaling law is a universal rule, and the biggest model you can afford is always the best choice.
      quiz item
      A prepared baseline is deliberately undertrained for its size; under the Chinchilla study's assumptions the learner picks the reallocation, then picks the scenario (a model served millions of times a day) where the answer could change.
    • id
      400.1-c
      text
      Explain how a subword tokenizer splits text, and spot a cost or quality gap it creates across languages or part numbers.
      misconception
      The model reads whole words and letters the way people do.
      quiz item
      The same service instruction in two languages yields very different token counts; the learner names the effect on cost, context and latency.
    starting sources
    • Vaswani et al. 2017, Attention Is All You Need (arXiv 1706.03762; NeurIPS 2017), architecture background only
    • Brown et al. 2020, Language Models are Few-Shot Learners (arXiv 2005.14165; NeurIPS 2020), for the autoregressive pre-training objective
    • Kaplan et al. 2020, Scaling Laws for Neural Language Models (arXiv 2001.08361, preprint)
    • Hoffmann et al. 2022, Training Compute-Optimal Large Language Models (arXiv 2203.15556; NeurIPS 2022)
    • Sennrich, Haddow and Birch 2016, subword units (ACL 2016, P16-1162)
    • Petrov et al. 2023, Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023)
  • id
    400.2
    slug
    adapting-a-model
    title
    Adapting a model: fine-tuning, preference tuning and their side effects
    strand
    how-it-works
    minutes
    50
    anchor cases
    • qi-finetuning-safety-2023
    objectives
    • id
      400.2-a
      text
      Decide between prompting, retrieval and fine-tuning for a given task, and name the evidence that would justify the choice.
      misconception
      Fine-tuning is the way to teach a model new company facts.
      quiz item
      Service bulletins change weekly; the learner picks retrieval over fine-tuning and cites the comparison that supports it.
    • id
      400.2-b
      text
      Separate the two design choices in adapting a model, which parameters are updated (all of them, or a low-rank adapter such as LoRA) and which objective and data drive the update (supervised examples, or preference pairs through RLHF or DPO), and explain what each choice needs.
      misconception
      LoRA, RLHF and DPO are three rival kinds of fine-tuning.
      quiz item
      A two-by-two grid (full or LoRA update; supervised or preference objective); the learner places four described runs and names the data each needs.
    • id
      400.2-c
      text
      Spot how a fine-tune can erode a base model's safety behaviour, and require re-evaluation before a fine-tuned model is released.
      misconception
      Fine-tuning on harmless data keeps the base model's safety intact.
      quiz item
      A team fine-tunes on ten benign-looking examples and skips the safety evaluation; the learner names the documented risk and the release check.
    starting sources
    • Hu et al. 2021, LoRA (arXiv 2106.09685; ICLR 2022)
    • Ouyang et al. 2022, InstructGPT (NeurIPS 2022)
    • Rafailov et al. 2023, Direct Preference Optimization (NeurIPS 2023)
    • Qi et al. 2023, Fine-tuning Aligned Language Models Compromises Safety (arXiv 2310.03693; ICLR 2024)
    • Ovadia et al. 2023, Fine-Tuning or Retrieval? (arXiv 2312.05934, preprint)
  • id
    400.L
    slug
    adaptation-lab
    title
    Adaptation lab: fine-tune, compare and re-check, offline
    strand
    building
    minutes
    60
    anchor cases
    • qi-finetuning-safety-2023
    plain route
    No-code route (Bob, decision 7). Plain-gear learners read the prepared before-and-after results (baseline and fine-tune scores with intervals, and the safety and cohort checks re-run on the fine-tuned model), apply the same preset release rule to them, and fill in the same release receipt as the code routes; they meet the same two objectives without running code.
    fixtures
    Synthetic fixtures only, owned by this course repo; no private-repo excerpts (Bob, 18 Nov 2026, decision 10). A prepared, non-production sandbox that runs offline, a small made-up Big Widget service-notes dataset with a held-out split, a small open base model, prepared run outputs for the plain gear, and a made-up labelled safety set in which each request is labelled prohibited (the model should refuse) or benign (the model should answer), with the number of eligible requests in each label stated
    objectives
    • id
      400.L-a
      text
      Run (standard and technical gears) or read the prepared outputs of (plain gear) a bounded LoRA fine-tune in the sandbox, and compare it with a prompt-only baseline and a retrieval baseline on the same held-out set, with an interval for each score.
      misconception
      If the fine-tune's training loss fell, the fine-tune worked.
      quiz item
      Three result rows (prompt-only, retrieval, LoRA) with intervals; the learner picks the conclusion the numbers support, including "no clear difference".
    • id
      400.L-b
      text
      Re-run the safety and cohort checks on the fine-tuned model against the labelled safety set (in the plain gear, read the prepared re-run results), apply the release rule written before the run, and record one bounded decision (release, or hold and re-check) as a receipt before the model may be released. The rule counts the two failures separately, complying with a prohibited request and refusing a benign one, each as a count out of its eligible requests with an interval; when the evidence is insufficient (the set is too small or an interval straddles the rule's limit), the rule's answer is hold and re-check.
      misconception
      Checks run on the base model still hold after a fine-tune, and a lower refusal rate means the model became more helpful.
      quiz item
      A table for base and fine-tune gives, per label, the eligible requests and how many the model refused, with intervals, plus the preset rule; the learner applies the rule, picks release or hold and re-check, and says which failure the change in refusals reflects and what the receipt records. The keyed answer is the same in plain and code routes.
    starting sources
    • Hu et al. 2021, LoRA (arXiv 2106.09685; ICLR 2022)
    • Ovadia et al. 2023, Fine-Tuning or Retrieval? (arXiv 2312.05934, preprint)
    • Qi et al. 2023, Fine-tuning Aligned Language Models Compromises Safety (arXiv 2310.03693; ICLR 2024)
    • Miller 2024, Adding Error Bars to Evals (arXiv 2411.00640, preprint)
  • id
    400.3
    slug
    data-with-a-paper-trail
    title
    Data with a paper trail
    strand
    building
    minutes
    45
    anchor cases
    • laion-csam-2023
    objectives
    • id
      400.3-a
      text
      Write a datasheet for a training or evaluation dataset (motivation, composition, collection, consent, licence, known gaps) and a model card for the model trained on it.
      misconception
      If the data is public, documenting it is optional.
      quiz item
      Given a datasheet missing its collection process and licence, the learner names the two gaps that block release.
    • id
      400.3-b
      text
      Trace a dataset's provenance and licence from its source to the model, and flag where the chain is missing or contradicts itself.
      misconception
      The licence label on a dataset hub is reliable.
      quiz item
      A fine-tuning set's hub licence says commercial use is allowed but its source says otherwise; the learner decides what the team may do.
    • id
      400.3-c
      text
      Explain how harmful, unlawful or poisoned content in a web-scale dataset reaches the models trained on it, and what removal requires.
      misconception
      Bad items in a huge dataset are too rare to matter.
      quiz item
      The learner orders the steps after a dataset is found to contain unlawful material (stop use, notify, remove, retrain or retire, record).
    starting sources
    • Gebru et al. 2018, Datasheets for Datasets (arXiv 1803.09010; CACM 2021)
    • Mitchell et al. 2018, Model Cards for Model Reporting (arXiv 1810.03993; FAT* 2019)
    • Longpre et al. 2024, A large-scale audit of dataset licensing and attribution in AI (Nature Machine Intelligence)
    • Thiel and Hancock 2023, Identifying and Eliminating CSAM in Generative ML Training Data and Models (Stanford Digital Repository)
    • Carlini et al. 2023, Poisoning Web-Scale Training Datasets is Practical (arXiv 2302.10149)
    • Regulation (EU) 2024/1689, Article 10 (data and data governance), original 2024 text; amended by Regulation (EU) 2026/1744, so check the amended text before teaching a current rule
  • id
    400.4
    slug
    operating-a-model
    title
    Operating a model: monitoring, drift and rollback
    strand
    building
    minutes
    50
    anchor cases
      anchor note
      No primary-backed operating case has been established yet. The module uses a labelled, made-up release-and-monitoring record as its fixture until one is found; the Zillow Offers lead was not opened from a primary record and must not be presented as an established example (source-packs/README.md).
      fixtures
      Synthetic fixtures only, owned by this course repo; no private-repo excerpts (Bob, 18 Nov 2026, decision 10). A made-up, clearly labelled Big Widget release-and-monitoring record with delayed labels, an input-shift alert, and the rollback that followed
      objectives
      • id
        400.4-a
        text
        Starting from the system's Level 300 hold card (holder, thresholds, stop), add versioned monitoring that detects input shift and performance decay when labels arrive late, and name the threshold that pages the holder.
        misconception
        A model that passed its tests stays good once deployed.
        quiz item
        In the made-up record, a demand-forecasting model's inputs shift after a new dealer region opens and labels arrive six weeks later; the learner picks the early signal, the threshold and who is paged.
      • id
        400.4-b
        text
        Write a release and rollback plan that versions model, data, code and configuration together, stages the rollout and names who can stop it.
        misconception
        Rolling back a model is the same as rolling back code.
        quiz item
        The learner spots the missing piece in a rollback plan that pins the model weights but not the feature pipeline.
      • id
        400.4-c
        text
        Spot hidden technical debt in an ML system (entanglement, hidden feedback loops, undeclared consumers) and name one control for each.
        misconception
        The model is most of the system.
        quiz item
        A model's output feeds a pricing tool that feeds next month's training data; the learner names the loop.
      starting sources
      • Sculley et al. 2015, Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015)
      • Breck et al. 2017, The ML Test Score (IEEE Big Data 2017)
      • Rabanser, Günnemann and Lipton 2019, Failing Loudly (NeurIPS 2019)
      • Shankar et al. 2022, Operationalizing Machine Learning
        An Interview Study (arXiv 2209.09125, preprint)
      • NIST AI RMF 1.0 (2023), Manage function
    • id
      400.5
      slug
      securing-a-model
      title
      Securing a model: poisoning, extraction and injection
      strand
      building
      minutes
      50
      anchor cases
      • carlini-extraction-2021
      objectives
      • id
        400.5-a
        text
        Name the main classes of attack on ML systems by lifecycle stage (poisoning, evasion, privacy and extraction, prompt injection) using a published taxonomy.
        misconception
        AI security is just keeping the API key secret.
        quiz item
        Sort five incidents into NIST's attack classes and the lifecycle stage each hits.
      • id
        400.5-b
        text
        Explain how a model can reveal memorised training data, and name one training-time and one release-time control.
        misconception
        Model weights do not contain the training data, so they are safe to share.
        quiz item
        The learner decides whether to release a model fine-tuned on warranty claims that include customers' names and numbers.
      • id
        400.5-c
        text
        Decide which limits to put around a model that reads untrusted content (least privilege, isolation, human approval for actions), rather than relying on instructions in the prompt.
        misconception
        A better system prompt fixes prompt injection.
        quiz item
        An assistant reads supplier emails and can send purchase orders; the learner picks the control that still holds when an email contains hidden instructions.
      starting sources
      • NIST AI 100-2 E2025, Adversarial Machine Learning taxonomy (24 Mar 2025)
      • Carlini et al. 2021, Extracting Training Data from Large Language Models (USENIX Security 2021)
      • Greshake et al. 2023, indirect prompt injection (arXiv 2302.12173)
      • Carlini et al. 2023, Poisoning Web-Scale Training Datasets is Practical (arXiv 2302.10149)
    • id
      400.6
      slug
      evaluations-that-hold-up
      title
      Evaluations that hold up
      strand
      verifying
      minutes
      50
      anchor cases
      • imagenet-v2-2019
      objectives
      • id
        400.6-a
        text
        Design an evaluation for a Big Widget task with held-out data that matches real use, a baseline, a metric tied to the harm, and a sample size planned in advance.
        misconception
        A public leaderboard score tells you how a model will do on your task.
        quiz item
        The learner picks the test set that best predicts performance on next quarter's warranty claims.
      • id
        400.6-b
        text
        Spot test-set contamination and test-set reuse, and explain why scores can drop on fresh data drawn the same way.
        misconception
        If the model never saw the test file, the score is clean.
        quiz item
        Given the ImageNet v2 result, the learner explains what the drop does and does not show.
      • id
        400.6-c
        text
        Report an evaluation result with its uncertainty, say when a difference between two models is too small to act on, and check an LLM judge against people before trusting it.
        misconception
        A two-point gain on a benchmark is a real improvement, and an LLM judge agrees with people.
        quiz item
        Two items: (1) a paired table of the same 200 items scored by two models, with the prepared interval for the difference; the learner chooses the honest report. (2) An LLM judge's verdicts beside people's on 50 items; the learner reads the agreement table and decides whether the judge may replace people.
      starting sources
      • Recht et al. 2019, Do ImageNet Classifiers Generalize to ImageNet? (ICML 2019)
      • Sainz et al. 2023, NLP Evaluation in trouble (Findings of EMNLP 2023)
      • Miller 2024, Adding Error Bars to Evals (arXiv 2411.00640, preprint)
      • Zheng et al. 2023, Judging LLM-as-a-Judge (NeurIPS 2023 Datasets and Benchmarks)
      • Liang et al. 2022, HELM (TMLR 2023)
    • id
      400.7
      slug
      testing-for-fairness-and-proxies
      title
      Testing for fairness and proxies
      strand
      verifying
      minutes
      45
      anchor cases
      • toeslagenaffaire-2020
      objectives
      • id
        400.7-a
        text
        Measure a model's error and outcome rates by cohort, and show how one aggregate number can hide harm to a group.
        misconception
        High overall accuracy means the model is fair to everyone.
        quiz item
        A small made-up cohort table for a supplier-risk model giving, per supplier group, how many suppliers were in fact low-risk and how many of those were flagged; the learner computes each group's false-positive rate (flagged low-risk suppliers divided by all low-risk suppliers in the group) and spots the group the overall rate hides.
      • id
        400.7-b
        text
        Tell direct use of a protected attribute apart from use of a proxy (a feature that stands in for it), and explain why deleting the protected column does not remove a proxy's effect.
        misconception
        If nationality is not a column, the model cannot discriminate by nationality.
        quiz item
        Two parts: the Dutch DPA case is labelled as direct use (nationality itself was an indicator); then, in a made-up Big Widget supplier dataset with the nationality column removed, the learner finds the feature that correlates with it (such as the country of a supplier's bank).
      • id
        400.7-c
        text
        Explain, from Kleinberg, Mullainathan and Raghavan's result on risk scores, why calibration within groups, balance for the positive class and balance for the negative class cannot all hold at once except in highly constrained special cases (such as perfect prediction or equal base rates), so the team must choose, write down why, and name who signed it.
        misconception
        There is one fairness test, and a model either passes it or fails it.
        quiz item
        With risk scores for two groups at different base rates, the learner picks which of the three Kleinberg conditions a described score meets and which it must give up, and says when the conflict disappears. Chouldechova's thresholded result is taught separately, not keyed in this item.
      starting sources
      • Kleinberg, Mullainathan and Raghavan 2016, Inherent Trade-Offs in the Fair Determination of Risk Scores (ITCS 2017)
      • Chouldechova 2017, Fair prediction with disparate impact (arXiv 1703.00056)
      • NIST SP 1270 (2022), Towards a Standard for Identifying and Managing Bias in AI
      • Autoriteit Persoonsgegevens, fine on the Belastingdienst for discriminatory processing of nationality, 7 Dec 2021
      • Barocas, Hardt and Narayanan 2023, Fairness and Machine Learning (MIT Press; fairmlbook.org)
    • id
      400.8
      slug
      records-a-stranger-can-verify
      title
      Records a stranger can verify
      strand
      governing
      minutes
      50
      reuses
      Practice 400 Prove it worksheet, step 4 (evidence to the regulator), and the Beacon evidence bundle
      anchor cases
      • post-office-horizon
      objectives
      • id
        400.8-a
        text
        Explain, using the Horizon judgments, why a system's output must not be taken as reliable evidence against a person just because a computer produced it.
        misconception
        Computer records are accurate unless someone proves otherwise.
        quiz item
        The learner picks which Horizon finding answers the claim "the system said the cash was short, so it was".
      • id
        400.8-b
        text
        Design a tamper-evident record (canonical serialisation, a hash chain, a signature) and state what it proves only when checked against pinned trust material and an expected checkpoint received through a channel independent of the bundle, and what it never proves on its own (including anything that happened before the record was signed).
        misconception
        A signed record proves that what it says is true, and a chain that verifies is complete.
        quiz item
        A chain verifies internally; the learner sorts claims into proved (unchanged since signing, by the holder of the pinned key, up to the expected checkpoint) and not proved (nothing was cut after the checkpoint without the checkpoint; whose key it is without pinned key material; that the decision was right).
      • id
        400.8-c
        text
        List what an evidence bundle must contain for an auditor who has never seen the system (chain, policy and its hash, framework crosswalk, VERIFY.md) and what it must never contain (prompts, outputs, personal data).
        misconception
        More evidence is always better, so include the prompts and outputs.
        quiz item
        The learner removes the two items from a draft bundle that would make it fail.
      starting sources
      • Bates v Post Office Ltd (No 6) "Horizon Issues" [2019] EWHC 3408 (QB), 16 Dec 2019
      • Hamilton v Post Office Ltd [2021] EWCA Crim 577, 23 Apr 2021
      • Crosby and Wallach 2009, Efficient Data Structures for Tamper-Evident Logging (USENIX Security 2009)
      • RFC 8785 (JCS, 2020), RFC 8032 (EdDSA, 2017), RFC 9162 (Certificate Transparency 2.0, 2021)
      • Regulation (EU) 2024/1689, Article 12 (record-keeping)
      • AiGovOps Beacon: evidence bundle with VERIFY.md (https://aigovops-foundation.github.io/aigovops-beacon/)
    • id
      400.9
      slug
      the-audit-gate
      title
      The audit gate: prove it to a stranger
      strand
      governing
      minutes
      60
      reuses
      Practice 400 Prove it worksheet (Robodebt, toeslagenaffaire, Horizon); the audit-training quiz; the Beacon evidence bundle
      anchor cases
      • robodebt-2023
      • toeslagenaffaire-2020
      • post-office-horizon
      objectives
      • id
        400.9-a
        text
        Explain, using the Robodebt Royal Commission, why the premise of an automated decision (its formula, threshold and data) must be independently validated as lawful and sound before the first decision is issued.
        misconception
        If the formula is simple arithmetic, it does not need validating.
        quiz item
        The learner names who should validate a new warranty-clawback formula, and what evidence they need before the first notice.
      • id
        400.9-b
        text
        Write the premise gate and the protected-attribute gate as policy (an independent validator distinct from the Level 300 holder, what evidence, deny unless a receipt exists) and as two sentences a minister could read.
        misconception
        The team that built the rule can approve it.
        quiz item
        Given a gate where the approver is the rule's author, the learner fixes the approver role and the deny condition.
      • id
        400.9-c
        text
        Verify a stranger's evidence bundle offline by its VERIFY.md against pinned trust material (the expected public key) and an expected checkpoint (the last sequence number and hash, and the policy version in scope), and write a one-sentence verification statement with your name, the date and what you found.
        misconception
        Verifying evidence needs the vendor's tool, an account or a network, and a bundle whose chain verifies is complete.
        quiz item
        Given a known-good fixture and a truncated fixture that both verify internally, the learner uses the expected checkpoint (received separately from the bundle) to reject the truncated one; then, for a valid bundle that carries an unauthenticated Level 300 receipt, the learner writes a statement bounded to what was verified and spots the one that overclaims. Interpreting assurance levels is left to 500.6.
      starting sources
      • Report of the Royal Commission into the Robodebt Scheme, 7 Jul 2023 (updated 11 Jul 2023)
      • Tweede Kamer, Parlementaire ondervragingscommissie Kinderopvangtoeslag, Ongekend onrecht, 17 Dec 2020 (Kamerstuk 35 510, nr. 2)
      • Autoriteit Persoonsgegevens, fine on the Belastingdienst, 7 Dec 2021
      • Hamilton v Post Office Ltd [2021] EWCA Crim 577, 23 Apr 2021
      • Practice Level 400 Prove it worksheet: https://practice.aigovops-foundation.com/levels/400-prove/WORKSHEET.html
      • AiGovOps Foundation, How do I create an AI audit plan? (audit-training quiz: https://www.aigovops-foundation.com/audit.html)
    • id
      400.R
      slug
      engineering-ai-in-your-role
      title
      Engineering AI in your role (role tracks)
      strand
      building
      minutes
      45
      tracks
      • ml-engineering
      • data-science
      • data-engineering
      • mlops-and-platform
      • security-engineering
      • evaluation-and-quality
      • manufacturing-and-quality-inspection
      • maintenance-and-reliability
      • forecasting-and-supply-chain
      • service-and-warranty
      • engineering-and-simulation
      objectives
      • id
        400.R-a
        text
        For one model in your role at Big Widget Company, name the evaluation that would show it works, the monitoring that would show it still works, and the evidence a stranger would need.
        misconception
        Evidence is the governance team's job, not the engineer's.
        quiz item
        For a visual-inspection model on the paint line, the learner picks the evaluation, the monitor and the receipt.
      • id
        400.R-b
        text
        Map your role's work to one cert-prep rail and the course objectives it draws on, and explain what a certification does and does not prove.
        misconception
        A certification proves the system I built is safe.
        quiz item
        The learner separates what a passed exam shows about a person from what an evidence bundle shows about a system.
      starting sources
      • Reuses the Level 400 case bank (levels/400/cases.yml) and the module source packs: each track row names the case and module it draws on
      • NIST AI RMF 1.0 (2023)
    • id
      400.P
      slug
      prove-it-to-a-stranger
      title
      Project: prove it to a stranger
      strand
      project
      minutes
      120
      time note
      The 120 minutes are the learner's build time in the sandbox. Marking by two 400s or a host is extra and not counted here. Hosting a Thursday call is optional, not a badge condition (Bob, 18 Nov 2026, decision 9). Timings approved as drafted (decision 8).
      reuses
      Practice 400 Prove it worksheet (60 minutes, plus the Thursday you host)
      fixtures
      Synthetic fixtures only, owned by this course repo; no private-repo excerpts (Bob, 18 Nov 2026, decision 10). The 400.L sandbox, model and dataset; the 400.4 made-up monitoring record; the learner's Level 300 hold card; an offline verifier and a pinned public key for the bundle. Nothing touches production data or a live system.
      pass evidence
      Minimum for every gear: the comparison table with intervals and one cohort row; a release gate naming an independent approver, shown denying one release run whose evidence is missing; one dry rollback that restores the previous model, data and configuration versions; a monitoring plan built on the 300 hold card; and a bundle, with carried 300 items provenance-labelled, that a stranger verified against a key and checkpoint received separately. Plain-gear learners use prepared sandbox states and outputs instead of running code.
      objectives
      • id
        400.P-a
        text
        Using the sandbox fixtures, adapt a model for one Big Widget task, evaluate it against a baseline with intervals and a cohort check, then exercise its release gate (one run denied for missing evidence) and one dry rollback, and write the monitoring plan that would govern it.
        misconception
        A model is done when its benchmark score is good.
        quiz item
        Project rubric, not a quiz item; the module quiz checks the rubric's terms (baseline, interval, cohort, gate, rollback).
      • id
        400.P-b
        text
        Produce an evidence bundle, with carried Level 300 items provenance-labelled, that a stranger verifies offline by its VERIFY.md against a key and checkpoint received separately, and states in one signed sentence bounded to what was verified; two 400s or a host mark it. Hosting a Thursday is optional, not a badge condition.
        misconception
        Evidence counts only if the auditor trusts the team that made it.
        quiz item
        Project rubric; the module quiz checks what the stranger's statement may and may not claim.
      starting sources
      • Practice Level 400 Prove it worksheet: https://practice.aigovops-foundation.com/levels/400-prove/WORKSHEET.html
      • AiGovOps Beacon: https://aigovops-foundation.github.io/aigovops-beacon/