Level 400 objectives
Objectives map signed: Ken Johnston, 2026-11-18 (by proxy: Ken gave his proxy for all levels to Bob Rapp, who approved and recorded it at 20:35 PST). Level 400 lessons are published.
- level
- 400
- name
- Engineering AI systems
- for
- Engineers and data scientists who fine-tune, evaluate and operate models.
- assumes
- completed
- Level 300, or placement at 300 and its project (a hold card and an audit receipt for one running system)
- skills
- Python
- read and edit a short script, run a prepared notebook, install a package
- Probability and statistics
- mean, variance, sampling, a proportion and its confidence interval
- Linear algebra
- vectors, matrices, the dot product, at the level needed to read a model's shapes
- Machine learning
- train, validation and test splits; a loss; overfitting; a baseline
- entry check
- Planned, not yet built. Bob decided on 18 Nov 2026 (decision 6) that Level 400 will build it. A short no-stakes diagnostic, one item per skill above, taken before 400.1; a learner who misses an item is pointed to the bridge, and may still start 400. Placement stays capped at 300.
- bridge
- Planned, not yet built, and optional. Bob decided on 18 Nov 2026 (decision 6) that Level 400 will build it. Each missed diagnostic item points to the matching bridge part (Python, statistics, linear algebra or ML basics). An optional bridge using a small tabular Big Widget dataset, a runnable baseline and a worked uncertainty example, drawing on the Foundation's free Bachelor's-level program list. Synthetic data only (decision 10).
- plain gear
- A plain-gear learner who misses the Python or linear-algebra items can still take the whole level, because the plain gear needs no code or equations; the bridge's statistics and ML-basics parts are recommended for them. Plain-gear learners interpret prepared, scaffolded outputs (tables, curves, receipts) without equations or writing code; standard and technical gears run the code.
- carries forward
- from 300
- The learner's Level 300 hold card and audit receipt (300.9-a, 300.9-c, 300.P) are the starting record for 400.4 and 400.9. Each carried item keeps a provenance label (made at 300, content-free, and not authenticated at source unless it was signed when made); signing the 400 bundle proves integrity only from packaging onward, never what happened before.
- adds at 400
- Versioned monitoring and a rollback plan (400.4), an independent validator distinct from the 300 holder (400.9-b), and an evidence bundle a stranger verifies offline against a key and checkpoint received through a channel independent of the bundle (400.8, 400.9-c, 400.P).
- to 500
- The verified bundle and the stranger's verification statement are inputs to 500.6 (One Receipt, where assurance levels are interpreted) and 500.8 (designing a governance regime).
- reading level
- Plain gear at US grade 8, as Level 100 (Bob, 18 Nov 2026, decision 3), without equations; standard and technical gears use light statistics and equations where they help
- setting
- Big Widget Company, a made-up 100,000-person maker of complex, expensive, car-like widgets
- signed by
- Ken Johnston, 2026-11-18 (by proxy: Ken gave his proxy for all levels to Bob Rapp, who approved and recorded it at 20:35 PST)
- prep rails
- id
- 400.rail-ml-engineering
- rail
- A cloud machine-learning engineering certification (build, deploy and monitor models)
- vendor
- not cleared
- exam
- not cleared
- draws on
- 400.2-a
- 400.2-c
- 400.L-a
- 400.4-a
- 400.4-b
- 400.6-a
- id
- 400.rail-data
- rail
- A data engineering or data science certification (pipelines, data quality, evaluation)
- vendor
- not cleared
- exam
- not cleared
- draws on
- 400.3-a
- 400.3-b
- 400.6-a
- 400.6-c
- 400.7-a
- id
- 400.rail-ai-security
- rail
- An AI or cloud security certification (attacks on ML systems, keys, logs)
- vendor
- not cleared
- exam
- not cleared
- draws on
- 400.5-a
- 400.5-b
- 400.5-c
- 400.8-b
- id
- 400.rail-ai-audit
- rail
- An AI governance or AI management-system audit certification
- vendor
- not cleared
- exam
- not cleared
- draws on
- 400.8-a
- 400.8-c
- 400.9-a
- 400.9-b
- 400.9-c
- modules
- id
- 400.1
- slug
- how-training-scales
- title
- How training scales: loss, compute and tokens
- strand
- how-it-works
- minutes
- 45
- anchor cases
- chinchilla-2022
- objectives
- id
- 400.1-a
- text
- Explain what next-token cross-entropy loss measures during pre-training, and what a lower loss does and does not say about a downstream Big Widget task.
- misconception
- A lower training loss means the model is better at the job I need.
- quiz item
- Two checkpoints, one with lower validation loss and one with a higher task score on the plant's defect-report summaries; the learner picks which claim each number supports.
- id
- 400.1-b
- text
- Given a published scaling study's stated conditions and a concrete baseline allocation, decide whether the study points to more parameters or more tokens for the same compute, and name one condition under which the result would not transfer (for example inference cost, limited data, or a budget outside the tested range).
- misconception
- A scaling law is a universal rule, and the biggest model you can afford is always the best choice.
- quiz item
- A prepared baseline is deliberately undertrained for its size; under the Chinchilla study's assumptions the learner picks the reallocation, then picks the scenario (a model served millions of times a day) where the answer could change.
- id
- 400.1-c
- text
- Explain how a subword tokenizer splits text, and spot a cost or quality gap it creates across languages or part numbers.
- misconception
- The model reads whole words and letters the way people do.
- quiz item
- The same service instruction in two languages yields very different token counts; the learner names the effect on cost, context and latency.
- starting sources
- Vaswani et al. 2017, Attention Is All You Need (arXiv 1706.03762; NeurIPS 2017), architecture background only
- Brown et al. 2020, Language Models are Few-Shot Learners (arXiv 2005.14165; NeurIPS 2020), for the autoregressive pre-training objective
- Kaplan et al. 2020, Scaling Laws for Neural Language Models (arXiv 2001.08361, preprint)
- Hoffmann et al. 2022, Training Compute-Optimal Large Language Models (arXiv 2203.15556; NeurIPS 2022)
- Sennrich, Haddow and Birch 2016, subword units (ACL 2016, P16-1162)
- Petrov et al. 2023, Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023)
- id
- 400.2
- slug
- adapting-a-model
- title
- Adapting a model: fine-tuning, preference tuning and their side effects
- strand
- how-it-works
- minutes
- 50
- anchor cases
- qi-finetuning-safety-2023
- objectives
- id
- 400.2-a
- text
- Decide between prompting, retrieval and fine-tuning for a given task, and name the evidence that would justify the choice.
- misconception
- Fine-tuning is the way to teach a model new company facts.
- quiz item
- Service bulletins change weekly; the learner picks retrieval over fine-tuning and cites the comparison that supports it.
- id
- 400.2-b
- text
- Separate the two design choices in adapting a model, which parameters are updated (all of them, or a low-rank adapter such as LoRA) and which objective and data drive the update (supervised examples, or preference pairs through RLHF or DPO), and explain what each choice needs.
- misconception
- LoRA, RLHF and DPO are three rival kinds of fine-tuning.
- quiz item
- A two-by-two grid (full or LoRA update; supervised or preference objective); the learner places four described runs and names the data each needs.
- id
- 400.2-c
- text
- Spot how a fine-tune can erode a base model's safety behaviour, and require re-evaluation before a fine-tuned model is released.
- misconception
- Fine-tuning on harmless data keeps the base model's safety intact.
- quiz item
- A team fine-tunes on ten benign-looking examples and skips the safety evaluation; the learner names the documented risk and the release check.
- starting sources
- Hu et al. 2021, LoRA (arXiv 2106.09685; ICLR 2022)
- Ouyang et al. 2022, InstructGPT (NeurIPS 2022)
- Rafailov et al. 2023, Direct Preference Optimization (NeurIPS 2023)
- Qi et al. 2023, Fine-tuning Aligned Language Models Compromises Safety (arXiv 2310.03693; ICLR 2024)
- Ovadia et al. 2023, Fine-Tuning or Retrieval? (arXiv 2312.05934, preprint)
- id
- 400.L
- slug
- adaptation-lab
- title
- Adaptation lab: fine-tune, compare and re-check, offline
- strand
- building
- minutes
- 60
- anchor cases
- qi-finetuning-safety-2023
- plain route
- No-code route (Bob, decision 7). Plain-gear learners read the prepared before-and-after results (baseline and fine-tune scores with intervals, and the safety and cohort checks re-run on the fine-tuned model), apply the same preset release rule to them, and fill in the same release receipt as the code routes; they meet the same two objectives without running code.
- fixtures
- Synthetic fixtures only, owned by this course repo; no private-repo excerpts (Bob, 18 Nov 2026, decision 10). A prepared, non-production sandbox that runs offline, a small made-up Big Widget service-notes dataset with a held-out split, a small open base model, prepared run outputs for the plain gear, and a made-up labelled safety set in which each request is labelled prohibited (the model should refuse) or benign (the model should answer), with the number of eligible requests in each label stated
- objectives
- id
- 400.L-a
- text
- Run (standard and technical gears) or read the prepared outputs of (plain gear) a bounded LoRA fine-tune in the sandbox, and compare it with a prompt-only baseline and a retrieval baseline on the same held-out set, with an interval for each score.
- misconception
- If the fine-tune's training loss fell, the fine-tune worked.
- quiz item
- Three result rows (prompt-only, retrieval, LoRA) with intervals; the learner picks the conclusion the numbers support, including "no clear difference".
- id
- 400.L-b
- text
- Re-run the safety and cohort checks on the fine-tuned model against the labelled safety set (in the plain gear, read the prepared re-run results), apply the release rule written before the run, and record one bounded decision (release, or hold and re-check) as a receipt before the model may be released. The rule counts the two failures separately, complying with a prohibited request and refusing a benign one, each as a count out of its eligible requests with an interval; when the evidence is insufficient (the set is too small or an interval straddles the rule's limit), the rule's answer is hold and re-check.
- misconception
- Checks run on the base model still hold after a fine-tune, and a lower refusal rate means the model became more helpful.
- quiz item
- A table for base and fine-tune gives, per label, the eligible requests and how many the model refused, with intervals, plus the preset rule; the learner applies the rule, picks release or hold and re-check, and says which failure the change in refusals reflects and what the receipt records. The keyed answer is the same in plain and code routes.
- starting sources
- Hu et al. 2021, LoRA (arXiv 2106.09685; ICLR 2022)
- Ovadia et al. 2023, Fine-Tuning or Retrieval? (arXiv 2312.05934, preprint)
- Qi et al. 2023, Fine-tuning Aligned Language Models Compromises Safety (arXiv 2310.03693; ICLR 2024)
- Miller 2024, Adding Error Bars to Evals (arXiv 2411.00640, preprint)
- id
- 400.3
- slug
- data-with-a-paper-trail
- title
- Data with a paper trail
- strand
- building
- minutes
- 45
- anchor cases
- laion-csam-2023
- objectives
- id
- 400.3-a
- text
- Write a datasheet for a training or evaluation dataset (motivation, composition, collection, consent, licence, known gaps) and a model card for the model trained on it.
- misconception
- If the data is public, documenting it is optional.
- quiz item
- Given a datasheet missing its collection process and licence, the learner names the two gaps that block release.
- id
- 400.3-b
- text
- Trace a dataset's provenance and licence from its source to the model, and flag where the chain is missing or contradicts itself.
- misconception
- The licence label on a dataset hub is reliable.
- quiz item
- A fine-tuning set's hub licence says commercial use is allowed but its source says otherwise; the learner decides what the team may do.
- id
- 400.3-c
- text
- Explain how harmful, unlawful or poisoned content in a web-scale dataset reaches the models trained on it, and what removal requires.
- misconception
- Bad items in a huge dataset are too rare to matter.
- quiz item
- The learner orders the steps after a dataset is found to contain unlawful material (stop use, notify, remove, retrain or retire, record).
- starting sources
- Gebru et al. 2018, Datasheets for Datasets (arXiv 1803.09010; CACM 2021)
- Mitchell et al. 2018, Model Cards for Model Reporting (arXiv 1810.03993; FAT* 2019)
- Longpre et al. 2024, A large-scale audit of dataset licensing and attribution in AI (Nature Machine Intelligence)
- Thiel and Hancock 2023, Identifying and Eliminating CSAM in Generative ML Training Data and Models (Stanford Digital Repository)
- Carlini et al. 2023, Poisoning Web-Scale Training Datasets is Practical (arXiv 2302.10149)
- Regulation (EU) 2024/1689, Article 10 (data and data governance), original 2024 text; amended by Regulation (EU) 2026/1744, so check the amended text before teaching a current rule
- id
- 400.4
- slug
- operating-a-model
- title
- Operating a model: monitoring, drift and rollback
- strand
- building
- minutes
- 50
- anchor cases
- anchor note
- No primary-backed operating case has been established yet. The module uses a labelled, made-up release-and-monitoring record as its fixture until one is found; the Zillow Offers lead was not opened from a primary record and must not be presented as an established example (source-packs/README.md).
- fixtures
- Synthetic fixtures only, owned by this course repo; no private-repo excerpts (Bob, 18 Nov 2026, decision 10). A made-up, clearly labelled Big Widget release-and-monitoring record with delayed labels, an input-shift alert, and the rollback that followed
- objectives
- id
- 400.4-a
- text
- Starting from the system's Level 300 hold card (holder, thresholds, stop), add versioned monitoring that detects input shift and performance decay when labels arrive late, and name the threshold that pages the holder.
- misconception
- A model that passed its tests stays good once deployed.
- quiz item
- In the made-up record, a demand-forecasting model's inputs shift after a new dealer region opens and labels arrive six weeks later; the learner picks the early signal, the threshold and who is paged.
- id
- 400.4-b
- text
- Write a release and rollback plan that versions model, data, code and configuration together, stages the rollout and names who can stop it.
- misconception
- Rolling back a model is the same as rolling back code.
- quiz item
- The learner spots the missing piece in a rollback plan that pins the model weights but not the feature pipeline.
- id
- 400.4-c
- text
- Spot hidden technical debt in an ML system (entanglement, hidden feedback loops, undeclared consumers) and name one control for each.
- misconception
- The model is most of the system.
- quiz item
- A model's output feeds a pricing tool that feeds next month's training data; the learner names the loop.
- starting sources
- Sculley et al. 2015, Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015)
- Breck et al. 2017, The ML Test Score (IEEE Big Data 2017)
- Rabanser, Günnemann and Lipton 2019, Failing Loudly (NeurIPS 2019)
- Shankar et al. 2022, Operationalizing Machine Learning
- An Interview Study (arXiv 2209.09125, preprint)
- NIST AI RMF 1.0 (2023), Manage function
- id
- 400.5
- slug
- securing-a-model
- title
- Securing a model: poisoning, extraction and injection
- strand
- building
- minutes
- 50
- anchor cases
- carlini-extraction-2021
- objectives
- id
- 400.5-a
- text
- Name the main classes of attack on ML systems by lifecycle stage (poisoning, evasion, privacy and extraction, prompt injection) using a published taxonomy.
- misconception
- AI security is just keeping the API key secret.
- quiz item
- Sort five incidents into NIST's attack classes and the lifecycle stage each hits.
- id
- 400.5-b
- text
- Explain how a model can reveal memorised training data, and name one training-time and one release-time control.
- misconception
- Model weights do not contain the training data, so they are safe to share.
- quiz item
- The learner decides whether to release a model fine-tuned on warranty claims that include customers' names and numbers.
- id
- 400.5-c
- text
- Decide which limits to put around a model that reads untrusted content (least privilege, isolation, human approval for actions), rather than relying on instructions in the prompt.
- misconception
- A better system prompt fixes prompt injection.
- quiz item
- An assistant reads supplier emails and can send purchase orders; the learner picks the control that still holds when an email contains hidden instructions.
- starting sources
- NIST AI 100-2 E2025, Adversarial Machine Learning taxonomy (24 Mar 2025)
- Carlini et al. 2021, Extracting Training Data from Large Language Models (USENIX Security 2021)
- Greshake et al. 2023, indirect prompt injection (arXiv 2302.12173)
- Carlini et al. 2023, Poisoning Web-Scale Training Datasets is Practical (arXiv 2302.10149)
- id
- 400.6
- slug
- evaluations-that-hold-up
- title
- Evaluations that hold up
- strand
- verifying
- minutes
- 50
- anchor cases
- imagenet-v2-2019
- objectives
- id
- 400.6-a
- text
- Design an evaluation for a Big Widget task with held-out data that matches real use, a baseline, a metric tied to the harm, and a sample size planned in advance.
- misconception
- A public leaderboard score tells you how a model will do on your task.
- quiz item
- The learner picks the test set that best predicts performance on next quarter's warranty claims.
- id
- 400.6-b
- text
- Spot test-set contamination and test-set reuse, and explain why scores can drop on fresh data drawn the same way.
- misconception
- If the model never saw the test file, the score is clean.
- quiz item
- Given the ImageNet v2 result, the learner explains what the drop does and does not show.
- id
- 400.6-c
- text
- Report an evaluation result with its uncertainty, say when a difference between two models is too small to act on, and check an LLM judge against people before trusting it.
- misconception
- A two-point gain on a benchmark is a real improvement, and an LLM judge agrees with people.
- quiz item
- Two items: (1) a paired table of the same 200 items scored by two models, with the prepared interval for the difference; the learner chooses the honest report. (2) An LLM judge's verdicts beside people's on 50 items; the learner reads the agreement table and decides whether the judge may replace people.
- starting sources
- Recht et al. 2019, Do ImageNet Classifiers Generalize to ImageNet? (ICML 2019)
- Sainz et al. 2023, NLP Evaluation in trouble (Findings of EMNLP 2023)
- Miller 2024, Adding Error Bars to Evals (arXiv 2411.00640, preprint)
- Zheng et al. 2023, Judging LLM-as-a-Judge (NeurIPS 2023 Datasets and Benchmarks)
- Liang et al. 2022, HELM (TMLR 2023)
- id
- 400.7
- slug
- testing-for-fairness-and-proxies
- title
- Testing for fairness and proxies
- strand
- verifying
- minutes
- 45
- anchor cases
- toeslagenaffaire-2020
- objectives
- id
- 400.7-a
- text
- Measure a model's error and outcome rates by cohort, and show how one aggregate number can hide harm to a group.
- misconception
- High overall accuracy means the model is fair to everyone.
- quiz item
- A small made-up cohort table for a supplier-risk model giving, per supplier group, how many suppliers were in fact low-risk and how many of those were flagged; the learner computes each group's false-positive rate (flagged low-risk suppliers divided by all low-risk suppliers in the group) and spots the group the overall rate hides.
- id
- 400.7-b
- text
- Tell direct use of a protected attribute apart from use of a proxy (a feature that stands in for it), and explain why deleting the protected column does not remove a proxy's effect.
- misconception
- If nationality is not a column, the model cannot discriminate by nationality.
- quiz item
- Two parts: the Dutch DPA case is labelled as direct use (nationality itself was an indicator); then, in a made-up Big Widget supplier dataset with the nationality column removed, the learner finds the feature that correlates with it (such as the country of a supplier's bank).
- id
- 400.7-c
- text
- Explain, from Kleinberg, Mullainathan and Raghavan's result on risk scores, why calibration within groups, balance for the positive class and balance for the negative class cannot all hold at once except in highly constrained special cases (such as perfect prediction or equal base rates), so the team must choose, write down why, and name who signed it.
- misconception
- There is one fairness test, and a model either passes it or fails it.
- quiz item
- With risk scores for two groups at different base rates, the learner picks which of the three Kleinberg conditions a described score meets and which it must give up, and says when the conflict disappears. Chouldechova's thresholded result is taught separately, not keyed in this item.
- starting sources
- Kleinberg, Mullainathan and Raghavan 2016, Inherent Trade-Offs in the Fair Determination of Risk Scores (ITCS 2017)
- Chouldechova 2017, Fair prediction with disparate impact (arXiv 1703.00056)
- NIST SP 1270 (2022), Towards a Standard for Identifying and Managing Bias in AI
- Autoriteit Persoonsgegevens, fine on the Belastingdienst for discriminatory processing of nationality, 7 Dec 2021
- Barocas, Hardt and Narayanan 2023, Fairness and Machine Learning (MIT Press; fairmlbook.org)
- id
- 400.8
- slug
- records-a-stranger-can-verify
- title
- Records a stranger can verify
- strand
- governing
- minutes
- 50
- reuses
- Practice 400 Prove it worksheet, step 4 (evidence to the regulator), and the Beacon evidence bundle
- anchor cases
- post-office-horizon
- objectives
- id
- 400.8-a
- text
- Explain, using the Horizon judgments, why a system's output must not be taken as reliable evidence against a person just because a computer produced it.
- misconception
- Computer records are accurate unless someone proves otherwise.
- quiz item
- The learner picks which Horizon finding answers the claim "the system said the cash was short, so it was".
- id
- 400.8-b
- text
- Design a tamper-evident record (canonical serialisation, a hash chain, a signature) and state what it proves only when checked against pinned trust material and an expected checkpoint received through a channel independent of the bundle, and what it never proves on its own (including anything that happened before the record was signed).
- misconception
- A signed record proves that what it says is true, and a chain that verifies is complete.
- quiz item
- A chain verifies internally; the learner sorts claims into proved (unchanged since signing, by the holder of the pinned key, up to the expected checkpoint) and not proved (nothing was cut after the checkpoint without the checkpoint; whose key it is without pinned key material; that the decision was right).
- id
- 400.8-c
- text
- List what an evidence bundle must contain for an auditor who has never seen the system (chain, policy and its hash, framework crosswalk, VERIFY.md) and what it must never contain (prompts, outputs, personal data).
- misconception
- More evidence is always better, so include the prompts and outputs.
- quiz item
- The learner removes the two items from a draft bundle that would make it fail.
- starting sources
- Bates v Post Office Ltd (No 6) "Horizon Issues" [2019] EWHC 3408 (QB), 16 Dec 2019
- Hamilton v Post Office Ltd [2021] EWCA Crim 577, 23 Apr 2021
- Crosby and Wallach 2009, Efficient Data Structures for Tamper-Evident Logging (USENIX Security 2009)
- RFC 8785 (JCS, 2020), RFC 8032 (EdDSA, 2017), RFC 9162 (Certificate Transparency 2.0, 2021)
- Regulation (EU) 2024/1689, Article 12 (record-keeping)
- AiGovOps Beacon: evidence bundle with VERIFY.md (https://aigovops-foundation.github.io/aigovops-beacon/)
- id
- 400.9
- slug
- the-audit-gate
- title
- The audit gate: prove it to a stranger
- strand
- governing
- minutes
- 60
- reuses
- Practice 400 Prove it worksheet (Robodebt, toeslagenaffaire, Horizon); the audit-training quiz; the Beacon evidence bundle
- anchor cases
- robodebt-2023
- toeslagenaffaire-2020
- post-office-horizon
- objectives
- id
- 400.9-a
- text
- Explain, using the Robodebt Royal Commission, why the premise of an automated decision (its formula, threshold and data) must be independently validated as lawful and sound before the first decision is issued.
- misconception
- If the formula is simple arithmetic, it does not need validating.
- quiz item
- The learner names who should validate a new warranty-clawback formula, and what evidence they need before the first notice.
- id
- 400.9-b
- text
- Write the premise gate and the protected-attribute gate as policy (an independent validator distinct from the Level 300 holder, what evidence, deny unless a receipt exists) and as two sentences a minister could read.
- misconception
- The team that built the rule can approve it.
- quiz item
- Given a gate where the approver is the rule's author, the learner fixes the approver role and the deny condition.
- id
- 400.9-c
- text
- Verify a stranger's evidence bundle offline by its VERIFY.md against pinned trust material (the expected public key) and an expected checkpoint (the last sequence number and hash, and the policy version in scope), and write a one-sentence verification statement with your name, the date and what you found.
- misconception
- Verifying evidence needs the vendor's tool, an account or a network, and a bundle whose chain verifies is complete.
- quiz item
- Given a known-good fixture and a truncated fixture that both verify internally, the learner uses the expected checkpoint (received separately from the bundle) to reject the truncated one; then, for a valid bundle that carries an unauthenticated Level 300 receipt, the learner writes a statement bounded to what was verified and spots the one that overclaims. Interpreting assurance levels is left to 500.6.
- starting sources
- Report of the Royal Commission into the Robodebt Scheme, 7 Jul 2023 (updated 11 Jul 2023)
- Tweede Kamer, Parlementaire ondervragingscommissie Kinderopvangtoeslag, Ongekend onrecht, 17 Dec 2020 (Kamerstuk 35 510, nr. 2)
- Autoriteit Persoonsgegevens, fine on the Belastingdienst, 7 Dec 2021
- Hamilton v Post Office Ltd [2021] EWCA Crim 577, 23 Apr 2021
- Practice Level 400 Prove it worksheet: https://practice.aigovops-foundation.com/levels/400-prove/WORKSHEET.html
- AiGovOps Foundation, How do I create an AI audit plan? (audit-training quiz: https://www.aigovops-foundation.com/audit.html)
- id
- 400.R
- slug
- engineering-ai-in-your-role
- title
- Engineering AI in your role (role tracks)
- strand
- building
- minutes
- 45
- tracks
- ml-engineering
- data-science
- data-engineering
- mlops-and-platform
- security-engineering
- evaluation-and-quality
- manufacturing-and-quality-inspection
- maintenance-and-reliability
- forecasting-and-supply-chain
- service-and-warranty
- engineering-and-simulation
- objectives
- id
- 400.R-a
- text
- For one model in your role at Big Widget Company, name the evaluation that would show it works, the monitoring that would show it still works, and the evidence a stranger would need.
- misconception
- Evidence is the governance team's job, not the engineer's.
- quiz item
- For a visual-inspection model on the paint line, the learner picks the evaluation, the monitor and the receipt.
- id
- 400.R-b
- text
- Map your role's work to one cert-prep rail and the course objectives it draws on, and explain what a certification does and does not prove.
- misconception
- A certification proves the system I built is safe.
- quiz item
- The learner separates what a passed exam shows about a person from what an evidence bundle shows about a system.
- starting sources
- Reuses the Level 400 case bank (levels/400/cases.yml) and the module source packs: each track row names the case and module it draws on
- NIST AI RMF 1.0 (2023)
- id
- 400.P
- slug
- prove-it-to-a-stranger
- title
- Project: prove it to a stranger
- strand
- project
- minutes
- 120
- time note
- The 120 minutes are the learner's build time in the sandbox. Marking by two 400s or a host is extra and not counted here. Hosting a Thursday call is optional, not a badge condition (Bob, 18 Nov 2026, decision 9). Timings approved as drafted (decision 8).
- reuses
- Practice 400 Prove it worksheet (60 minutes, plus the Thursday you host)
- fixtures
- Synthetic fixtures only, owned by this course repo; no private-repo excerpts (Bob, 18 Nov 2026, decision 10). The 400.L sandbox, model and dataset; the 400.4 made-up monitoring record; the learner's Level 300 hold card; an offline verifier and a pinned public key for the bundle. Nothing touches production data or a live system.
- pass evidence
- Minimum for every gear: the comparison table with intervals and one cohort row; a release gate naming an independent approver, shown denying one release run whose evidence is missing; one dry rollback that restores the previous model, data and configuration versions; a monitoring plan built on the 300 hold card; and a bundle, with carried 300 items provenance-labelled, that a stranger verified against a key and checkpoint received separately. Plain-gear learners use prepared sandbox states and outputs instead of running code.
- objectives
- id
- 400.P-a
- text
- Using the sandbox fixtures, adapt a model for one Big Widget task, evaluate it against a baseline with intervals and a cohort check, then exercise its release gate (one run denied for missing evidence) and one dry rollback, and write the monitoring plan that would govern it.
- misconception
- A model is done when its benchmark score is good.
- quiz item
- Project rubric, not a quiz item; the module quiz checks the rubric's terms (baseline, interval, cohort, gate, rollback).
- id
- 400.P-b
- text
- Produce an evidence bundle, with carried Level 300 items provenance-labelled, that a stranger verifies offline by its VERIFY.md against a key and checkpoint received separately, and states in one signed sentence bounded to what was verified; two 400s or a host mark it. Hosting a Thursday is optional, not a badge condition.
- misconception
- Evidence counts only if the auditor trusts the team that made it.
- quiz item
- Project rubric; the module quiz checks what the stranger's statement may and may not claim.
- starting sources
- Practice Level 400 Prove it worksheet: https://practice.aigovops-foundation.com/levels/400-prove/WORKSHEET.html
- AiGovOps Beacon: https://aigovops-foundation.github.io/aigovops-beacon/