Level 300 objectives
Objectives map signed: Ken Johnston, 2026-11-18 (by proxy: Ken gave his proxy for all levels to Bob Rapp, who approved and recorded it at 20:35 PST). Level 300 lessons are published.
- level
- 300
- name
- Building with gen AI
- for
- Makers, admins and early developers who build or configure AI systems at work. Comfortable with spreadsheets, admin consoles or a little scripting; no maths beyond arithmetic.
- reading level
- US grade 8 (plain gear)
- setting
- Big Widget Company, a made-up 100,000-person maker of complex, expensive, car-like widgets
- carries forward
- One artifact runs up the ladder. 200.P brings a tested policy for one tool; 300.P enforces it as a hold on a running system and produces its receipts; 400.P packs those receipts into an evidence bundle a stranger can verify. Entrants without a 200.P policy use the supplied sample policy.
- prepares for 400
- provides
- a held system with named holder
- authority table and receipts (300.P)
- a test set scored by stage with stated denominators and a human-graded calibration sample (300.5)
- a monitor with a tested stop (300.9)
- the provider and deployer analysis (300.7)
- does not provide
- programming beyond a little scripting
- linear algebra
- probability
- machine-learning basics
- signed by
- Ken Johnston, 2026-11-18 (by proxy: Ken gave his proxy for all levels to Bob Rapp, who approved and recorded it at 20:35 PST)
- modules
- id
- 300.1
- slug
- how-retrieval-works
- title
- How retrieval works
- strand
- how-it-works
- minutes
- 25
- anchor cases
- objectives
- id
- 300.1-a
- text
- Explain retrieval-augmented generation as two steps, finding passages in a chosen collection and then having the model write an answer from them.
- misconception
- The model has read our documents and remembers them.
- quiz idea
- The right policy page was never added to the collection. Which step failed, and what does the user see?
- id
- 300.1-b
- text
- Explain what retrieved passages change inside the model's input (they become more text in the context window, used unevenly by position) and what they cannot guarantee (that the model reads them correctly, uses the middle of a long context, or treats an instruction inside a passage as data).
- misconception
- If it cites a document, the answer is right.
- quiz idea
- The same passage sits first, then in the middle of twenty passages. Which run is more likely to miss it, and why does a citation not prove it was used?
- id
- 300.1-c
- text
- Spot whether a wrong answer came from retrieval (the wrong or no passage) or from generation (the right passage, a wrong answer).
- misconception
- Every wrong answer is a hallucination, and nothing more can be said.
- quiz idea
- Two short traces (question, passages found, answer). Label each failure as retrieval or generation.
- starting sources
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020: https://arxiv.org/abs/2005.11401
- Liu et al., Lost in the Middle, TACL 2024: https://arxiv.org/abs/2307.03172
- id
- 300.2
- slug
- building-a-retrieval-assistant
- title
- Building a retrieval assistant
- strand
- building
- minutes
- 40
- anchor cases
- objectives
- id
- 300.2-a
- text
- Write a permission matrix (collection, owner, who may see answers from it, checked at query time against each document's own access list) and an allowed/denied test pair that checks the passages and sources returned, not only the final answer.
- misconception
- If a document is in the index everyone can see it, if it is out it is safe, and checking the final answer is enough.
- quiz idea
- A fictional HR collection is indexed. The denied test user gets a harmless answer but the trace shows an HR passage was retrieved. Pass or fail, and which matrix line failed?
- id
- 300.2-b
- text
- In the supplied sandbox with fictional Big Widget documents, configure a small retrieval path to a one-page spec (questions, collections, citation shown, what it says when nothing relevant is found) and read its traces to show the no-answer case works.
- misconception
- A good prompt is the whole build.
- quiz idea
- Given a sandbox trace in which no relevant passage was found, pick the output that meets the spec.
- id
- 300.2-c
- text
- Explain how stale or duplicate documents change answers, and name who keeps the collection current.
- misconception
- Once a document is indexed, the answers stay correct.
- quiz idea
- Two versions of a torque spec are both indexed. Which answer might come back, and whose job is the fix?
- starting sources
- OWASP GenAI Security Project, Top 10 for LLM Applications 2025: https://genai.owasp.org/llm-top-10/
- Karpukhin et al., Dense Passage Retrieval, EMNLP 2020: https://arxiv.org/abs/2004.04906
- id
- 300.3
- slug
- how-agents-work
- title
- How agents use tools
- strand
- how-it-works
- minutes
- 25
- anchor cases
- objectives
- id
- 300.3-a
- text
- Explain an agent as a loop in which the model writes text naming a tool and its arguments, predicted from everything in its context, the system may run it, the result is added to the context, and that result shapes the next proposal until the task ends or something stops it.
- misconception
- An agent is just a smarter chatbot.
- quiz idea
- Put five steps of an agent loop in order and mark the one where a tool touches a real system.
- id
- 300.3-b
- text
- Explain that the model only proposes a tool call and the system decides whether to execute it, under a standing policy or a fresh human approval, and mark every effect a call has (read, disclosure, write, irreversible), since one call can have several.
- misconception
- If it only writes text it can't do harm, a read-only call is always safe, and a call has only one kind of effect.
- quiz idea
- Given a short policy (reads run alone; disclosures and writes need approval; payments are never automatic), mark all effects of four calls (search the parts catalogue, email search results outside the company, update a ticket, send a payment) and say which runs under the policy and which waits for a person.
- id
- 300.3-c
- text
- Spot the points in an agent loop where a person can approve, limit or stop it.
- misconception
- Once an agent starts, nobody can pause it.
- quiz idea
- On a drawn loop, mark the two places a stop or approval belongs.
- starting sources
- Yao et al., ReAct, ICLR 2023: https://arxiv.org/abs/2210.03629
- UK AI Security Institute, incident report on unsanctioned agent behaviour: https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing
- id
- 300.4
- slug
- building-a-bounded-agent
- title
- Building an agent with limits
- strand
- building
- minutes
- 40
- anchor cases
- hong-kong-deepfake-meeting-2024
- objectives
- id
- 300.4-a
- text
- Write an authority table for an agent (what it may do alone, what needs a named approver, what it may never do), configure it on the sandbox agent, and dry-run the agent to show from its trace that a blocked call was refused.
- misconception
- Give the agent admin rights so it doesn't get stuck.
- quiz idea
- Sort six purchasing-agent actions into alone, approver, or never, then read a sandbox dry-run trace and spot the call the table should have blocked.
- id
- 300.4-b
- text
- Decide when an action needs out-of-band confirmation on a channel the requester does not control, using the Hong Kong deepfake video-conference fraud.
- misconception
- A familiar face or voice on a call proves who is asking.
- quiz idea
- The "CFO" on a video call asks for an urgent transfer. Which confirmation step would have held?
- id
- 300.4-c
- text
- Explain why an agent needs its own identity and credentials rather than a person's shared login, and why its log records that identity and a non-secret credential reference or authorisation event, never the secret itself.
- misconception
- It's fine for the agent to use my account.
- quiz idea
- An action log shows only "j.smith". Was it the person or the agent, and why does that matter after an incident?
- starting sources
- Course-owned hold practice prompts (synthetic): levels/300/practice/hold-prompts.md
- Hong Kong Government, LCQ9 Combating frauds involving deepfake, 26 June 2024: https://www.info.gov.hk/gia/general/202406/26/P2024062600192.htm
- id
- 300.5
- slug
- evaluating-what-you-built
- title
- Evaluating what you built
- strand
- verifying
- minutes
- 35
- anchor cases
- cigna-pxdx-2023
- objectives
- id
- 300.5-a
- text
- Extend a test set (built in 200.4) to score each stage of a build separately (retrieval misses, answers the passages do not support, and the agent's approval and stop tests), each with its own denominator.
- misconception
- One end-to-end pass rate tells you where it fails.
- quiz idea
- From a 20-question run: 4 retrieval misses, and 3 unsupported answers among the 16 with the right passage. Give each rate with its denominator.
- id
- 300.5-b
- text
- Check a model grader against a small human-graded sample, report agreement and false accepts each with its denominator, and decide against a threshold set before the test whether the grader may gate a release or a person must grade.
- misconception
- Once a model grader is set up, its scores can be trusted.
- quiz idea
- A person failed 5 of 20 answers; the grader passed 3 of those 5. The preset rule allows at most 1 false accept in 5 human fails. Give the false-accept rate with its denominator, and say whether the grader may gate the release.
- id
- 300.5-c
- text
- Decide what to count and out of how many, and report the honest number, including "cannot count yet" when the logs do not record it.
- misconception
- A success rate means something without saying out of how many.
- quiz idea
- A slide says "95% accurate" over a synthetic log. Write the one question to ask and the count you cannot make from that log.
- starting sources
- Zheng et al., Judging LLM-as-a-Judge, NeurIPS 2023 Datasets and Benchmarks: https://arxiv.org/abs/2306.05685
- NIST AI 600-1, Generative AI Profile: https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence
- id
- 300.6
- slug
- securing-the-basics
- title
- Securing the basics: injection, leaks and too much agency
- strand
- verifying
- minutes
- 35
- anchor cases
- objectives
- id
- 300.6-a
- text
- Explain prompt injection, including indirect injection hidden in documents, web pages or emails that the system reads.
- misconception
- Only the user can give the AI instructions.
- quiz idea
- A supplier PDF contains a hidden line telling the assistant to approve the invoice. Spot the attack and its route in.
- id
- 300.6-b
- text
- Spot three basic risks in a Big Widget build (injection, sensitive-data disclosure, excessive agency) and name one control for each.
- misconception
- The model provider handles security for us.
- quiz idea
- Match three short incidents to the risk and the control that would have limited each.
- id
- 300.6-c
- text
- Explain why filters cannot be relied on to stop prompt injection, so the design must limit what a fooled system can do.
- misconception
- A better system prompt fixes injection.
- quiz idea
- Choose the change that limits harm even if the injection works (tighter prompt, read-only tool, longer warning).
- starting sources
- OWASP GenAI Security Project, Top 10 for LLM Applications 2025: https://genai.owasp.org/llm-top-10/
- Greshake et al., Not what you've signed up for, AISec 2023: https://arxiv.org/abs/2302.12173
- UK NCSC, Guidelines for secure AI system development: https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development
- id
- 300.7
- slug
- eu-ai-act-for-builders
- title
- The EU AI Act for builders (Learn 103)
- strand
- governing
- minutes
- 30
- folds in
- Learn 103 EU AI Act for builders
- anchor cases
- objectives
- id
- 300.7-a
- text
- Explain the Act's risk-based structure (prohibited practices, high-risk systems, transparency duties, general-purpose AI models) and, given a build's intended use, where its users are and an as-of date, list every duty that may apply (more than one can), checking each application date against the amending regulation in force (e.g. Regulation (EU) 2026/1744) and the Commission's current page, because amendments move dates set in the 2024 text.
- misconception
- The AI Act bans AI, applies only to companies based in the EU, puts each system in exactly one box, and its 2024 dates still hold.
- quiz idea
- As of a stated date, a CV-screening tool that also chats with applicants in the EU. Select all that apply (high-risk use, transparency duty, neither), and say which legal text gives the date its high-risk duties apply.
- id
- 300.7-b
- text
- Decide which capacities Big Widget holds for a system it built or configured (provider, deployer, or both), using Articles 3 and 25, and name one duty that follows from each.
- misconception
- Only the model vendor has duties, and a company is either provider or deployer, never both.
- quiz idea
- Big Widget puts its own name on a hiring tool built on a bought model and uses it on its own applicants. Select all capacities that apply.
- id
- 300.7-c
- text
- Spot when a chatbot or generated content needs a transparency notice under the Act, and write one in plain words.
- misconception
- People always know when they are talking to AI.
- quiz idea
- Pick the dealer-chat screen that is missing a required notice, and write the line.
- id
- 300.7-d
- text
- Explain human oversight and record-keeping as Act duties for high-risk systems, and link them to the operate gate.
- misconception
- Human oversight means someone could look at it if they wanted to.
- quiz idea
- Which of three oversight setups lets a person understand, override and stop the system?
- starting sources
- Regulation (EU) 2024/1689 (AI Act), EUR-Lex: https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
- European Commission, AI Act Service Desk: https://ai-act-service-desk.ec.europa.eu/en
- Regulation (EU) 2026/1744 (Digital Omnibus on AI), amending Art. 113 dates (Official Journal text, archived copy): https://web.archive.org/web/20261116050222/https://eur-lex.europa.eu/eli/reg/2026/1744/oj/eng
- European Commission, AI Act (regulatory framework for AI), application timeline: https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
- id
- 300.8
- slug
- the-operate-gate
- title
- Hold: the operate gate
- strand
- governing
- minutes
- 40
- reuses
- Practice 300 Hold gate, as the course-owned prompts in levels/300/practice/hold-prompts.md (learners are not linked to the live worksheet)
- anchor cases
- hong-kong-deepfake-meeting-2024
- cigna-pxdx-2023
- nh-predict-2023
- objectives
- id
- 300.8-a
- text
- Decide, for each action a running system takes, the named person who holds it, when their approval expires, and the stop condition that fails closed.
- misconception
- The team owns it, so no single name is needed.
- quiz idea
- An approval was given eight months ago by someone who has left. Is the system still held? What fails closed?
- id
- 300.8-b
- text
- Explain aggregation, where small acts become a high-stakes act in volume, using the alleged Cigna totals as the warning, and set a budget on a synthetic time-bucketed log that separates per-reviewer from system-wide counts and peaks from averages.
- misconception
- Each decision is low-risk, so the system is low-risk.
- quiz idea
- In a synthetic hourly log of denials per reviewer, which hour breaks a 50-per-reviewer budget, and why does the daily average hide it?
- id
- 300.8-c
- text
- Spot a rubber-stamp review, where a person signs without actually reviewing, and explain why it is not human oversight.
- misconception
- A human in the loop means a human decided.
- quiz idea
- A reviewer clears a batch of denials in seconds each. Real review or rubber stamp? What evidence would tell you?
- id
- 300.8-d
- text
- Keep apart, in the nH Predict case, what the complaint alleges, what UnitedHealth said in response, and the Foundation's gate (a model may inform but not replace clinical judgment), and explain why the gate holds whoever is right.
- misconception
- If the model is accurate on average it can decide each case, and a lawsuit is a finding.
- quiz idea
- Sort four statements into allegation, company statement or gate, then say who decides when a model's predicted stay and the clinician disagree.
- starting sources
- Course-owned hold practice prompts (synthetic): levels/300/practice/hold-prompts.md
- Kisting-Leung v. Cigna Corp., 2:23-cv-01477 (E.D. Cal.): https://www.courtlistener.com/docket/67631023/kisting-leung-v-cigna-corp/
- Estate of Gene B. Lokken v. UnitedHealth Group, 0:23-cv-03514 (D. Minn.): https://www.courtlistener.com/docket/68006832/estate-of-gene-b-lokken-the-v-unitedhealth-group-inc/
- STAT, UnitedHealth spokesperson statement on nH Predict, 14 Nov 2023: https://www.statnews.com/2023/11/14/unitedhealth-class-action-lawsuit-algorithm-medicare-advantage/
- id
- 300.9
- slug
- running-it-for-a-named-human
- title
- Running it for a named human
- strand
- building
- minutes
- 30
- anchor cases
- cigna-pxdx-2023
- objectives
- id
- 300.9-a
- text
- Build the monitor and stop that the 300.8 hold calls for (an alert that pages the holder when the budget is blown, and a stop that fails closed) and prove both in a sandbox dry run.
- misconception
- Shipping is the end of the job, and a stop that exists has been tested.
- quiz idea
- A dry-run log shows the alert fired but the system kept acting for ten minutes. Which part failed, and what should the log show instead?
- id
- 300.9-b
- text
- Explain why a system's behaviour drifts after launch as documents, data and model versions change, and decide when to re-run its tests.
- misconception
- If it passed its tests at launch, it keeps passing.
- quiz idea
- The vendor updates the model version overnight. What runs before Monday's shift?
- id
- 300.9-c
- text
- Write a content-free audit receipt for one action (what, who held it, which approval, result, how to undo, with references instead of personal data or document text) and an incident note when something goes wrong.
- misconception
- Logs are only for engineers, and a good receipt copies the whole record.
- quiz idea
- Spot the field in a sample receipt that leaks an employee's data, and rewrite it as a reference.
- starting sources
- Sculley et al., Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015: https://proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
- NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
- id
- 300.R
- slug
- building-in-your-job
- title
- Building in your job (role tracks and prep rails)
- strand
- building
- minutes
- 40
- tracks
- makers-and-citizen-developers
- it-admins
- early-developers
- cybersecurity
- quality
- engineering-and-design
- supply-chain-and-procurement
- customer-service
- finance
- hr
- legal
- prep rails
- vendor
- not cleared
- certification
- not cleared
- feeds from
- 300.1
- 300.2
- 300.3
- 300.4
- 300.5
- 300.6
- vendor
- not cleared
- certification
- not cleared
- feeds from
- 300.6
- 300.7
- 300.8
- 300.9
- objectives
- id
- 300.R-a
- text
- For your own role, name one retrieval or agent build worth doing, its named holder, and the gate it must pass before it runs.
- misconception
- Building with AI is only for developers.
- quiz idea
- For a procurement admin, pick the build, the holder and the gate from three options each.
- id
- 300.R-b
- text
- Explain what a prep rail does and does not do, that it prepares you for a vendor-proctored exam you book and sit yourself and that the Foundation does not sit, grant or endorse it.
- misconception
- Finishing the prep rail gives me the certification.
- quiz idea
- True or false, with a reason, "the Foundation's badge counts as the vendor certification".
- starting sources
- Reuses the case bank (levels/300/cases.yml): each track row names the case it draws on
- id
- 300.P
- slug
- hold-a-running-system
- title
- Project: hold a running system
- strand
- project
- minutes
- 60
- reuses
- Practice 300 Hold gate, as the course-owned prompts in levels/300/practice/hold-prompts.md (learners are not linked to the live worksheet)
- objectives
- id
- 300.P-a
- text
- In the course-owned sandbox, turn the supplied sample policy (or your 200.P policy, only once it is share-safe and mapped to synthetic actions) into an enforced hold for one sandbox retrieval or agent system, with no change to any real system, with a named holder and expiry, an authority table, an out-of-band rule for irreversible actions, an aggregation budget, a fail-closed stop, content-free audit receipts, and an honest coverage number or "cannot count yet". The receipts are what 400.P bundles. A second person reviews it, and a host marks it on the Foundation's Thursday call, verifying that every hold part is present, the readiness check passed, and the dry-run receipts match the synthetic log.
- misconception
- A system with a dashboard is being held.
- quiz idea
- Readiness check, auto-graded before the host marks the project. Given a sample hold with no expiry, pick the line that fails the gate.
- id
- 300.P-b
- text
- Run the hold as a dry run on the course's synthetic log (synthetic fixtures only; no real logs or private-repo excerpts), including one scenario with a missing approval record and one that must stop, and report what it caught, when, and what it could not count.
- misconception
- A paper exercise proves nothing about a running system.
- quiz idea
- Readiness check, auto-graded. In a synthetic log one action has no approval record. Does the hold allow, stop or page, and what does its receipt say?
- starting sources
- Course-owned hold practice prompts (synthetic): levels/300/practice/hold-prompts.md