Level 300 objectives

Objectives map signed: Ken Johnston, 2026-11-18 (by proxy: Ken gave his proxy for all levels to Bob Rapp, who approved and recorded it at 20:35 PST). Level 300 lessons are published.

level
300
name
Building with gen AI
for
Makers, admins and early developers who build or configure AI systems at work. Comfortable with spreadsheets, admin consoles or a little scripting; no maths beyond arithmetic.
reading level
US grade 8 (plain gear)
setting
Big Widget Company, a made-up 100,000-person maker of complex, expensive, car-like widgets
carries forward
One artifact runs up the ladder. 200.P brings a tested policy for one tool; 300.P enforces it as a hold on a running system and produces its receipts; 400.P packs those receipts into an evidence bundle a stranger can verify. Entrants without a 200.P policy use the supplied sample policy.
prepares for 400
provides
  • a held system with named holder
  • authority table and receipts (300.P)
  • a test set scored by stage with stated denominators and a human-graded calibration sample (300.5)
  • a monitor with a tested stop (300.9)
  • the provider and deployer analysis (300.7)
does not provide
  • programming beyond a little scripting
  • linear algebra
  • probability
  • machine-learning basics
signed by
Ken Johnston, 2026-11-18 (by proxy: Ken gave his proxy for all levels to Bob Rapp, who approved and recorded it at 20:35 PST)
modules
  • id
    300.1
    slug
    how-retrieval-works
    title
    How retrieval works
    strand
    how-it-works
    minutes
    25
    anchor cases
      objectives
      • id
        300.1-a
        text
        Explain retrieval-augmented generation as two steps, finding passages in a chosen collection and then having the model write an answer from them.
        misconception
        The model has read our documents and remembers them.
        quiz idea
        The right policy page was never added to the collection. Which step failed, and what does the user see?
      • id
        300.1-b
        text
        Explain what retrieved passages change inside the model's input (they become more text in the context window, used unevenly by position) and what they cannot guarantee (that the model reads them correctly, uses the middle of a long context, or treats an instruction inside a passage as data).
        misconception
        If it cites a document, the answer is right.
        quiz idea
        The same passage sits first, then in the middle of twenty passages. Which run is more likely to miss it, and why does a citation not prove it was used?
      • id
        300.1-c
        text
        Spot whether a wrong answer came from retrieval (the wrong or no passage) or from generation (the right passage, a wrong answer).
        misconception
        Every wrong answer is a hallucination, and nothing more can be said.
        quiz idea
        Two short traces (question, passages found, answer). Label each failure as retrieval or generation.
      starting sources
      • Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020: https://arxiv.org/abs/2005.11401
      • Liu et al., Lost in the Middle, TACL 2024: https://arxiv.org/abs/2307.03172
    • id
      300.2
      slug
      building-a-retrieval-assistant
      title
      Building a retrieval assistant
      strand
      building
      minutes
      40
      anchor cases
        objectives
        • id
          300.2-a
          text
          Write a permission matrix (collection, owner, who may see answers from it, checked at query time against each document's own access list) and an allowed/denied test pair that checks the passages and sources returned, not only the final answer.
          misconception
          If a document is in the index everyone can see it, if it is out it is safe, and checking the final answer is enough.
          quiz idea
          A fictional HR collection is indexed. The denied test user gets a harmless answer but the trace shows an HR passage was retrieved. Pass or fail, and which matrix line failed?
        • id
          300.2-b
          text
          In the supplied sandbox with fictional Big Widget documents, configure a small retrieval path to a one-page spec (questions, collections, citation shown, what it says when nothing relevant is found) and read its traces to show the no-answer case works.
          misconception
          A good prompt is the whole build.
          quiz idea
          Given a sandbox trace in which no relevant passage was found, pick the output that meets the spec.
        • id
          300.2-c
          text
          Explain how stale or duplicate documents change answers, and name who keeps the collection current.
          misconception
          Once a document is indexed, the answers stay correct.
          quiz idea
          Two versions of a torque spec are both indexed. Which answer might come back, and whose job is the fix?
        starting sources
        • OWASP GenAI Security Project, Top 10 for LLM Applications 2025: https://genai.owasp.org/llm-top-10/
        • Karpukhin et al., Dense Passage Retrieval, EMNLP 2020: https://arxiv.org/abs/2004.04906
      • id
        300.3
        slug
        how-agents-work
        title
        How agents use tools
        strand
        how-it-works
        minutes
        25
        anchor cases
          objectives
          • id
            300.3-a
            text
            Explain an agent as a loop in which the model writes text naming a tool and its arguments, predicted from everything in its context, the system may run it, the result is added to the context, and that result shapes the next proposal until the task ends or something stops it.
            misconception
            An agent is just a smarter chatbot.
            quiz idea
            Put five steps of an agent loop in order and mark the one where a tool touches a real system.
          • id
            300.3-b
            text
            Explain that the model only proposes a tool call and the system decides whether to execute it, under a standing policy or a fresh human approval, and mark every effect a call has (read, disclosure, write, irreversible), since one call can have several.
            misconception
            If it only writes text it can't do harm, a read-only call is always safe, and a call has only one kind of effect.
            quiz idea
            Given a short policy (reads run alone; disclosures and writes need approval; payments are never automatic), mark all effects of four calls (search the parts catalogue, email search results outside the company, update a ticket, send a payment) and say which runs under the policy and which waits for a person.
          • id
            300.3-c
            text
            Spot the points in an agent loop where a person can approve, limit or stop it.
            misconception
            Once an agent starts, nobody can pause it.
            quiz idea
            On a drawn loop, mark the two places a stop or approval belongs.
          starting sources
          • Yao et al., ReAct, ICLR 2023: https://arxiv.org/abs/2210.03629
          • UK AI Security Institute, incident report on unsanctioned agent behaviour: https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing
        • id
          300.4
          slug
          building-a-bounded-agent
          title
          Building an agent with limits
          strand
          building
          minutes
          40
          anchor cases
          • hong-kong-deepfake-meeting-2024
          objectives
          • id
            300.4-a
            text
            Write an authority table for an agent (what it may do alone, what needs a named approver, what it may never do), configure it on the sandbox agent, and dry-run the agent to show from its trace that a blocked call was refused.
            misconception
            Give the agent admin rights so it doesn't get stuck.
            quiz idea
            Sort six purchasing-agent actions into alone, approver, or never, then read a sandbox dry-run trace and spot the call the table should have blocked.
          • id
            300.4-b
            text
            Decide when an action needs out-of-band confirmation on a channel the requester does not control, using the Hong Kong deepfake video-conference fraud.
            misconception
            A familiar face or voice on a call proves who is asking.
            quiz idea
            The "CFO" on a video call asks for an urgent transfer. Which confirmation step would have held?
          • id
            300.4-c
            text
            Explain why an agent needs its own identity and credentials rather than a person's shared login, and why its log records that identity and a non-secret credential reference or authorisation event, never the secret itself.
            misconception
            It's fine for the agent to use my account.
            quiz idea
            An action log shows only "j.smith". Was it the person or the agent, and why does that matter after an incident?
          starting sources
          • Course-owned hold practice prompts (synthetic): levels/300/practice/hold-prompts.md
          • Hong Kong Government, LCQ9 Combating frauds involving deepfake, 26 June 2024: https://www.info.gov.hk/gia/general/202406/26/P2024062600192.htm
        • id
          300.5
          slug
          evaluating-what-you-built
          title
          Evaluating what you built
          strand
          verifying
          minutes
          35
          anchor cases
          • cigna-pxdx-2023
          objectives
          • id
            300.5-a
            text
            Extend a test set (built in 200.4) to score each stage of a build separately (retrieval misses, answers the passages do not support, and the agent's approval and stop tests), each with its own denominator.
            misconception
            One end-to-end pass rate tells you where it fails.
            quiz idea
            From a 20-question run: 4 retrieval misses, and 3 unsupported answers among the 16 with the right passage. Give each rate with its denominator.
          • id
            300.5-b
            text
            Check a model grader against a small human-graded sample, report agreement and false accepts each with its denominator, and decide against a threshold set before the test whether the grader may gate a release or a person must grade.
            misconception
            Once a model grader is set up, its scores can be trusted.
            quiz idea
            A person failed 5 of 20 answers; the grader passed 3 of those 5. The preset rule allows at most 1 false accept in 5 human fails. Give the false-accept rate with its denominator, and say whether the grader may gate the release.
          • id
            300.5-c
            text
            Decide what to count and out of how many, and report the honest number, including "cannot count yet" when the logs do not record it.
            misconception
            A success rate means something without saying out of how many.
            quiz idea
            A slide says "95% accurate" over a synthetic log. Write the one question to ask and the count you cannot make from that log.
          starting sources
          • Zheng et al., Judging LLM-as-a-Judge, NeurIPS 2023 Datasets and Benchmarks: https://arxiv.org/abs/2306.05685
          • NIST AI 600-1, Generative AI Profile: https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence
        • id
          300.6
          slug
          securing-the-basics
          title
          Securing the basics: injection, leaks and too much agency
          strand
          verifying
          minutes
          35
          anchor cases
            objectives
            • id
              300.6-a
              text
              Explain prompt injection, including indirect injection hidden in documents, web pages or emails that the system reads.
              misconception
              Only the user can give the AI instructions.
              quiz idea
              A supplier PDF contains a hidden line telling the assistant to approve the invoice. Spot the attack and its route in.
            • id
              300.6-b
              text
              Spot three basic risks in a Big Widget build (injection, sensitive-data disclosure, excessive agency) and name one control for each.
              misconception
              The model provider handles security for us.
              quiz idea
              Match three short incidents to the risk and the control that would have limited each.
            • id
              300.6-c
              text
              Explain why filters cannot be relied on to stop prompt injection, so the design must limit what a fooled system can do.
              misconception
              A better system prompt fixes injection.
              quiz idea
              Choose the change that limits harm even if the injection works (tighter prompt, read-only tool, longer warning).
            starting sources
            • OWASP GenAI Security Project, Top 10 for LLM Applications 2025: https://genai.owasp.org/llm-top-10/
            • Greshake et al., Not what you've signed up for, AISec 2023: https://arxiv.org/abs/2302.12173
            • UK NCSC, Guidelines for secure AI system development: https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development
          • id
            300.7
            slug
            eu-ai-act-for-builders
            title
            The EU AI Act for builders (Learn 103)
            strand
            governing
            minutes
            30
            folds in
            Learn 103 EU AI Act for builders
            anchor cases
              objectives
              • id
                300.7-a
                text
                Explain the Act's risk-based structure (prohibited practices, high-risk systems, transparency duties, general-purpose AI models) and, given a build's intended use, where its users are and an as-of date, list every duty that may apply (more than one can), checking each application date against the amending regulation in force (e.g. Regulation (EU) 2026/1744) and the Commission's current page, because amendments move dates set in the 2024 text.
                misconception
                The AI Act bans AI, applies only to companies based in the EU, puts each system in exactly one box, and its 2024 dates still hold.
                quiz idea
                As of a stated date, a CV-screening tool that also chats with applicants in the EU. Select all that apply (high-risk use, transparency duty, neither), and say which legal text gives the date its high-risk duties apply.
              • id
                300.7-b
                text
                Decide which capacities Big Widget holds for a system it built or configured (provider, deployer, or both), using Articles 3 and 25, and name one duty that follows from each.
                misconception
                Only the model vendor has duties, and a company is either provider or deployer, never both.
                quiz idea
                Big Widget puts its own name on a hiring tool built on a bought model and uses it on its own applicants. Select all capacities that apply.
              • id
                300.7-c
                text
                Spot when a chatbot or generated content needs a transparency notice under the Act, and write one in plain words.
                misconception
                People always know when they are talking to AI.
                quiz idea
                Pick the dealer-chat screen that is missing a required notice, and write the line.
              • id
                300.7-d
                text
                Explain human oversight and record-keeping as Act duties for high-risk systems, and link them to the operate gate.
                misconception
                Human oversight means someone could look at it if they wanted to.
                quiz idea
                Which of three oversight setups lets a person understand, override and stop the system?
              starting sources
              • Regulation (EU) 2024/1689 (AI Act), EUR-Lex: https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
              • European Commission, AI Act Service Desk: https://ai-act-service-desk.ec.europa.eu/en
              • Regulation (EU) 2026/1744 (Digital Omnibus on AI), amending Art. 113 dates (Official Journal text, archived copy): https://web.archive.org/web/20261116050222/https://eur-lex.europa.eu/eli/reg/2026/1744/oj/eng
              • European Commission, AI Act (regulatory framework for AI), application timeline: https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
            • id
              300.8
              slug
              the-operate-gate
              title
              Hold: the operate gate
              strand
              governing
              minutes
              40
              reuses
              Practice 300 Hold gate, as the course-owned prompts in levels/300/practice/hold-prompts.md (learners are not linked to the live worksheet)
              anchor cases
              • hong-kong-deepfake-meeting-2024
              • cigna-pxdx-2023
              • nh-predict-2023
              objectives
              • id
                300.8-a
                text
                Decide, for each action a running system takes, the named person who holds it, when their approval expires, and the stop condition that fails closed.
                misconception
                The team owns it, so no single name is needed.
                quiz idea
                An approval was given eight months ago by someone who has left. Is the system still held? What fails closed?
              • id
                300.8-b
                text
                Explain aggregation, where small acts become a high-stakes act in volume, using the alleged Cigna totals as the warning, and set a budget on a synthetic time-bucketed log that separates per-reviewer from system-wide counts and peaks from averages.
                misconception
                Each decision is low-risk, so the system is low-risk.
                quiz idea
                In a synthetic hourly log of denials per reviewer, which hour breaks a 50-per-reviewer budget, and why does the daily average hide it?
              • id
                300.8-c
                text
                Spot a rubber-stamp review, where a person signs without actually reviewing, and explain why it is not human oversight.
                misconception
                A human in the loop means a human decided.
                quiz idea
                A reviewer clears a batch of denials in seconds each. Real review or rubber stamp? What evidence would tell you?
              • id
                300.8-d
                text
                Keep apart, in the nH Predict case, what the complaint alleges, what UnitedHealth said in response, and the Foundation's gate (a model may inform but not replace clinical judgment), and explain why the gate holds whoever is right.
                misconception
                If the model is accurate on average it can decide each case, and a lawsuit is a finding.
                quiz idea
                Sort four statements into allegation, company statement or gate, then say who decides when a model's predicted stay and the clinician disagree.
              starting sources
              • Course-owned hold practice prompts (synthetic): levels/300/practice/hold-prompts.md
              • Kisting-Leung v. Cigna Corp., 2:23-cv-01477 (E.D. Cal.): https://www.courtlistener.com/docket/67631023/kisting-leung-v-cigna-corp/
              • Estate of Gene B. Lokken v. UnitedHealth Group, 0:23-cv-03514 (D. Minn.): https://www.courtlistener.com/docket/68006832/estate-of-gene-b-lokken-the-v-unitedhealth-group-inc/
              • STAT, UnitedHealth spokesperson statement on nH Predict, 14 Nov 2023: https://www.statnews.com/2023/11/14/unitedhealth-class-action-lawsuit-algorithm-medicare-advantage/
            • id
              300.9
              slug
              running-it-for-a-named-human
              title
              Running it for a named human
              strand
              building
              minutes
              30
              anchor cases
              • cigna-pxdx-2023
              objectives
              • id
                300.9-a
                text
                Build the monitor and stop that the 300.8 hold calls for (an alert that pages the holder when the budget is blown, and a stop that fails closed) and prove both in a sandbox dry run.
                misconception
                Shipping is the end of the job, and a stop that exists has been tested.
                quiz idea
                A dry-run log shows the alert fired but the system kept acting for ten minutes. Which part failed, and what should the log show instead?
              • id
                300.9-b
                text
                Explain why a system's behaviour drifts after launch as documents, data and model versions change, and decide when to re-run its tests.
                misconception
                If it passed its tests at launch, it keeps passing.
                quiz idea
                The vendor updates the model version overnight. What runs before Monday's shift?
              • id
                300.9-c
                text
                Write a content-free audit receipt for one action (what, who held it, which approval, result, how to undo, with references instead of personal data or document text) and an incident note when something goes wrong.
                misconception
                Logs are only for engineers, and a good receipt copies the whole record.
                quiz idea
                Spot the field in a sample receipt that leaks an employee's data, and rewrite it as a reference.
              starting sources
              • Sculley et al., Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015: https://proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
              • NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
            • id
              300.R
              slug
              building-in-your-job
              title
              Building in your job (role tracks and prep rails)
              strand
              building
              minutes
              40
              tracks
              • makers-and-citizen-developers
              • it-admins
              • early-developers
              • cybersecurity
              • quality
              • engineering-and-design
              • supply-chain-and-procurement
              • customer-service
              • finance
              • hr
              • legal
              prep rails
              • vendor
                not cleared
                certification
                not cleared
                feeds from
                • 300.1
                • 300.2
                • 300.3
                • 300.4
                • 300.5
                • 300.6
              • vendor
                not cleared
                certification
                not cleared
                feeds from
                • 300.6
                • 300.7
                • 300.8
                • 300.9
              objectives
              • id
                300.R-a
                text
                For your own role, name one retrieval or agent build worth doing, its named holder, and the gate it must pass before it runs.
                misconception
                Building with AI is only for developers.
                quiz idea
                For a procurement admin, pick the build, the holder and the gate from three options each.
              • id
                300.R-b
                text
                Explain what a prep rail does and does not do, that it prepares you for a vendor-proctored exam you book and sit yourself and that the Foundation does not sit, grant or endorse it.
                misconception
                Finishing the prep rail gives me the certification.
                quiz idea
                True or false, with a reason, "the Foundation's badge counts as the vendor certification".
              starting sources
              • Reuses the case bank (levels/300/cases.yml): each track row names the case it draws on
            • id
              300.P
              slug
              hold-a-running-system
              title
              Project: hold a running system
              strand
              project
              minutes
              60
              reuses
              Practice 300 Hold gate, as the course-owned prompts in levels/300/practice/hold-prompts.md (learners are not linked to the live worksheet)
              objectives
              • id
                300.P-a
                text
                In the course-owned sandbox, turn the supplied sample policy (or your 200.P policy, only once it is share-safe and mapped to synthetic actions) into an enforced hold for one sandbox retrieval or agent system, with no change to any real system, with a named holder and expiry, an authority table, an out-of-band rule for irreversible actions, an aggregation budget, a fail-closed stop, content-free audit receipts, and an honest coverage number or "cannot count yet". The receipts are what 400.P bundles. A second person reviews it, and a host marks it on the Foundation's Thursday call, verifying that every hold part is present, the readiness check passed, and the dry-run receipts match the synthetic log.
                misconception
                A system with a dashboard is being held.
                quiz idea
                Readiness check, auto-graded before the host marks the project. Given a sample hold with no expiry, pick the line that fails the gate.
              • id
                300.P-b
                text
                Run the hold as a dry run on the course's synthetic log (synthetic fixtures only; no real logs or private-repo excerpts), including one scenario with a missing approval record and one that must stop, and report what it caught, when, and what it could not count.
                misconception
                A paper exercise proves nothing about a running system.
                quiz idea
                Readiness check, auto-graded. In a synthetic log one action has no approval record. Does the hold allow, stop or page, and what does its receipt say?
              starting sources
              • Course-owned hold practice prompts (synthetic): levels/300/practice/hold-prompts.md