Level 500: Governance, safety and research · Module 500.8

A colleague holds up a red card during a five-person table meeting.
Who owns the stop?

Designing a governance regime · Plain gear

Reviewed and published, version 1.0.0.

The law here is as it stood on 18 November 2026. Big Widget Company, Priya and the hiring tool are made up. The laws, reports and papers are real and cited. Where we have not checked something, we say so, and we say who checks it next. This is not legal advice.

The story

Five audit stages in order: scoping, then mapping, artifact collection, testing and reflection. Reflection is the audit and engineering teams' analysis and mitigation plan, not a route for an affected person to challenge a decision.
The internal audit stages produce documents throughout the development lifecycle. Reflection is the audit and engineering teams' analysis and mitigation plan, not an affected person's challenge route (s2).
Figure data as text

End-to-end audit stages

Five stages in order. [s2]

  1. Scoping. [s2]
  2. Mapping. [s2]
  3. Artifact collection. [s2]
  4. Testing. [s2]
  5. Reflection: the audit and engineering teams’ analysis and mitigation plan, not a route for an affected person to challenge a decision. [s2]

Objective: 500.8-a. Text equivalent of figures/end-to-end-audit-stages.svg.

Priya leads HR governance at Big Widget Company's office in Europe. The company has bought an AI tool. It reads job applications and puts them in order, best match first. People in HR then look at the list.

The buying team sent Priya a note. It said: "We only bought it, so we are just the user. The law's word for that is deployer. Nothing else to do."

Priya is not so sure. This lesson follows her as she answers three questions:

  1. Who decides what, before the tool touches anyone's job chances?
  2. What does the law actually say, and from which date?
  3. If the tool gets it wrong, how does an applicant push back?

Part 1: A rulebook is not the same as a rule

Some companies think they "do AI governance" because they have a document with a nice title. A document cannot say yes or no. People do.

Priya checks four things for every big step the tool is part of. These four checks are this course's idea. No guide below lists them this way.

Think of a school exam. A teacher sets it (decider). You need your student card to sit it (gate). Your paper is kept (record). You can ask for a re-mark, and the mark can go up (push back). Take away the kept paper and a re-mark is impossible. Take away the re-mark and a wrong mark stays wrong.

A US standards body, NIST, has a guide for managing AI risk. Following it is a choice, not the law.1 It does not use our four checks. But some of what it asks for overlaps. It asks that people's jobs in managing AI risk are written down and clear to everyone.1 It asks for ways to hear from people outside the team, and for ways to appeal and override the system after it is in use.1 NIST's helper guide, the Playbook, gives suggested actions. NIST says it is for voluntary use. You can borrow "as many – or as few – suggestions as apply".2

Researchers led by Deborah Raji suggested checking an AI system at every stage of building it, not just at the end.3 Each stage leaves papers behind. Together those papers make the full check.3 Their checks are done by the company's own audit and engineering teams. They are not a way for the people affected to push back.3 That fourth check is something this course adds.

People decide, not the tool

The Foundation's manifesto has a rule: "No direct model-to-tool path for consequential actions."4 Turning someone down for a job is a big, consequential step. So the tool can sort the list, but it can never send a "no" by itself. A named person does that.

The manifesto also gives a gate four answers: "go ahead · only this far · wait for a person · no".4 This course writes them in short as allow, constrain, hold and deny. Each gate in Priya's plan gives one of those answers. If the answer is "wait for a person", the plan names that person. Nobody else can say yes in their place, and the tool never can.

Priya's plan

The buying team is not in it as a decider. Priya is.

Big stepWho decidesGate before itWhat we keepHow people push back
Change the tool's settingsPriya, with the lawyersNotes on who we are under the law, and when the tool went on sale, are up to dateWhat changed, who said yes, the new rules versionAnyone trained to watch it can pause a change
Start using itPriyaNamed staff trained to watch it; applicants told AI is used; an outside expert can see the tool's rules; a tested "stop"The start recordApplicants told how to ask for a person and for an explanation
Turn someone downA named HR person, who makes the call. The tool cannot send a "no"A person read the application, not just the rank, and wrote a reasonWho, what rank, which version, which rules, what reason, when, and where the application is keptThe letter says how to challenge
Answer a challengePriya, plus someone who did not make the first callThe old record and the rules used were found and looked at againWhat was claimed and what changedThe person can be put back in
Stop the toolPriya; any trained watcher can pauseNone, on purpose. Stopping is always allowed, and the plan says soWhy it stoppedRestart only through "start using it"

The last two columns depend on each other. If nothing was kept, nobody can check a challenge.

A record is not a receipt

The "what we keep" record holds applicants' names and papers. So only a few people can open it, and it is not kept forever. Sometimes Big Widget makes a receipt to show a gate was passed. The Foundation's rule is that a receipt carries "hashes, decision, tier, policy hash, key id". It never carries "prompts, outputs, payloads, personal data".5 So no applicant's name or rank ever goes in a receipt.

The three yeses

The Foundation's manifesto names three steps in a tool's life.4

At no step does the tool decide.

Part 2: What the law says, and when

"We only bought it"

The EU's AI Act uses two words you met in Level 300.

Big Widget uses the tool under its own control. So "deployer" fits what we know.6

But buying does not mean only deployer. The law says a user can also count as the provider of a high-risk system. This is about a system that is already on sale or in use. It happens if the user puts its own name on the system, makes a big change to it, or changes what the system is for in a way that makes it high-risk.6 One company can be both.

"Puts its own name on" means doing something. Seeing Big Widget's logo on a screen is a clue, not an answer. Priya asks who put it there, and what the contract says.

Priya's packet does not answer four questions:

  1. Who built the tool? Did Big Widget pay to have it built?
  2. Who put it on sale or into use, under whose name, and when?
  3. Did Big Widget put its own name on it later?
  4. Did Big Widget's settings change what it does, or change it a lot?

So her honest answer is: "Deployer: yes, on what we know. Provider too: we can't tell yet." She writes the questions down. She does not guess.

Is this a high-risk tool?

The Act has a list of high-risk uses. One is AI used to pick or choose people for jobs, including tools that sort job applications and judge candidates.6 There is an exception. A listed tool is not high-risk if it does not pose a big risk of harm to people's health, safety or rights. That includes not really changing the outcome of a decision.6 But a listed tool that builds a profile of a person always counts as high-risk.6

So Priya asks, in order:

  1. Is this tool really for picking people for jobs?
  2. Does it build profiles of people? If yes, it is high-risk.
  3. If not, does its ranking really shape who gets picked? For example, does it decide which applications HR reads first, or at all?

The exception has more parts that we did not read. The lawyers check those.

Which date?

The AI Act was first published in 2024 with its own timetable.6 In July 2026 the EU passed a second law that moved some of those dates.7 For high-risk uses on the list, like this hiring tool, many rules for users and makers now start on 2 December 2027.7 So on 18 November 2026, those rules do not apply yet.7

Two traps:

Was it already in use?

A date tells you the earliest a rule could apply. It does not promise the rule reaches this tool. Some high-risk tools were already on sale or in use before the December 2027 date. For those, the rules apply only if the tool's design changes in a big way on or after that date.7 Tools meant to be used by public bodies have a separate deadline in 2030.7 As far as we know, Big Widget's hiring tool is not meant for public bodies. The lawyers confirm that too.

So Priya adds two more questions:

  1. When did the very first copy of this same tool go on sale or into use anywhere in the EU? The 2026 law's notes say that date is the one that counts.7 The day Big Widget switches it on is not. The same note says at least one copy must have gone on sale or into use lawfully before the start date.7 That copy need not be the first one. So Priya writes down two separate things: the date of the first copy, and proof that some copy went on sale or into use lawfully before the start date. The dates alone do not prove the second. If it is in doubt, the lawyers decide.
  2. What would count as a big change to its design, and who decides?

The lawyers answer these, in writing, before the tool starts. Until then, the gate's answer is "wait for a person".

We opened only some parts of the 2026 law: the dates, this rule about tools already in use, and the note on which date counts. Other changes it makes are not checked here.

Priya's table

What the law saysFrom when (as we read it)GateWhat we keepOwner
Who is a provider or deployer6Definitions already apply;6 the role-change rule from 2 Dec 20277Before using itThe open questions and the answersPriya, with the lawyers
Tools already in use only count after a big design change, made on or after the start date7Depends on when the first copy of the same tool went on sale or into use in the EU, and on changes on or after 2 Dec 20277Before using it, and at every big changeWhen the first copy went on sale or into use; separately, proof that a copy went on sale or into use lawfully before the start date; whether Big Widget's is the same tool; what changedThe lawyers, with Priya
Give the job of watching to trained people with power to act62 Dec 2027 at the earliest7Before using itNames and trainingPriya
Keep the logs the tool makes by itself, where Big Widget controls them, for at least six months. Other laws, like privacy law, can change this62 Dec 2027 at the earliest7Before using itLog settings, checked by the privacy ownerPriya, with IT
Tell people a high-risk AI system is used on them62 Dec 2027 at the earliest7Before using itThe noticePriya
Some people turned down can ask for a clear explanation of the AI's part. It covers decisions based on a listed high-risk tool that seriously affect them in a way they feel harms their health, safety or rights. Other laws can limit it6Not sure. We have not checked when, or whether, this reaches a tool like this.Before turning anyone downA record that can explainPriya, with the lawyers
Big Widget's own rule: anyone turned down can ask for a plain-words explanationNowBefore turning anyone downThe explanation sentPriya

The last row is a company rule. It is wider than the law. Priya keeps the two apart, so nobody thinks the company rule is the law.

Priya also decides to do the 2027 things now anyway, as company rules. She writes "company rule, not yet the law" next to them, so nobody mixes them up.

Every "not sure" in the table has a name next to it. The Foundation's Practice site says "I don't know yet" "is always an answer, and it always pairs you with someone."5 Here, that someone is Priya or the lawyers.

What about ISO?

There is an international standard, ISO/IEC 42001, about running an AI management system.9 We read only the shop page that describes it. We did not read the standard itself. So Priya's table names no ISO rule and claims no certificate. If someone shows her "ISO says X" without the real text, she asks for the real text.

Part 3: Plan the push-back first

"We'll check it later"

Some people think a check after launch will catch any harm. Two real government failures show why that is not enough. Neither one was a modern chatbot.

Australia: the Robodebt report

An official inquiry in Australia, called a Royal Commission, looked into a scheme called Robodebt. Its report came out on 7 July 2023.10 Among its advice:

This is advice. The report page we read does not tell us whether it has been done. That is not checked.

All four need something built before the first decision. You cannot review a decision that left no record. You cannot read rules nobody wrote down.

Some easy answers only go part of the way. Telling people "we use AI" is a start, but it does not say how the tool decides. A list of setting changes helps, but it may not show the real rules. So Priya keeps a rules pack: the settings, scores and maker's papers that actually drive the ranking. An outside expert can look at it, under written terms. Each "no" record says which rules pack was used. If the maker will not show its rules, Priya writes that down as a gap, with an owner.

The Netherlands: Ongekend onrecht

A Dutch parliamentary inquiry looked into how childcare benefits were run. Its report, from 17 December 2020, is called Ongekend onrecht, "unprecedented injustice".11 It found that basic principles of the rule of law were broken.11 It also described information to Parliament that was late, incomplete or wrong, and poor handling of information inside ministries.11 The report says some questions, about risk profiles based on nationality, were outside its job and were looked at by a different body.11

What Priya takes from both

The first three rows are things the Robodebt report asks for. They are advice for the future. We read only the advice, so we do not say what was missing in Robodebt itself. The last row is different: the Dutch report found it.

What the report saysWhat Priya builds before day one
Asks for: a clear way to get a review10The challenge route in her plan, with someone who can change the answer
Asks for: plain-language information10The notice to applicants, plus an explanation on request. The notice alone is only a start
Asks for: rules that outside experts can check10The rules pack, open to an outside expert. A list of changes alone is not enough
Found: information that was late, incomplete or wrong11A record that links each decision to who made it and why

If nothing was kept, a check after launch has little to work with. If there was no way to change a decision, people already turned down may never get one. Other papers or people's memories might fill some gaps, but nobody can count on that. That is why Raji's team put checks at every stage,3 and why NIST asks for appeal and override.1

We do not claim Priya's plan would have changed what happened in Australia or the Netherlands. The reports do not say that, and neither do we.

Remember

Sources

  1. s1 Elham Tabassi (National Institute of Standards and Technology) (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. Standard. Checked 2026-11-19 by source-checker role, separate Claude session (round-2 re-check, not the writer), at 4b113a9; nvlpubs PDF of NIST AI 100-1. ↩ ↩ ↩ ↩ ↩
  2. s5 National Institute of Standards and Technology (2026). NIST AI RMF Playbook. Official doc. Checked 2026-11-19 by source-checker role, separate Claude session (round-2 re-check, not the writer), at 4b113a9; cited page re-opened. ↩
  3. s2 Raji, Inioluwa Deborah; Smart, Andrew; White, Rebecca N.; Mitchell, Margaret; Gebru, Timnit; Hutchinson, Ben; Smith-Loud, Jamila; Theron, Daniel; Barnes, Parker (2020). Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing. Paper. Checked 2026-11-19 by source-checker role, separate Claude session (round-2 re-check, not the writer), at 4b113a9; arXiv PDF 2001.00973v1. ↩ ↩ ↩ ↩
  4. s10 AiGovOps Foundation (2026). The AiGovOps Practitioner Manifesto. Official doc. Checked 2026-11-20 by source-checker role, separate Claude session (round-4 re-check, not the writer), at a3afc63; live page, all seven quoted strings present verbatim. ↩ ↩ ↩ ↩ ↩ ↩
  5. s11 AiGovOps Foundation (2026). The AiGovOps Foundation café — start with your problem (Practice home page). Official doc. Checked 2026-11-19 by source-checker role, separate Claude session (round-2 re-check, not the writer), at 4b113a9; live page, Receipt row and "I don't know yet" line verbatim. ↩ ↩ ↩
  6. s7 European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act), original text as published (Internet Archive copy). Standard. Checked 2026-11-20 by source-checker role, separate Claude session (round-6 re-check, not the writer), at 465498c; cited archive copy (id_ raw replay) and the read_via Cellar DOC_1, both OJ text. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  7. s9 European Parliament and Council of the European Union (2026). Regulation (EU) 2026/1744 of 8 July 2026, amending Regulation (EU) 2024/1689 (Internet Archive copy). Standard. Checked 2026-11-20 by source-checker role, separate Claude session (round-10 re-check, not the writer), at 0e752ce; archive id_ copy and Cellar DOC_1 of 2026/1744, Art. 111(2) and recital 39 verbatim. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  8. s3 European Commission, Directorate-General for Communications Networks, Content and Technology (2026). AI Act (European Commission policy page, regulatory framework for AI). Official doc. Checked 2026-11-18 by source-checker role, separate Claude session aa749102 (not the writer), at e9574a1. ↩ ↩
  9. s8 International Organization for Standardization and International Electrotechnical Commission (2023). ISO/IEC 42001:2023, Information technology, Artificial intelligence, Management system (ISO catalogue page, Internet Archive copy). Standard. Checked 2026-11-19 by source-checker role, separate Claude session (round-2 re-check, not the writer), at 4b113a9; cited archive copy (id_ raw replay); catalogue page only. ↩
  10. s6 Royal Commission into the Robodebt Scheme (2023). Report of the Royal Commission into the Robodebt Scheme (report page, Internet Archive copy). Record. Checked 2026-11-19 by source-checker role, separate Claude session (round-3 re-check, not the writer), at a15b9b5; cited archive copy (id_ raw replay), report page with Rec. 17.1 and 17.2. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  11. s4 Parlementaire ondervragingscommissie Kinderopvangtoeslag, Tweede Kamer der Staten-Generaal (2020). Ongekend onrecht (final report of the parliamentary inquiry committee on the childcare benefit). Record. Checked 2026-11-18 by source-checker role, separate Claude session aa749102 (not the writer), at e9574a1. ↩ ↩ ↩ ↩ ↩ ↩

Next: the quiz