
Designing a governance regime · Plain gear
Reviewed and published, version 1.0.0.
The law here is as it stood on 18 November 2026. Big Widget Company, Priya and the hiring tool are made up. The laws, reports and papers are real and cited. Where we have not checked something, we say so, and we say who checks it next. This is not legal advice.
The story
Figure data as text
End-to-end audit stages
Five stages in order. [s2]
- Scoping. [s2]
- Mapping. [s2]
- Artifact collection. [s2]
- Testing. [s2]
- Reflection: the audit and engineering teams’ analysis and mitigation plan, not a route for an affected person to challenge a decision. [s2]
Objective: 500.8-a. Text equivalent of figures/end-to-end-audit-stages.svg.
Priya leads HR governance at Big Widget Company's office in Europe. The company has bought an AI tool. It reads job applications and puts them in order, best match first. People in HR then look at the list.
The buying team sent Priya a note. It said: "We only bought it, so we are just the user. The law's word for that is deployer. Nothing else to do."
Priya is not so sure. This lesson follows her as she answers three questions:
- Who decides what, before the tool touches anyone's job chances?
- What does the law actually say, and from which date?
- If the tool gets it wrong, how does an applicant push back?
Part 1: A rulebook is not the same as a rule
Some companies think they "do AI governance" because they have a document with a nice title. A document cannot say yes or no. People do.
Priya checks four things for every big step the tool is part of. These four checks are this course's idea. No guide below lists them this way.
- Who decides? One named person or role, not "the team". And that person really makes the call.
- What is the gate? What must be true, and checked, before the step happens.
- What do we keep? A record good enough for someone else to see what happened and why.
- How do people push back? A real way to complain, with someone who can change the answer.
Think of a school exam. A teacher sets it (decider). You need your student card to sit it (gate). Your paper is kept (record). You can ask for a re-mark, and the mark can go up (push back). Take away the kept paper and a re-mark is impossible. Take away the re-mark and a wrong mark stays wrong.
A US standards body, NIST, has a guide for managing AI risk. Following it is a choice, not the law.1 It does not use our four checks. But some of what it asks for overlaps. It asks that people's jobs in managing AI risk are written down and clear to everyone.1 It asks for ways to hear from people outside the team, and for ways to appeal and override the system after it is in use.1 NIST's helper guide, the Playbook, gives suggested actions. NIST says it is for voluntary use. You can borrow "as many – or as few – suggestions as apply".2
Researchers led by Deborah Raji suggested checking an AI system at every stage of building it, not just at the end.3 Each stage leaves papers behind. Together those papers make the full check.3 Their checks are done by the company's own audit and engineering teams. They are not a way for the people affected to push back.3 That fourth check is something this course adds.
People decide, not the tool
The Foundation's manifesto has a rule: "No direct model-to-tool path for consequential actions."4 Turning someone down for a job is a big, consequential step. So the tool can sort the list, but it can never send a "no" by itself. A named person does that.
The manifesto also gives a gate four answers: "go ahead · only this far · wait for a person · no".4 This course writes them in short as allow, constrain, hold and deny. Each gate in Priya's plan gives one of those answers. If the answer is "wait for a person", the plan names that person. Nobody else can say yes in their place, and the tool never can.
Priya's plan
The buying team is not in it as a decider. Priya is.
| Big step | Who decides | Gate before it | What we keep | How people push back |
|---|---|---|---|---|
| Change the tool's settings | Priya, with the lawyers | Notes on who we are under the law, and when the tool went on sale, are up to date | What changed, who said yes, the new rules version | Anyone trained to watch it can pause a change |
| Start using it | Priya | Named staff trained to watch it; applicants told AI is used; an outside expert can see the tool's rules; a tested "stop" | The start record | Applicants told how to ask for a person and for an explanation |
| Turn someone down | A named HR person, who makes the call. The tool cannot send a "no" | A person read the application, not just the rank, and wrote a reason | Who, what rank, which version, which rules, what reason, when, and where the application is kept | The letter says how to challenge |
| Answer a challenge | Priya, plus someone who did not make the first call | The old record and the rules used were found and looked at again | What was claimed and what changed | The person can be put back in |
| Stop the tool | Priya; any trained watcher can pause | None, on purpose. Stopping is always allowed, and the plan says so | Why it stopped | Restart only through "start using it" |
The last two columns depend on each other. If nothing was kept, nobody can check a challenge.
A record is not a receipt
The "what we keep" record holds applicants' names and papers. So only a few people can open it, and it is not kept forever. Sometimes Big Widget makes a receipt to show a gate was passed. The Foundation's rule is that a receipt carries "hashes, decision, tier, policy hash, key id". It never carries "prompts, outputs, payloads, personal data".5 So no applicant's name or rank ever goes in a receipt.
The three yeses
The Foundation's manifesto names three steps in a tool's life.4
- Get to Yes, "before a single action runs".4 The "start using it" gate. The yes covers this tool, this job and these settings. Nothing more.
- Stay at Yes, "while it runs, in real time".4 Trained staff watch it. Priya reads the challenges each month. She counts how many "no" letters went through a person, out of how many were sent.
- Recover to Yes, "after — on demand, on incident".4 If something goes wrong, Priya stops the tool and finds out why. Stopping is not the same as fixing. The tool is back at yes only when it passes the "start using it" gate again.
At no step does the tool decide.
Part 2: What the law says, and when
"We only bought it"
The EU's AI Act uses two words you met in Level 300.
- A provider makes an AI system, or has it made. Then it puts the system on sale, or into use, under its own name.6 Both parts count.
- A deployer uses an AI system under its own control at work.6
Big Widget uses the tool under its own control. So "deployer" fits what we know.6
But buying does not mean only deployer. The law says a user can also count as the provider of a high-risk system. This is about a system that is already on sale or in use. It happens if the user puts its own name on the system, makes a big change to it, or changes what the system is for in a way that makes it high-risk.6 One company can be both.
"Puts its own name on" means doing something. Seeing Big Widget's logo on a screen is a clue, not an answer. Priya asks who put it there, and what the contract says.
Priya's packet does not answer four questions:
- Who built the tool? Did Big Widget pay to have it built?
- Who put it on sale or into use, under whose name, and when?
- Did Big Widget put its own name on it later?
- Did Big Widget's settings change what it does, or change it a lot?
So her honest answer is: "Deployer: yes, on what we know. Provider too: we can't tell yet." She writes the questions down. She does not guess.
Is this a high-risk tool?
The Act has a list of high-risk uses. One is AI used to pick or choose people for jobs, including tools that sort job applications and judge candidates.6 There is an exception. A listed tool is not high-risk if it does not pose a big risk of harm to people's health, safety or rights. That includes not really changing the outcome of a decision.6 But a listed tool that builds a profile of a person always counts as high-risk.6
So Priya asks, in order:
- Is this tool really for picking people for jobs?
- Does it build profiles of people? If yes, it is high-risk.
- If not, does its ranking really shape who gets picked? For example, does it decide which applications HR reads first, or at all?
The exception has more parts that we did not read. The lawyers check those.
Which date?
The AI Act was first published in 2024 with its own timetable.6 In July 2026 the EU passed a second law that moved some of those dates.7 For high-risk uses on the list, like this hiring tool, many rules for users and makers now start on 2 December 2027.7 So on 18 November 2026, those rules do not apply yet.7
Two traps:
- Not everything moved. The new law moved some dates, not all of them.7
- A summary is not the law. The European Commission has a helpful web page about the Act. It is a summary. It is not the law itself.8
Was it already in use?
A date tells you the earliest a rule could apply. It does not promise the rule reaches this tool. Some high-risk tools were already on sale or in use before the December 2027 date. For those, the rules apply only if the tool's design changes in a big way on or after that date.7 Tools meant to be used by public bodies have a separate deadline in 2030.7 As far as we know, Big Widget's hiring tool is not meant for public bodies. The lawyers confirm that too.
So Priya adds two more questions:
- When did the very first copy of this same tool go on sale or into use anywhere in the EU? The 2026 law's notes say that date is the one that counts.7 The day Big Widget switches it on is not. The same note says at least one copy must have gone on sale or into use lawfully before the start date.7 That copy need not be the first one. So Priya writes down two separate things: the date of the first copy, and proof that some copy went on sale or into use lawfully before the start date. The dates alone do not prove the second. If it is in doubt, the lawyers decide.
- What would count as a big change to its design, and who decides?
The lawyers answer these, in writing, before the tool starts. Until then, the gate's answer is "wait for a person".
We opened only some parts of the 2026 law: the dates, this rule about tools already in use, and the note on which date counts. Other changes it makes are not checked here.
Priya's table
| What the law says | From when (as we read it) | Gate | What we keep | Owner |
|---|---|---|---|---|
| Who is a provider or deployer6 | Definitions already apply;6 the role-change rule from 2 Dec 20277 | Before using it | The open questions and the answers | Priya, with the lawyers |
| Tools already in use only count after a big design change, made on or after the start date7 | Depends on when the first copy of the same tool went on sale or into use in the EU, and on changes on or after 2 Dec 20277 | Before using it, and at every big change | When the first copy went on sale or into use; separately, proof that a copy went on sale or into use lawfully before the start date; whether Big Widget's is the same tool; what changed | The lawyers, with Priya |
| Give the job of watching to trained people with power to act6 | 2 Dec 2027 at the earliest7 | Before using it | Names and training | Priya |
| Keep the logs the tool makes by itself, where Big Widget controls them, for at least six months. Other laws, like privacy law, can change this6 | 2 Dec 2027 at the earliest7 | Before using it | Log settings, checked by the privacy owner | Priya, with IT |
| Tell people a high-risk AI system is used on them6 | 2 Dec 2027 at the earliest7 | Before using it | The notice | Priya |
| Some people turned down can ask for a clear explanation of the AI's part. It covers decisions based on a listed high-risk tool that seriously affect them in a way they feel harms their health, safety or rights. Other laws can limit it6 | Not sure. We have not checked when, or whether, this reaches a tool like this. | Before turning anyone down | A record that can explain | Priya, with the lawyers |
| Big Widget's own rule: anyone turned down can ask for a plain-words explanation | Now | Before turning anyone down | The explanation sent | Priya |
The last row is a company rule. It is wider than the law. Priya keeps the two apart, so nobody thinks the company rule is the law.
Priya also decides to do the 2027 things now anyway, as company rules. She writes "company rule, not yet the law" next to them, so nobody mixes them up.
Every "not sure" in the table has a name next to it. The Foundation's Practice site says "I don't know yet" "is always an answer, and it always pairs you with someone."5 Here, that someone is Priya or the lawyers.
What about ISO?
There is an international standard, ISO/IEC 42001, about running an AI management system.9 We read only the shop page that describes it. We did not read the standard itself. So Priya's table names no ISO rule and claims no certificate. If someone shows her "ISO says X" without the real text, she asks for the real text.
Part 3: Plan the push-back first
"We'll check it later"
Some people think a check after launch will catch any harm. Two real government failures show why that is not enough. Neither one was a modern chatbot.
Australia: the Robodebt report
An official inquiry in Australia, called a Royal Commission, looked into a scheme called Robodebt. Its report came out on 7 July 2023.10 Among its advice:
- People need a clear way to get an automatic decision reviewed.10
- People need plain-language information about automatic decisions.10
- Outside experts need access to the rules and the computer logic, to check them.10
- The government should consider a body to watch and check automatic decisions for fairness, bias and ease of use.10
This is advice. The report page we read does not tell us whether it has been done. That is not checked.
All four need something built before the first decision. You cannot review a decision that left no record. You cannot read rules nobody wrote down.
Some easy answers only go part of the way. Telling people "we use AI" is a start, but it does not say how the tool decides. A list of setting changes helps, but it may not show the real rules. So Priya keeps a rules pack: the settings, scores and maker's papers that actually drive the ranking. An outside expert can look at it, under written terms. Each "no" record says which rules pack was used. If the maker will not show its rules, Priya writes that down as a gap, with an owner.
The Netherlands: Ongekend onrecht
A Dutch parliamentary inquiry looked into how childcare benefits were run. Its report, from 17 December 2020, is called Ongekend onrecht, "unprecedented injustice".11 It found that basic principles of the rule of law were broken.11 It also described information to Parliament that was late, incomplete or wrong, and poor handling of information inside ministries.11 The report says some questions, about risk profiles based on nationality, were outside its job and were looked at by a different body.11
What Priya takes from both
The first three rows are things the Robodebt report asks for. They are advice for the future. We read only the advice, so we do not say what was missing in Robodebt itself. The last row is different: the Dutch report found it.
| What the report says | What Priya builds before day one |
|---|---|
| Asks for: a clear way to get a review10 | The challenge route in her plan, with someone who can change the answer |
| Asks for: plain-language information10 | The notice to applicants, plus an explanation on request. The notice alone is only a start |
| Asks for: rules that outside experts can check10 | The rules pack, open to an outside expert. A list of changes alone is not enough |
| Found: information that was late, incomplete or wrong11 | A record that links each decision to who made it and why |
If nothing was kept, a check after launch has little to work with. If there was no way to change a decision, people already turned down may never get one. Other papers or people's memories might fill some gaps, but nobody can count on that. That is why Raji's team put checks at every stage,3 and why NIST asks for appeal and override.1
We do not claim Priya's plan would have changed what happened in Australia or the Netherlands. The reports do not say that, and neither do we.
Remember
- Every big step needs four parts: who decides, the gate, what is kept, and how people push back. That is this course's check. NIST asks for some of the same things, like clear jobs and ways to appeal.1
- "We only bought it" is not the end. Check the facts, and write down what you don't know.6
- Read the date from the law and its change, not from an old timetable or a summary.7 8 Then ask: was the tool already in use, and has it changed?7
- Build the push-back route and the record first.10 11
- A receipt never holds anyone's personal details.5