Level 400: Engineering AI systems · Module 400.9

At a café counter, a reviewer reads a sheet from a plain folder while a colleague waits.
Who checked the premise?

The audit gate: prove it to a stranger · Plain gear

As of 20 November 2026. Reviewed and published, version 1.0.0. People, not computers, signed the reviews. Our source checker read the Robodebt report again and found our two wording fixes right. The practice part now uses the course's sandbox. Big Widget Company and its people are made up. The real cases are real, and each one is cited. If we do not know something, we say so, and we say who checks next.

Could a stranger check it?

Premise gate (Get to Yes): an independent validator checks the legal basis, a statistical validation and an equity test by cohort. Protected-attribute gate (Get to Yes, then Stay at Yes): protected attributes and stand-ins are barred, and a person reviews each adverse decision. Stranger's check (Recover to Yes): someone outside the team verifies the evidence offline, with a key and checkpoint got separately.
The course's Level 400 worksheet draws the three gates and the premise evidence (s5). Anchor cases are in the lesson, not drawn here.
Figure data as text

Three gates on the three yeses

WhereWhat it shows
GET TO YES [s5]Premise gate. An independent validator checks the legal basis, a statistical validation and an equity test by cohort.
GET TO YES, THEN STAY [s5]Protected-attribute gate. Protected attributes and stand-ins barred; a person reviews each adverse decision.
RECOVER TO YES [s5]Stranger's check. Someone outside the team verifies the evidence offline, with a key and checkpoint got separately.

At Level 300 you named a person who holds a system and can stop it. You kept a receipt: a short record of what the system did, with no private details in it.

This lesson asks a harder question. Imagine a stranger. They have never met you. They do not trust you. Could they check that your system should have run at all? Could they check that your records are whole?

We answer with three gates. Each comes from a real case that went badly wrong. The course's Level 400 worksheet uses the same three cases.1 Each gate fits one of the three yeses: Get to Yes (before it runs), Stay at Yes (while it runs) and Recover to Yes (after, when someone asks what happened).

  1. The premise gate (Get to Yes). Before a computer rule makes its first decision about people, someone independent checks that the rule is legal and sound. (Case: Robodebt, in Australia.2)
  2. The protected-attribute gate (Get to Yes, then Stay at Yes). The rule may not judge people by things like nationality, or by anything that stands in for them. A person checks each decision that goes against someone. (Case: the childcare-benefit scandal in the Netherlands.34)
  3. The stranger's check (Recover to Yes). Someone outside the team can check the records on their own, with no internet and no help from you. (Case: the Post Office Horizon scandal in the UK.56)

The worksheet is the course's own exercise. This lesson does not approve or certify any system. Doing the exercise shows what you can do. It does not show that any system is safe.

The premise gate

A simple rule with a hidden guess

Big Widget Company pays its dealers to fix broken widgets under warranty. Dana runs the numbers. Dana suggests a rule to get back money the company may have overpaid:

Look at each dealer's warranty claims for the past year and work out a normal month. If any month is much higher than normal, treat the extra as overcharged. Send the dealer a letter asking for it back.

It is just an average. What could go wrong?

A lot. The sums are fine. The guess behind the sums is the problem. The rule guesses that a busy month means the dealer cheated. But widgets may break more in winter. A small dealer may fix three widgets one month and nine the next, all honestly. The rule also guesses that Big Widget's contracts let it demand money on this evidence alone. Nobody has checked. That hidden guess is called the premise. (Dana, the rule and the numbers are made up. They are a comparison, not a copy of Robodebt.)

Robodebt

Australia had a real case like this. A government scheme, later called Robodebt, raised debts using income averaging.2 A Royal Commission looked into it. Its head, Catherine Holmes, handed over the report on 7 July 2023.2 It made 57 recommendations.2

The report said: "Robodebt was a crude and cruel mechanism, neither fair nor legal, and it made many people feel like criminals."2

Here is the part that matters for this gate. The report says that in late 2014, before the scheme started, one department already had advice. Its lawyers warned that averaging might not fit the law, and its policy staff said it did not. The scheme went ahead anyway, without the change to the law that the department had said was needed.2 Later, in mid-2018, one department passed another a tribunal's decision that income averaging was unlawful.2

So asking the question was not enough. The warning was there, and nothing made it stop the scheme. A real gate does three things. Someone gives advice. Someone independent decides yes or no. And a "no" actually blocks the letters.

We do not explain exactly how Robodebt's averaging worked, or how many people it hit. Those details are not checked yet, so we leave them out until the source checker confirms them.

Horizon

In the UK, the Post Office used a computer system called Horizon to keep the books in local post offices.6 When Horizon showed money missing, the Post Office prosecuted sub-postmasters for dishonesty.6 A court later looked at 42 of these cases, from 2003 to 2013.6 In the cases it called "Horizon cases", the court said there was "no independent evidence of an actual shortfall".5 The whole case rested on Horizon's numbers being right.5 In 2019, another court had already found that an earlier version, called Legacy Horizon, "was not robust".7 It judged later versions differently, so this finding is about that version, not every version.7

The premise was "the computer is right". Nobody independent had checked it.

What the gate asks for

Think of a new bridge. The builder does not get to sign off their own bridge. An inspector who did not build it checks it before anyone drives across.

The premise gate works the same way. The course's worksheet asks for three kinds of proof before the first letter goes out:1

And the person who checks must be independent: "not the team that wrote the rule".1 Dana cannot approve Dana's own rule.

Writing the two gates

Gate 1: no checked premise, no letter

The worksheet sums the premise gate up in six words: "no validated premise, no notice — ever".1 Written out as a rule for Big Widget, it says:

Before the rule is switched on: wait for a person. An independent checker looks at the legal answer, the test results and the fairness check. If all three pass, the checker says "go ahead" and signs a receipt for this exact version of the rule.

Before any letter goes out: the answer is no, unless there is a signed receipt that says "go ahead" for the version of the rule being used now.

That second line matters most. A receipt that says "wait" or "no" does not count. A receipt for last month's version does not count. If the right receipt is missing, nothing goes out. The safe answer is the default. (The worksheet's own line only asks whether a receipt exists. The course makes it stricter, for the reasons above.)

A gate has four answers: go ahead, only this far, wait for a person, or no. This gate uses three. The rule starts at "wait for a person". Ines turns that into "go ahead" or "no". Letters stay at "no" until a "go ahead" receipt exists. And a "wait" cannot last forever: in the course's design, if nobody decides in time, it turns into "no", never into "go ahead".

Who checks? Not Dana, who wrote the rule. And not Marcus, who runs the system day to day and can stop it (the Level 300 holder). Marcus's job is to keep it running well. The checker's job is to ask whether it should run at all. So Big Widget picks Ines, from a team outside warranty. Only Ines can say "go ahead". A computer helper can gather the papers for her. It never signs.

What goes in the receipt? Only short facts: which rule version, what Ines decided, that each check passed, and a code that points to the full papers. The legal advice and the dealer records stay in a separate, locked file. The receipt never holds private details.

Spot the mistake. A draft rule says "Dana approves the rule, and letters go out unless someone objects." Two fixes: an independent checker approves, and letters stay blocked until there is a "go ahead" receipt for the current rule.

Gate 2: no judging people by who they are

In the Netherlands, a committee of the Dutch parliament looked into how childcare benefits were run. In December 2020 it published a report called Ongekend onrecht, which means "unprecedented injustice".3 It found that basic principles of the rule of law had been broken.3 Its report says it did not look at the use of nationality. The Dutch data protection authority did.3

In November 2021 the data protection authority fined the tax office EUR 2.75 million in total.4 It said the tax office had used whether applicants were Dutch or not Dutch as a sign of risk in a system.4 It called this unlawful and discriminatory.4

So the second gate bars things like nationality from the rule. But that is not enough on its own. Some facts can quietly stand in for nationality, like a postcode or the language someone speaks. These are called proxies. The course's worksheet says both are barred: "protected attributes and their proxies".1 It also says a person must properly review every decision that goes against someone.1 (Lesson 400.7 shows how to test for proxies.)

The course adds one more step. The rule can only be switched on if the proxy test was run and passed, and a receipt says so. A test that found a proxy is a "no", not a tick in a box.

After the rule is on, the review never stops. Each letter waits for a named person, Priya in dealer relations, who reads it and can stop it. "Someone checks" is not enough. Say who.

Two sentences a minister could read

If a busy minister or a reporter cannot understand your gate, it is not finished. Here are the two gates in plain words:

  1. "No letter asking a dealer for money goes out until someone who did not write the rule, and does not run it, has checked this exact version and written down that it is legal, that it gets the right answer, and that it is not wrong more often for some dealers than others."
  2. "The rule may not use nationality, race or anything shown to stand in for them, and a named person checks every letter before it is sent."

The stranger's check

Why a stranger

In the Horizon cases, the evidence was claimed, not shown. The court found the Post Office kept saying Horizon was "robust and reliable".5 People "were prosecuted, convicted and sentenced on the basis that the Horizon data must be correct".5 In all 39 Horizon cases in that appeal, the court found the convictions unsafe.6

So the last gate is this: anyone who relies on your records must be able to check them without trusting you. That is Recover to Yes. When someone asks what happened, you hand them records they can test themselves.

The evidence bundle

Big Widget's records come as a bundle: one package of files. The Foundation's Beacon tool makes bundles like this. It puts in the receipts, the rules, the public keys, some checklists, and a one-page instruction sheet called VERIFY.md.8 Its page says: "Anyone with openssl can re-check it. No vendor login. No phoning home."8

The worksheet's exercise is to check a bundle you did not make.1 You put it on a computer with no internet, "airplane mode counts".1 Then you follow VERIFY.md step by step, "without Beacon, without an account, without us".1 You tick three boxes: Is the signature good? Is the chain of receipts unbroken? Is there any private content inside? If private content is inside, the bundle fails.1

Two things the bundle cannot prove about itself

The worksheet does not cover these two. They are the course's own additions.

Whose seal is it? A digital signature is like a wax seal. Checking it needs the seal-maker's public key.9 But the bundle brings its own keys.8 Checking a seal against a key that came in the same envelope proves only that whoever packed the envelope sealed it. So get the right key from Big Widget another way first, for example from its website or by hand at the start of the audit. An extra file sent with the bundle does not count. That is the pinned key.

Is the end missing? Picture a notebook with numbered pages. Each page also copies a short code from the page before. If someone tears out a page in the middle, you can tell. But if someone tears out the last fifty pages, the rest still looks perfect. So you also need to know, from somewhere else, what the last page should be. You need three things: its number, its code and which version of the rules was in force. The number alone is not enough. Someone with the seal could write a different notebook with the same number of pages. The code catches that. These three things together are the checkpoint.

Two bundles

Kofi is an auditor from outside Big Widget. Before any bundle arrives, Big Widget gives Kofi, by a separate route, the pinned key and the checkpoint: last page 214, the code for page 214, and rules version 0.3.

Now you try

The course has a practice kit, called the sandbox, in the folder apps/sandbox400/. It holds three made-up bundles from Big Widget: A, B and C. A and B are Kofi's. C has the right number of receipts, 214, but a different story inside, so its last code does not match. Each bundle has its own one-page VERIFY.md. The sandbox's instruction page is apps/sandbox400/README.md.

You get the key codes and the checkpoint from a small file, apps/sandbox400/trust/bw-warranty-gate.trust.json. It stands for what Big Widget would give you by a separate route. The checkpoint in it says: last receipt 214, the code for receipt 214, and rules version 0.3. In a real audit, you would get these in person or from a page you already trust, never in the same download as the bundle.

There are two ways to do it. Both lead to the same answers.

For each bundle, write "accept" or "reject", and the one check that decides it.

The checker can spot a file that should not be there, or a box filled in where no box should be. It refuses a box filled in twice, so nothing can hide in a copy nobody reads. It can spot a line that looks like a question to a chatbot, a chatbot's answer or someone's email address. But it cannot read like a person. A private sentence that looks ordinary gets past it. So read the VERIFY.md page and the other short text files yourself, line by line, before you say "no private content".

The sandbox has a no-code result for the two rule gates in this lesson too. In apps/sandbox400/prepared/4009-b-premise-gate.txt, the worksheet's "is there a receipt?" check lets letters out on a receipt that says "wait", or one for an old version of the rule. The course's check stops them.

Be honest about what this is. The bundles are made up, and anyone could make the same seal, because the seal's secret is public on purpose. So a good result shows you can do the check. It shows nothing about any real system.

The old Level 300 receipt

Bundle A also carries your old Level 300 receipt, which was not sealed when it was made. Sealing it now only protects it from now on. Checking a seal tells you which key made it.9 It does not tell you that what is inside is true. That last part is our own reasoning, not a rule in the standard. So say it in your statement.

Write one honest sentence

The worksheet asks for one sentence, with your name and the date.1 Say what you checked, and no more. Here is Kofi's:

Good: "I, Kofi Mensah, on 18 November 2026, checked Bundle A offline using its VERIFY.md: it is sealed with Big Widget's pinned key, it runs with no gaps up to receipt 214, its last code and rules version (0.3) match the checkpoint I was given separately, and I found no private content, with the checker and by reading its pages myself; I did not check the old Level 300 receipt inside, which was never sealed."

Too much: "I checked the bundle, so the system is legal, fair and complete."

A bundle can show the records are whole and untouched, up to the checkpoint. It cannot show that the decisions in them were right. That was the premise gate's job. It also cannot show that everything that happened was written down. Kofi signs his own sentence. No tool signs it for him. (Lesson 500.6 covers how much trust to give different kinds of evidence.)

Your sentence is about what you did. If you checked a practice bundle offline, say which one, and say it was a practice bundle. If you only read about Kofi, say "I have not checked a bundle yet." That is an honest answer too.

To practise, the Foundation's audit page has a 10-minute Gate Check of yes-or-no questions, "each tied to evidence".10 Keep private work details out of it.

The three gates in one breath

Check yourself:

Words to know: premise, independent checker, holder, protected attribute, proxy, receipt, bundle, VERIFY.md, pinned key, checkpoint, verification statement.

Sources

  1. s5 AiGovOps Foundation (2026). Level 400 · Prove it to a stranger — worksheet. Official doc. Checked 2026-11-19 by source checker (separate Claude session 80843764), round 4. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  2. s1 Royal Commission into the Robodebt Scheme (Commissioner Catherine Holmes AC SC) (2023). Report of the Royal Commission into the Robodebt Scheme. Record. Checked 2026-11-20 by source checker (separate Claude session b2f704ca), round 4 re-check (20 Nov 2026). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s2 Parlementaire ondervragingscommissie Kinderopvangtoeslag, Tweede Kamer der Staten-Generaal (2020). Ongekend onrecht (Parlementaire ondervragingscommissie Kinderopvangtoeslag), Kamerstuk 35 510, nr. 2. Record. Checked 2026-11-19 by source checker (separate Claude session 1d1f9266), round 3. ↩ ↩ ↩ ↩
  4. s3 Autoriteit Persoonsgegevens (2021). Besluit tot boeteoplegging (Autoriteit Persoonsgegevens fine decision, Belastingdienst/Toeslagen), 25 November 2021. Record. Checked 2026-11-19 by source checker (separate Claude session 1d1f9266), round 3. ↩ ↩ ↩ ↩
  5. s4 Lord Justice Holroyde, Mr Justice Picken and Mrs Justice Farbey DBE, Court of Appeal (Criminal Division) (2021). Hamilton v Post Office Ltd [2021] EWCA Crim 577. Record. Checked 2026-11-19 by source checker (separate Claude session 1d1f9266), round 3. ↩ ↩ ↩ ↩ ↩
  6. s8 Lord Justice Holroyde, Mr Justice Picken and Mrs Justice Farbey DBE, Court of Appeal (Criminal Division) (2021). Hamilton v Post Office Ltd [2021] EWCA Crim 577 (summary). Record. Checked 2026-11-19 by source checker (separate Claude session 1d1f9266), round 3. ↩ ↩ ↩ ↩ ↩
  7. s9 Mr Justice Fraser, High Court of Justice, Queen's Bench Division (2019). Bates v Post Office Ltd (No 6) "Horizon Issues" [2019] EWHC 3408 (QB). Record. Checked 2026-11-19 by source checker (separate Claude session 1d1f9266), round 3. ↩ ↩
  8. s7 AiGovOps Foundation (2026). AiGovOps Beacon — find every AI, sign what matters. Official doc. Checked 2026-11-20 by source checker (separate Claude session ddfcc6f0), round 5 re-check (20 Nov 2026). ↩ ↩ ↩
  9. s10 S. Josefsson and I. Liusvaara (2017). Edwards-Curve Digital Signature Algorithm (EdDSA), RFC 8032. Standard. Checked 2026-11-20 by source checker (separate Claude session ddfcc6f0), round 5 re-check (20 Nov 2026). ↩ ↩
  10. s6 AiGovOps Foundation (Ken Johnston and Bob Rapp) (2026). How do I create an AI audit plan?. Official doc. Checked 2026-11-19 by source checker (separate Claude session 1d1f9266), round 3. ↩

Next: the quiz