Level 400: Engineering AI systems · Module 400.7

Two colleagues sort blank index cards into three separate piles and count each one.
Count by group

Testing for fairness and proxies · Plain gear

You will be able to: check how a model treats each group, not just everyone together; explain why deleting a column like "nationality" does not stop a model from treating people differently by nationality; and explain why three fair-sounding tests for a risk score clash, so a named person has to choose and sign.

This module is as of November 2026. Reviewed and published, version 1.0.0. The Big Widget Company tables are made up for teaching. The real case comes from the Dutch data protection authority.

1. Look at each group, not just the total

A model can be right most of the time and still be unfair to one group. A textbook on fairness in machine learning warns that a "5% error" score "might hide the fact that a model performs terribly for a minority group". Its advice: report how well the model does for each group.1

Think of a school where the average test score is good. That does not tell you whether one class is failing. You have to look class by class.

Four kinds of result

When a model flags something ("look at this one more closely"), each case ends up in one of four boxes.2

For each group, count the false alarms out of everyone in that group who was fine. That is the false-alarm rate. Count the misses out of everyone who was a problem. That is the miss rate. Fairness researchers compare these rates between groups.2

Big Widget example (made up)

Big Widget uses a model to flag suppliers for an extra check. A flag holds the supplier's next payment until the check is done. So a false alarm hurts a supplier who did nothing wrong.

Six months later the team knows which of 1,000 suppliers were really risky. Overall, the model was right about 912 of them. That sounds good. Now look by region:

RegionFine suppliersFine, but flagged anywayFalse alarms
North54027about 1 in 20
East27014about 1 in 20
South9027about 3 in 10

A fine supplier in the South is about six times more likely to have its payment held than a fine supplier in the North. The total hid this, because the South is small: only 100 of the 1,000 suppliers. The miss rate was the same in every region, so checking only misses would not have shown it either.

What to do. Before you look, write down which groups you will check. Pick the result that matches the harm. Here the harm is a held payment, so check false alarms. Put the table for each group in the record, not just the total.

The mistake this fixes: "If the model is right most of the time, it is fair to everyone." It was right 91% of the time, and it still held payments from almost a third of the fine suppliers in the South.

2. Deleting the column is not enough

A model can treat a group differently in two ways.

Real case: nationality in a Dutch risk model (direct use)

In the Netherlands, part of the tax office pays parents a benefit to help with childcare. The Dutch data protection authority (the AP) looked at how that office used people's nationality. In a decision dated 25 November 2021, it fined the Minister of Finance a total of €2.75 million.4

The office used a computer model to pick which benefit applications to check by hand. Each month, the model gave every application a risk score. The 100 with the highest scores went to a staff member to check.4 From at least March 2016 to October 2018, one of the things the model looked at was simply: is this person Dutch, or not?4 The AP found that not being Dutch, together with other signals, could raise a person's score and lead to a check by hand.4

The AP said this was discrimination. The office treated people differently because of nationality, with no fair reason for it.4 It also said the office could have used a better signal: one based on the actual rule for who can get the benefit, which depends on things like legal residence, not on nationality alone.4 The minister told the AP that nationality had not been used in the model since October 2018.4 The government sent the decision to the Dutch parliament with a letter dated 7 December 2021, and said it would not fight the fine.5

This is direct use. Nationality itself was in the model. It is not a proxy case.

Proxies: the column is gone, the pattern is not

What if you just delete the nationality column? NIST says that does not fix the problem, because the model can work out the same thing from other information. Its example: where someone lives can give away their race.3 The fairness textbook explains that lots of small clues, put together, can point clearly to the deleted fact. A model can then "find" the deleted fact in the other columns.1

It is like hiding a friend's name on a party list but leaving their address, phone number and birthday. Anyone who knows them can still tell who it is.

Big Widget example (made up)

Big Widget's supplier model has no column for the owner's nationality. For a test, the compliance team keeps that information to one side, outside the model, for 200 suppliers. Then it compares it with the model's other columns.

Foreign ownerLocal owner
Supplier's bank is abroad449
Supplier's bank is at home6141

Most suppliers with a bank abroad have a foreign owner (44 of 53). Almost none with a bank at home do (6 of 147). So "bank country" is a proxy for the owner's nationality. If the model leans on bank country, it can treat foreign owners differently, with no nationality column at all.

How to check for a proxy.

  1. Keep the protected fact (like nationality) to one side, just for testing, with strict rules on who can see it. Whether you are allowed to keep it is a question for your lawyers.
  2. See which of the model's inputs give that fact away. Test this on suppliers the check did not learn from, and compare it with plain guessing. If nothing shows up, write "none found by this check", not "no proxy".
  3. Ask whether each of those inputs has a real link to what you are predicting. NIST suggests using fewer inputs, and only ones with a logical link to the thing being predicted.3 The AP's better signal in the Dutch case shows the idea: use the real rule, not a stand-in for a group.4
  4. Run the group-by-group table from section 1 again, and write down what you found.

The mistake this fixes: "If nationality is not a column, the model cannot treat people differently by nationality." It can, through bank country, address, language or other clues.

3. Three fair-sounding tests that clash

Three fairness conditions for a risk score: calibration within groups, balance for the negative class and balance for the positive class. All three can hold only with perfect prediction or equal base rates across groups.
The three conditions and Theorem 1.1 are from Kleinberg, Mullainathan and Raghavan (s1).
Figure data as text

Three fairness conditions

ConditionWhat it shows
(A) [s1]Calibration within groups. In each group, among people given score v, a v fraction are in fact positive.
(B) [s1]Balance for the negative class. People who are in fact negative get the same average score in each group.
(C) [s1]Balance for the positive class. People who are in fact positive get the same average score in each group.
THEOREM 1.1 [s1]All three only if. Prediction is perfect, or the groups have equal base rates.

People have suggested many fairness tests. Three of them all sound sensible for a risk score:6

Three researchers, Kleinberg, Mullainathan and Raghavan, proved that you cannot have all three at once, except in two special cases. Either the model knows every answer for certain, or the groups have exactly the same share of risky people.6 So check which case you are in, and show the numbers, before you lean on the result. "Almost always right" does not count: only a model that is never wrong escapes. Their result holds whether the score comes from a computer or from people.6 And their paper does not say which test to pick.6

Picture a referee who must be fair to two teams in these three ways at once. When the teams are different and the referee cannot see the future, the maths means one of the three has to give. That is true of these three tests. It does not mean fair decisions are impossible.

Another researcher, Chouldechova, found a similar clash for simple yes-or-no flags. The same paper says that fairness "is a social and ethical concept, not a statistical one."2

At Big Widget: choose, write it down, sign it

Big Widget's supplier score is used in two regions where the share of risky suppliers is different (made up). The team cannot meet all three tests, so it must choose:

Careful: the second test is not the same as fair payment holds. It is about average scores. A payment is held when a score goes over the line. Two regions can have the same average score for fine suppliers, but very different shares going over the line. (Made up: in one region every fine supplier scores 2 out of 10. In the other, one in five scores 10 and the rest score 0. The averages match. With the line at 5, nobody is held in the first region and one in five in the second.) So whatever test the team keeps, it still counts the false alarms, group by group, at the real line.

This course's rule: write the choice down before the model is used. Say which test you kept, which you gave up, by how much, and why. Write down the false-alarm rate for each group, and the biggest gap the signer will accept. A named person signs it, with their role and the date. An AI agent never signs.

The mistake this fixes: "There is one fairness test, and a model passes or fails it." There are several. These three clash, and choosing one means giving up another.

4. The three yeses

A yes is not a blank cheque. It covers one model, one job and the groups you named, and a named person gives it.

Keep two things apart. The full record, with the group tables and reasons, stays locked away, because numbers for a small group can still give people away. The shareable receipt holds only the decision, short codes (hashes) that stand for the record, and the date. It never holds any person's nationality, prompts, answers or other private details.

The codes alone do not prove anyone signed. Anyone can copy them onto a page. A receipt shows what was signed only once someone checks its signature against a key they already trust. You learn how in 400.8 and 400.9. Even then, it shows what was decided, not that the model is fair.

What this module does not know. The Big Widget numbers are made up. Nobody ran a real model or a real check for this lesson. If your own record has a gap, write "not checked" and name who will check it.

Words to know: group, false alarm, miss, proxy, protected fact, fairness test, record, receipt.

Sources

  1. s5 Solon Barocas, Moritz Hardt, and Arvind Narayanan (2023). Fairness and Machine Learning: Limitations and Opportunities. Book. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, separate session from the lesson writer and the round 2 reviser), round 2 re-check, 19 Nov 2026. ↩ ↩
  2. s2 Alexandra Chouldechova (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, separate session from the lesson writer and the round 2 reviser), round 2 re-check, 19 Nov 2026. ↩ ↩ ↩
  3. s3 Reva Schwartz, Apostol Vassilev, Kristen Greene, Lori Perine, Andrew Burt, and Patrick Hall (2022). Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (NIST SP 1270). Standard. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, separate session from the lesson writer), 18 Nov 2026. ↩ ↩ ↩
  4. s4 Autoriteit Persoonsgegevens (2021). Besluit tot boeteoplegging (Autoriteit Persoonsgegevens to the Minister of Finance, Belastingdienst/Toeslagen). Record. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, separate session from the lesson writer), 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  5. s6 A.C. van Huffelen, Staatssecretaris van Financiën (Tweede Kamer der Staten-Generaal) (2021). Kamerstuk 31066, nr. 936: Belastingdienst, brief van de staatssecretaris van Financiën (Besluit tot boeteoplegging Autoriteit Persoonsgegevens). Record. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, separate session from the lesson writer), 18 Nov 2026. ↩
  6. s1 Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan (2016). Inherent Trade-Offs in the Fair Determination of Risk Scores. Paper. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, separate session from the lesson writer), 18 Nov 2026. ↩ ↩ ↩ ↩

Next: the quiz