Level 400: Engineering AI systems · Module 400.7

Two colleagues sort blank index cards into three separate piles and count each one.
Count by group

Testing for fairness and proxies · Standard gear

You will be able to: measure a model's error and outcome rates group by group, and show how one overall number can hide harm to a group; tell a model that uses a protected attribute directly from one that uses a proxy for it, and explain why deleting the protected column does not remove the proxy's effect; and explain why three common fairness conditions for risk scores cannot all hold at once except in narrow special cases, so that the team must choose, write down why, and name the person who signed the choice.

This module is as of November 2026. Reviewed and published, version 1.0.0. Every Big Widget Company table in it is made up for teaching. The real case is the Dutch tax administration's use of nationality in a risk model, as recorded by the Dutch data protection authority.

1. Measure by group, not just overall

Why one number is not enough

A model can score well overall and still fail one group badly. The textbook Fairness and Machine Learning puts it plainly: "we typically only test their overall accuracy; so a '5% error' statistic might hide the fact that a model performs terribly for a minority group. Reporting accuracy rates by group will help alert us to problems".1 One reason it happens is sample size: a group with fewer examples in the training data can get a model that works less well for it, if that group differs from the majority in ways that matter to the task.1

So the first fairness test is not a formula. It is a table: the same counts you already use to judge a model, split by group.

The four counts, and the rates built from them

For a model that flags cases (a flag means "look harder"), every case falls in one of four cells. Chouldechova's paper on risk tools sets them out as a confusion matrix: true negatives, false positives, false negatives and true positives.2 From those counts, per group:

Chouldechova defines error rate balance as equal false-positive and equal false-negative rates across groups at the chosen threshold, and predictive parity as equal PPV across groups.2 The textbook groups the many proposed criteria into three families, each equalising one statistic across groups: the acceptance (flag) rate, the error rates, or the outcome frequency at each score.1 Section 3 shows why you cannot have all of them at once. This section is about measuring them.

Worked example (made up): Big Widget's supplier-risk model

Big Widget's procurement team uses a model to flag suppliers for an extra audit. A flag holds the supplier's next payment until the audit is done, so a wrong flag costs a supplier who did nothing wrong. Six months after release, the team checks 1,000 suppliers whose real risk is now known. The data below is made up.

RegionSuppliersIn fact low-riskLow-risk but flaggedIn fact high-riskHigh-risk and flagged
North600540276048
East300270143024
South1009027108
All1,0009006810080

The overall picture looks fine. The model is right about 912 of 1,000 suppliers (91.2%). Its overall false-positive rate is 68 out of 900, about 7.6%. Now split it:

RegionFalse-positive rate95% intervalFalse-negative rateFlag rateAccuracy
North27/540 = 5.0%3.5% to 7.2%12/60 = 20%75/600 = 12.5%93.5%
East14/270 = 5.2%3.1% to 8.5%6/30 = 20%38/300 = 12.7%93.3%
South27/90 = 30.0%21.5% to 40.1%2/10 = 20%35/100 = 35.0%71.0%

(Intervals are Wilson score intervals, computed for this made-up table; module 400.6 covers intervals.)

A low-risk supplier in the South is about six times as likely to have its payment held as a low-risk supplier in the North. The overall 7.6% hid that, because the South is only a tenth of the suppliers. The false-negative rates are the same in every region, so a team that checked only missed risks would also have missed this. And the South has only 90 low-risk suppliers, so its rate is uncertain, but even the bottom of its interval (21.5%) is far above the top of the North's (7.2%).

What to do with it. Write the per-group table into the evaluation record, not just the overall score. Decide in advance which rate matters most for the harm (here, wrong payment holds, so the false-positive rate). Name the groups you will check before you look, so you are not choosing groups after the fact to make the numbers look good. Then ask why the South differs: different data, a feature that behaves differently there, or a feature standing in for something it should not (section 2).

The misconception this corrects: "High overall accuracy means the model is fair to everyone." The supplier model is 91.2% accurate overall and holds payments from almost a third of the South's low-risk suppliers.

2. Direct use and proxies

There are two ways a model can treat a protected group differently. Direct use: the protected attribute (nationality, sex, ethnicity and so on) is itself an input. Proxy use: the attribute is not an input, but another feature stands in for it. NIST defines a proxy as "a variable that can stand in for another, usually not directly observable or measurable, variable."3 You test for the two in different ways, so tell them apart first.

Real case: nationality as an indicator (direct use)

The Netherlands' tax administration (the Belastingdienst) pays childcare benefit through its benefits department, Toeslagen. In a decision dated 25 November 2021, the Dutch data protection authority, the Autoriteit Persoonsgegevens (AP), fined the Minister of Finance a total of €2,750,000 for how Toeslagen had processed applicants' nationality.4 The AP found three unlawful kinds of processing:4

  1. keeping the dual nationality of Dutch citizens in its benefits system, at least from 1 January 2016 to 30 June 2020;
  2. using applicants' nationality for an indicator in its "risico-classificatiemodel" (risk classification model), at least from March 2016 to October 2018;
  3. using applicants' nationality to detect organised fraud, at least from 1 January 2016 to February 2019.

How the model worked, as the decision records it: the model assessed all applications in a month with a self-learning algorithm and estimated, from a set of indicators, how great the risk was that an application was incorrect. The 100 applications with the highest risk score that month went to an employee for a manual check.4 One indicator, at least from March 2016 to October 2018, was "Nederlanderschap/niet-Nederlanderschap" (Dutch nationality or not). A person with dual nationality, one of them Dutch, counted as Dutch.4 The AP found that not having Dutch nationality, combined with other indicators, could lead to a higher risk score and a manual check.4 In the minister's own account, recorded in the decision, applicants without Dutch nationality therefore had a relatively higher chance of being selected for a manual check: "Dit had nooit mogen gebeuren" ("this should never have happened").4

The AP judged the risk-model and fraud-detection processing to be discriminatory, because it made a distinction on the basis of nationality "zonder dat daar een objectieve rechtvaardiging voor bestaat" (without an objective justification for it).4 It also pointed to a better indicator. Nationality alone did not settle whether someone could claim the benefit; an indicator closer to the actual rule would be: Dutch nationality, or an EU nationality with registration in a Dutch municipality, or a non-EU nationality with a valid residence permit.4 The minister told the AP that nationality had not been used in the childcare-benefit risk model since October 2018.4 The State Secretary for Finance sent the decision to Parliament with a letter dated 7 December 2021, which says: "Ik zal mij niet verzetten tegen de boete" ("I will not oppose the fine").5

Label it correctly. This is direct use: nationality itself was an indicator in the model. It is not an example of a proxy. The case teaches the first half of this objective. The parliamentary inquiry into the wider childcare-benefit scandal answered different questions and is taught in 400.9; this module relies only on the AP's decision and the minister's letter.

Proxies: why deleting the column is not enough

Suppose the nationality column is deleted. Is the problem gone? Usually not. NIST's bias guidance says: "Developers sometimes exclude protected attributes, associated with social groups which have historically been discriminated against. However, this does not remedy the problem, since the information can be inadvertently inferred in other ways through proxy or latent variables." Its example: "race can be inferred through zip code."3

The textbook explains the mechanism. Several features, each only slightly linked to the protected attribute, can together predict it well; "In large feature spaces sensitive attributes are generally redundant given the other features." So "If a classifier trained on the original data uses the sensitive attribute and we remove the attribute, the classifier will then find a redundant encoding in terms of the other features."1 Kleinberg, Mullainathan and Raghavan make the same point in their model of risk scores: the scoring rule "does not have access to the group" of the person, and "this does not mean that the rule is incapable of exhibiting bias between the two groups."6

Worked example (made up): finding the proxy

Big Widget's supplier model never had a nationality column for supplier owners; the data team removed it. For a fairness test, the compliance team holds the owners' nationality aside (it is not a model input) and compares it with the remaining features for 200 suppliers. The data is made up.

Owner foreign nationalOwner domestic national
Supplier banks abroad449
Supplier banks at home6141

"Bank country" is a strong proxy. Of the 53 suppliers that bank abroad, 44 (83%) have a foreign owner; of the 147 that bank at home, 6 (4%) do. The rule "banks abroad, so foreign owner" is right for 185 of 200 suppliers (92.5%), against 75% for simply guessing "domestic" every time. If the model leans on bank country, it can treat foreign-owned suppliers differently with no nationality column at all.

How to test for proxies:

  1. Hold the protected attribute aside for testing, under its own access rules, even if it is never a model input. Without it you cannot build the section 1 table or find proxies. (Whether and how your organisation may hold it is a legal question for counsel; this module does not teach the law.)
  2. Try to predict the attribute from the model's inputs. The textbook observes that if you treat the protected attribute as the target of a classification problem, "there is reason to believe that the remaining features would give a non-trivial classifier" for it.1 So try it (this step is the course's method): a feature or set of features that predicts the attribute well is a candidate proxy. Score the predictor on data it was not fitted on, and compare it with the right baseline: accuracy against always guessing the larger group (75% in the table above), or ROC-AUC against 0.5. A predictor well above baseline is evidence the inputs encode the attribute, not proof the model relies on it. One near baseline means "no encoding found by this test", not "no proxy".
  3. Ask whether each strong predictor has a logical link to the target. NIST lists, as a mitigation for proxy discrimination, "minimizing the number of input variables, and ensuring that there is no strong correlation amongst them and a logical relationship to the prediction target."3 Bank country might link to real payment risk, or it might mostly encode ownership. The AP's alternative indicator in the Dutch case shows the move: replace a feature that stands in for a group with one tied to the actual rule.4
  4. Run the per-group table (section 1) by the protected attribute, with and without the candidate proxy, and record what changes.

NIST also warns that proxies "may still reveal sensitive information about individuals or groups" even when they are used to hide protected attributes.3 Dropping the column hides the attribute from the reader of the data, not from the model.

The misconception this corrects: "If nationality is not a column, the model cannot discriminate by nationality." It can, through bank country, postcode, language or any feature that encodes it.

3. Three fair-sounding conditions you cannot all meet

Three fairness conditions for a risk score: calibration within groups, balance for the negative class and balance for the positive class. All three can hold only with perfect prediction or equal base rates across groups.
The three conditions and Theorem 1.1 are from Kleinberg, Mullainathan and Raghavan (s1).
Figure data as text

Three fairness conditions

ConditionWhat it shows
(A) [s1]Calibration within groups. In each group, among people given score v, a v fraction are in fact positive.
(B) [s1]Balance for the negative class. People who are in fact negative get the same average score in each group.
(C) [s1]Balance for the positive class. People who are in fact positive get the same average score in each group.
THEOREM 1.1 [s1]All three only if. Prediction is perfect, or the groups have equal base rates.

Three reasonable conditions

Kleinberg, Mullainathan and Raghavan (2016; ITCS 2017) studied risk scores: a model puts each person in a "bin" with a score, meant as the probability that the person is a positive case.6 They formalised three conditions that had each been proposed as fairness:6

The authors note that the two balance conditions "can be viewed as generalizations of the notions that both groups should have equal false negative and false positive rates."6

The result

Their Theorem 1.1: if a risk assignment satisfies (A), (B) and (C), then the situation "must either allow for perfect prediction" (every person's outcome is known for certain) "or have equal base rates" (the same share of positives in each group).6 Outside those two special cases, every way of assigning scores breaks at least one condition. Three points matter for practice:

The paper's own example: in the debate over the COMPAS risk tool, the critique of the tool focused on its violation of conditions (B) and (C), and the counter-arguments established that it satisfied (A).6 Both sides could be right at the same time, because the three cannot all hold when base rates differ.

A separate result, for thresholds. Chouldechova (2017) shows a related conflict for a yes-or-no flag at a fixed threshold: when the base rate differs across groups, a tool with equal precision (predictive parity) in each group cannot also have equal false-positive and false-negative rates.2 This is a different theorem from Kleinberg's (it is about a thresholded flag, not a score), and is not the one this module's quiz keys. It has an edge the course adds from the algebra behind it: the conflict needs precision above 0% and below 100%, with both outcomes present in each group. If every flag is correct (precision 100%), no group has false alarms, so the false-positive rates match at zero, and the false-negative rates can match too, even though the base rates differ. Chouldechova adds a warning that fits both: "fairness itself ... is a social and ethical concept, not a statistical one."2

At Big Widget: choose, write it down, name who signed

First, a check on section 1. In that made-up table the base rate is the same in every region: 60 of 600, 30 of 300 and 10 of 100 suppliers are high-risk, 10% each. Equal base rates are one of the two cases where Kleinberg's conflict does not apply,6 so the theorem cannot excuse the South's 30% false-positive rate. That is a model defect to fix, not a trade-off forced by mathematics. Use the theorem to explain conflicts that cannot be avoided, never to excuse ones that can.

Now suppose instead that Big Widget's supplier score is used in two regions where the share of suppliers that turn out high-risk differs (a made-up situation). Kleinberg's result says the team cannot have calibration within each region and both balance conditions. The team has to choose, for this use:

Do not mistake (B) for equal payment holds. A payment is held when a score crosses the threshold. (B) is about average scores, and two regions can have the same average score among low-risk suppliers while very different shares of them cross the threshold. (Made up: in one region every low-risk supplier scores 0.2; in the other, one in five scores 1 and the rest score 0. Both averages are 0.2. At a threshold of 0.5, none are held in the first region and one in five in the second.) So whichever condition the team keeps, it still measures the false-positive rate at the actual threshold, region by region, as in section 1.

The course's rule (not a rule from the paper): the choice is a decision, not a test result. Record it before release in the evaluation record:

If the base rates are equal, or the model predicts every case perfectly, the conflict disappears; say so in the record if that is your case, and show the numbers. "Nearly perfect" is not the same: with unequal base rates, a small error still leaves the conflict in place.

The misconception this corrects: "There is one fairness test, and a model either passes it or fails it." There are several reasonable tests. For these three, outside the two special cases, they conflict. Passing one is a choice that gives up another, and the record must say which.

4. The three yeses for fairness testing

A yes here is not a blanket licence. It covers one model, one decision, one threshold and the groups named in advance, and a named person gives it.

What you keep, and what you share. Keep two things apart. The evaluation record holds the per-group counts, the proxy results and the reasons. It stays in the controlled test environment, because rates for a small group can still point to individuals. The shareable receipt holds no private content: the decision, hashes of the policy and the evaluation record, the signing key's id and the date. It never holds any individual's protected attribute, prompts, model outputs or other personal data.

What a receipt can show, and what it cannot. A hash or a key id on its own proves no one signed anything: anyone can write those fields down without signing. A receipt shows which record was signed only after someone checks its signature against a key they already trust; then the record hash, compared with the kept record, tells them which bytes that signature covers. How to make and check that signature is 400.8 and 400.9; this module does not supply it. Even a checked receipt shows what was decided and recorded, by the holder of that key. It does not prove the model is fair.

What this module does not know. Every Big Widget number here is made up. No live model, proxy probe or receipt check was run for this module. It does not say what happened to the people affected in the Dutch case. Where your own record has a gap, write "not checked" and name who will check it.

Words to know: cohort, false-positive rate, false-negative rate, flag rate, precision, protected attribute, proxy, redundant encoding, calibration within groups, balance for the positive and negative class, base rate, evaluation record, receipt.

Further learning (not copied; background only)

Sources

  1. s5 Solon Barocas, Moritz Hardt, and Arvind Narayanan (2023). Fairness and Machine Learning: Limitations and Opportunities. Book. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, separate session from the lesson writer and the round 2 reviser), round 2 re-check, 19 Nov 2026. ↩ ↩ ↩ ↩ ↩
  2. s2 Alexandra Chouldechova (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5, separate session from the lesson writer and the round 2 reviser), round 2 re-check, 19 Nov 2026. ↩ ↩ ↩ ↩
  3. s3 Reva Schwartz, Apostol Vassilev, Kristen Greene, Lori Perine, Andrew Burt, and Patrick Hall (2022). Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (NIST SP 1270). Standard. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, separate session from the lesson writer), 18 Nov 2026. ↩ ↩ ↩ ↩
  4. s4 Autoriteit Persoonsgegevens (2021). Besluit tot boeteoplegging (Autoriteit Persoonsgegevens to the Minister of Finance, Belastingdienst/Toeslagen). Record. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, separate session from the lesson writer), 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  5. s6 A.C. van Huffelen, Staatssecretaris van Financiën (Tweede Kamer der Staten-Generaal) (2021). Kamerstuk 31066, nr. 936: Belastingdienst, brief van de staatssecretaris van Financiën (Besluit tot boeteoplegging Autoriteit Persoonsgegevens). Record. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, separate session from the lesson writer), 18 Nov 2026. ↩
  6. s1 Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan (2016). Inherent Trade-Offs in the Fair Determination of Risk Scores. Paper. Checked 2026-11-18 by source-checker agent (Claude Opus 5.5, separate session from the lesson writer), 18 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩

Next: the quiz