
Testing for fairness and proxies · Standard gear
You will be able to: measure a model's error and outcome rates group by group, and show how one overall number can hide harm to a group; tell a model that uses a protected attribute directly from one that uses a proxy for it, and explain why deleting the protected column does not remove the proxy's effect; and explain why three common fairness conditions for risk scores cannot all hold at once except in narrow special cases, so that the team must choose, write down why, and name the person who signed the choice.
This module is as of November 2026. Reviewed and published, version 1.0.0. Every Big Widget Company table in it is made up for teaching. The real case is the Dutch tax administration's use of nationality in a risk model, as recorded by the Dutch data protection authority.
1. Measure by group, not just overall
Why one number is not enough
A model can score well overall and still fail one group badly. The textbook Fairness and Machine Learning puts it plainly: "we typically only test their overall accuracy; so a '5% error' statistic might hide the fact that a model performs terribly for a minority group. Reporting accuracy rates by group will help alert us to problems".1 One reason it happens is sample size: a group with fewer examples in the training data can get a model that works less well for it, if that group differs from the majority in ways that matter to the task.1
So the first fairness test is not a formula. It is a table: the same counts you already use to judge a model, split by group.
The four counts, and the rates built from them
For a model that flags cases (a flag means "look harder"), every case falls in one of four cells. Chouldechova's paper on risk tools sets them out as a confusion matrix: true negatives, false positives, false negatives and true positives.2 From those counts, per group:
- False-positive rate (FPR): of the cases that were in fact fine, the share the model flagged. These are the people or firms who bore the extra scrutiny for nothing.
- False-negative rate (FNR): of the cases that were in fact a problem, the share the model missed.
- Flag rate (an outcome rate): of all cases in the group, the share flagged.
- Precision (positive predictive value, PPV): of the flagged cases, the share that were in fact a problem.
Chouldechova defines error rate balance as equal false-positive and equal false-negative rates across groups at the chosen threshold, and predictive parity as equal PPV across groups.2 The textbook groups the many proposed criteria into three families, each equalising one statistic across groups: the acceptance (flag) rate, the error rates, or the outcome frequency at each score.1 Section 3 shows why you cannot have all of them at once. This section is about measuring them.
Worked example (made up): Big Widget's supplier-risk model
Big Widget's procurement team uses a model to flag suppliers for an extra audit. A flag holds the supplier's next payment until the audit is done, so a wrong flag costs a supplier who did nothing wrong. Six months after release, the team checks 1,000 suppliers whose real risk is now known. The data below is made up.
| Region | Suppliers | In fact low-risk | Low-risk but flagged | In fact high-risk | High-risk and flagged |
|---|---|---|---|---|---|
| North | 600 | 540 | 27 | 60 | 48 |
| East | 300 | 270 | 14 | 30 | 24 |
| South | 100 | 90 | 27 | 10 | 8 |
| All | 1,000 | 900 | 68 | 100 | 80 |
The overall picture looks fine. The model is right about 912 of 1,000 suppliers (91.2%). Its overall false-positive rate is 68 out of 900, about 7.6%. Now split it:
| Region | False-positive rate | 95% interval | False-negative rate | Flag rate | Accuracy |
|---|---|---|---|---|---|
| North | 27/540 = 5.0% | 3.5% to 7.2% | 12/60 = 20% | 75/600 = 12.5% | 93.5% |
| East | 14/270 = 5.2% | 3.1% to 8.5% | 6/30 = 20% | 38/300 = 12.7% | 93.3% |
| South | 27/90 = 30.0% | 21.5% to 40.1% | 2/10 = 20% | 35/100 = 35.0% | 71.0% |
(Intervals are Wilson score intervals, computed for this made-up table; module 400.6 covers intervals.)
A low-risk supplier in the South is about six times as likely to have its payment held as a low-risk supplier in the North. The overall 7.6% hid that, because the South is only a tenth of the suppliers. The false-negative rates are the same in every region, so a team that checked only missed risks would also have missed this. And the South has only 90 low-risk suppliers, so its rate is uncertain, but even the bottom of its interval (21.5%) is far above the top of the North's (7.2%).
What to do with it. Write the per-group table into the evaluation record, not just the overall score. Decide in advance which rate matters most for the harm (here, wrong payment holds, so the false-positive rate). Name the groups you will check before you look, so you are not choosing groups after the fact to make the numbers look good. Then ask why the South differs: different data, a feature that behaves differently there, or a feature standing in for something it should not (section 2).
The misconception this corrects: "High overall accuracy means the model is fair to everyone." The supplier model is 91.2% accurate overall and holds payments from almost a third of the South's low-risk suppliers.
2. Direct use and proxies
There are two ways a model can treat a protected group differently. Direct use: the protected attribute (nationality, sex, ethnicity and so on) is itself an input. Proxy use: the attribute is not an input, but another feature stands in for it. NIST defines a proxy as "a variable that can stand in for another, usually not directly observable or measurable, variable."3 You test for the two in different ways, so tell them apart first.
Real case: nationality as an indicator (direct use)
The Netherlands' tax administration (the Belastingdienst) pays childcare benefit through its benefits department, Toeslagen. In a decision dated 25 November 2021, the Dutch data protection authority, the Autoriteit Persoonsgegevens (AP), fined the Minister of Finance a total of €2,750,000 for how Toeslagen had processed applicants' nationality.4 The AP found three unlawful kinds of processing:4
- keeping the dual nationality of Dutch citizens in its benefits system, at least from 1 January 2016 to 30 June 2020;
- using applicants' nationality for an indicator in its "risico-classificatiemodel" (risk classification model), at least from March 2016 to October 2018;
- using applicants' nationality to detect organised fraud, at least from 1 January 2016 to February 2019.
How the model worked, as the decision records it: the model assessed all applications in a month with a self-learning algorithm and estimated, from a set of indicators, how great the risk was that an application was incorrect. The 100 applications with the highest risk score that month went to an employee for a manual check.4 One indicator, at least from March 2016 to October 2018, was "Nederlanderschap/niet-Nederlanderschap" (Dutch nationality or not). A person with dual nationality, one of them Dutch, counted as Dutch.4 The AP found that not having Dutch nationality, combined with other indicators, could lead to a higher risk score and a manual check.4 In the minister's own account, recorded in the decision, applicants without Dutch nationality therefore had a relatively higher chance of being selected for a manual check: "Dit had nooit mogen gebeuren" ("this should never have happened").4
The AP judged the risk-model and fraud-detection processing to be discriminatory, because it made a distinction on the basis of nationality "zonder dat daar een objectieve rechtvaardiging voor bestaat" (without an objective justification for it).4 It also pointed to a better indicator. Nationality alone did not settle whether someone could claim the benefit; an indicator closer to the actual rule would be: Dutch nationality, or an EU nationality with registration in a Dutch municipality, or a non-EU nationality with a valid residence permit.4 The minister told the AP that nationality had not been used in the childcare-benefit risk model since October 2018.4 The State Secretary for Finance sent the decision to Parliament with a letter dated 7 December 2021, which says: "Ik zal mij niet verzetten tegen de boete" ("I will not oppose the fine").5
Label it correctly. This is direct use: nationality itself was an indicator in the model. It is not an example of a proxy. The case teaches the first half of this objective. The parliamentary inquiry into the wider childcare-benefit scandal answered different questions and is taught in 400.9; this module relies only on the AP's decision and the minister's letter.
Proxies: why deleting the column is not enough
Suppose the nationality column is deleted. Is the problem gone? Usually not. NIST's bias guidance says: "Developers sometimes exclude protected attributes, associated with social groups which have historically been discriminated against. However, this does not remedy the problem, since the information can be inadvertently inferred in other ways through proxy or latent variables." Its example: "race can be inferred through zip code."3
The textbook explains the mechanism. Several features, each only slightly linked to the protected attribute, can together predict it well; "In large feature spaces sensitive attributes are generally redundant given the other features." So "If a classifier trained on the original data uses the sensitive attribute and we remove the attribute, the classifier will then find a redundant encoding in terms of the other features."1 Kleinberg, Mullainathan and Raghavan make the same point in their model of risk scores: the scoring rule "does not have access to the group" of the person, and "this does not mean that the rule is incapable of exhibiting bias between the two groups."6
Worked example (made up): finding the proxy
Big Widget's supplier model never had a nationality column for supplier owners; the data team removed it. For a fairness test, the compliance team holds the owners' nationality aside (it is not a model input) and compares it with the remaining features for 200 suppliers. The data is made up.
| Owner foreign national | Owner domestic national | |
|---|---|---|
| Supplier banks abroad | 44 | 9 |
| Supplier banks at home | 6 | 141 |
"Bank country" is a strong proxy. Of the 53 suppliers that bank abroad, 44 (83%) have a foreign owner; of the 147 that bank at home, 6 (4%) do. The rule "banks abroad, so foreign owner" is right for 185 of 200 suppliers (92.5%), against 75% for simply guessing "domestic" every time. If the model leans on bank country, it can treat foreign-owned suppliers differently with no nationality column at all.
How to test for proxies:
- Hold the protected attribute aside for testing, under its own access rules, even if it is never a model input. Without it you cannot build the section 1 table or find proxies. (Whether and how your organisation may hold it is a legal question for counsel; this module does not teach the law.)
- Try to predict the attribute from the model's inputs. The textbook observes that if you treat the protected attribute as the target of a classification problem, "there is reason to believe that the remaining features would give a non-trivial classifier" for it.1 So try it (this step is the course's method): a feature or set of features that predicts the attribute well is a candidate proxy. Score the predictor on data it was not fitted on, and compare it with the right baseline: accuracy against always guessing the larger group (75% in the table above), or ROC-AUC against 0.5. A predictor well above baseline is evidence the inputs encode the attribute, not proof the model relies on it. One near baseline means "no encoding found by this test", not "no proxy".
- Ask whether each strong predictor has a logical link to the target. NIST lists, as a mitigation for proxy discrimination, "minimizing the number of input variables, and ensuring that there is no strong correlation amongst them and a logical relationship to the prediction target."3 Bank country might link to real payment risk, or it might mostly encode ownership. The AP's alternative indicator in the Dutch case shows the move: replace a feature that stands in for a group with one tied to the actual rule.4
- Run the per-group table (section 1) by the protected attribute, with and without the candidate proxy, and record what changes.
NIST also warns that proxies "may still reveal sensitive information about individuals or groups" even when they are used to hide protected attributes.3 Dropping the column hides the attribute from the reader of the data, not from the model.
The misconception this corrects: "If nationality is not a column, the model cannot discriminate by nationality." It can, through bank country, postcode, language or any feature that encodes it.
3. Three fair-sounding conditions you cannot all meet
Figure data as text
Three fairness conditions
| Condition | What it shows |
|---|---|
| (A) [s1] | Calibration within groups. In each group, among people given score v, a v fraction are in fact positive. |
| (B) [s1] | Balance for the negative class. People who are in fact negative get the same average score in each group. |
| (C) [s1] | Balance for the positive class. People who are in fact positive get the same average score in each group. |
| THEOREM 1.1 [s1] | All three only if. Prediction is perfect, or the groups have equal base rates. |
Three reasonable conditions
Kleinberg, Mullainathan and Raghavan (2016; ITCS 2017) studied risk scores: a model puts each person in a "bin" with a score, meant as the probability that the person is a positive case.6 They formalised three conditions that had each been proposed as fairness:6
- (A) Calibration within groups. In each group, a score means what it says: among people from group t given score v, a v fraction are in fact positive. Scores "mean what they claim to mean, even when considered separately in each group."
- (B) Balance for the negative class. People who are in fact negative get the same average score in each group.
- (C) Balance for the positive class. People who are in fact positive get the same average score in each group.
The authors note that the two balance conditions "can be viewed as generalizations of the notions that both groups should have equal false negative and false positive rates."6
The result
Their Theorem 1.1: if a risk assignment satisfies (A), (B) and (C), then the situation "must either allow for perfect prediction" (every person's outcome is known for certain) "or have equal base rates" (the same share of positives in each group).6 Outside those two special cases, every way of assigning scores breaks at least one condition. Three points matter for practice:
- It does not depend on the method. The result "applies regardless of how the risk assignment is computed", algorithmic or otherwise; the authors say it is "equally true whether the risk score is determined by an algorithm or by a system of human decision-makers."6
- Near enough does not escape it. An approximate version (Theorem 1.2) shows that meeting all three conditions approximately requires the situation to be approximately one of the two special cases.6
- It does not tell you which to choose. The authors say their intention "isn't to make a recommendation on how conflicts between different definitions of fairness should be handled."6
The paper's own example: in the debate over the COMPAS risk tool, the critique of the tool focused on its violation of conditions (B) and (C), and the counter-arguments established that it satisfied (A).6 Both sides could be right at the same time, because the three cannot all hold when base rates differ.
A separate result, for thresholds. Chouldechova (2017) shows a related conflict for a yes-or-no flag at a fixed threshold: when the base rate differs across groups, a tool with equal precision (predictive parity) in each group cannot also have equal false-positive and false-negative rates.2 This is a different theorem from Kleinberg's (it is about a thresholded flag, not a score), and is not the one this module's quiz keys. It has an edge the course adds from the algebra behind it: the conflict needs precision above 0% and below 100%, with both outcomes present in each group. If every flag is correct (precision 100%), no group has false alarms, so the false-positive rates match at zero, and the false-negative rates can match too, even though the base rates differ. Chouldechova adds a warning that fits both: "fairness itself ... is a social and ethical concept, not a statistical one."2
At Big Widget: choose, write it down, name who signed
First, a check on section 1. In that made-up table the base rate is the same in every region: 60 of 600, 30 of 300 and 10 of 100 suppliers are high-risk, 10% each. Equal base rates are one of the two cases where Kleinberg's conflict does not apply,6 so the theorem cannot excuse the South's 30% false-positive rate. That is a model defect to fix, not a trade-off forced by mathematics. Use the theorem to explain conflicts that cannot be avoided, never to excuse ones that can.
Now suppose instead that Big Widget's supplier score is used in two regions where the share of suppliers that turn out high-risk differs (a made-up situation). Kleinberg's result says the team cannot have calibration within each region and both balance conditions. The team has to choose, for this use:
- If buyers read the score as a probability and act on it the same way everywhere, calibration within groups (A) protects that reading.
- If the team cares most that low-risk suppliers are scored alike, balance for the negative class (B) equalises their average score across regions.
Do not mistake (B) for equal payment holds. A payment is held when a score crosses the threshold. (B) is about average scores, and two regions can have the same average score among low-risk suppliers while very different shares of them cross the threshold. (Made up: in one region every low-risk supplier scores 0.2; in the other, one in five scores 1 and the rest score 0. Both averages are 0.2. At a threshold of 0.5, none are held in the first region and one in five in the second.) So whichever condition the team keeps, it still measures the false-positive rate at the actual threshold, region by region, as in section 1.
The course's rule (not a rule from the paper): the choice is a decision, not a test result. Record it before release in the evaluation record:
- which score condition the team keeps, which it gives up and by how much;
- the false-positive and false-negative rates at the actual threshold for each named group, with intervals, and the largest gap the signer will accept (the signer sets that number; this module does not);
- why, in terms of the harm;
- the name and role of the person who signed it, and the date. A person signs it, never an agent.
If the base rates are equal, or the model predicts every case perfectly, the conflict disappears; say so in the record if that is your case, and show the numbers. "Nearly perfect" is not the same: with unequal base rates, a small error still leaves the conflict in place.
The misconception this corrects: "There is one fairness test, and a model either passes it or fails it." There are several reasonable tests. For these three, outside the two special cases, they conflict. Passing one is a choice that gives up another, and the record must say which.
4. The three yeses for fairness testing
A yes here is not a blanket licence. It covers one model, one decision, one threshold and the groups named in advance, and a named person gives it.
- Get to Yes (before release): the evaluation record holds the per-group table for named groups at the actual threshold, the proxy check (what predicts the protected attribute, how it was scored, and what the team did about it), and the signed choice of fairness condition with what it gives up. The gate's answer is go ahead, only this far, wait for a person, or no (allow, constrain, hold, deny). If any part is missing, the answer is wait for a person or no, never go ahead; only the named signer can release the hold, never an agent. (The protected-attribute gate as policy is written in 400.9.)
- Stay at Yes (while it runs): re-run the per-group table whenever the data, the threshold or the model changes, and on a schedule (monitoring is 400.4). A model fair at release can drift, and a signature covers the version it was given for.
- Recover to Yes (after harm is found): stop using the feature or the model for that decision, name the owner, re-check the people or firms it affected, fix the cause, and record what changed. Stopping is not the same as being back at yes. You are back when a re-run table shows the gap closed within the signed limit and a person signs again. In the Dutch case, the minister told the AP that nationality had not been used in the childcare-benefit risk model since October 2018.4 That is a stop, as the minister reported it. This module does not use its sources to say what happened to the people affected; the wider inquiry is 400.9.
What you keep, and what you share. Keep two things apart. The evaluation record holds the per-group counts, the proxy results and the reasons. It stays in the controlled test environment, because rates for a small group can still point to individuals. The shareable receipt holds no private content: the decision, hashes of the policy and the evaluation record, the signing key's id and the date. It never holds any individual's protected attribute, prompts, model outputs or other personal data.
What a receipt can show, and what it cannot. A hash or a key id on its own proves no one signed anything: anyone can write those fields down without signing. A receipt shows which record was signed only after someone checks its signature against a key they already trust; then the record hash, compared with the kept record, tells them which bytes that signature covers. How to make and check that signature is 400.8 and 400.9; this module does not supply it. Even a checked receipt shows what was decided and recorded, by the holder of that key. It does not prove the model is fair.
What this module does not know. Every Big Widget number here is made up. No live model, proxy probe or receipt check was run for this module. It does not say what happened to the people affected in the Dutch case. Where your own record has a gap, write "not checked" and name who will check it.
Words to know: cohort, false-positive rate, false-negative rate, flag rate, precision, protected attribute, proxy, redundant encoding, calibration within groups, balance for the positive and negative class, base rate, evaluation record, receipt.
Further learning (not copied; background only)
- University of California, Berkeley, CS 294: Fairness in Machine Learning (Fall 2017), https://fairmlclass.github.io/ : seminar syllabus and readings. No content licence was found; link only.
- Massachusetts Institute of Technology, RES.EC-001: Exploring Fairness in Machine Learning for International Development (MIT OpenCourseWare, CC BY-NC-SA 4.0), https://ocw.mit.edu/courses/res-ec-001-exploring-fairness-in-machine-learning-for-international-development-spring-2020/
- Stanford University, CS 329T: Trustworthy Machine Learning, https://web.stanford.edu/class/cs329t/ : course page. No content licence was found; link only.