
Testing for fairness and proxies · Technical gear
You will be able to: compute group-conditional error and outcome rates with uncertainty, and show how an aggregate metric hides a group-level disparity; separate direct use of a protected attribute from proxy use (redundant encoding), and test for proxies by predicting the attribute from the model's inputs; and state Kleinberg, Mullainathan and Raghavan's impossibility result for calibration within groups and balance for the positive and negative classes, its two degenerate escape cases, and why it forces a recorded choice that a named person signs.
As of November 2026. Reviewed and published, version 1.0.0. All Big Widget data here is made up. Notation: A is the protected (sensitive) attribute, Y ∈ {0, 1} the true outcome, R a score, Ŷ = 1{R > t} a thresholded flag, X the model's inputs.
1. Group-conditional metrics
Definitions
For each group a:
- base rate (prevalence) p*ₐ = P(*Y = 1 | A = a)
- false-positive rate FPRₐ = P(Ŷ = 1 | Y = 0, A = a)
- false-negative rate FNRₐ = P(Ŷ = 0 | Y = 1, A = a)
- positive predictive value PPVₐ = P(Y = 1 | Ŷ = 1, A = a)
- acceptance (flag) rate P(Ŷ = 1 | A = a)
Chouldechova defines error rate balance as equality of the group FPRs and of the group FNRs at a threshold, predictive parity as equality of the group PPVs at a threshold, and statistical parity as equality of the flag rates.1 Barocas, Hardt and Narayanan reduce the landscape of observational criteria to three families, which equalise across groups the acceptance rate, the error rates, or the outcome frequency given the score:2
| Independence | Separation | Sufficiency | ||
|---|---|---|---|---|
| R ⊥ A | R ⊥ A \ | Y | Y ⊥ A \ | R |
For a binary classifier, separation is equal true-positive and false-positive rates across groups.2
Why aggregates hide harm
The aggregate FPR is a base-rate-weighted mixture of the group FPRs:
FPR = Σₐ P(A = a | Y = 0) · FPRₐ
so a small group with a large FPRₐ moves the aggregate very little. The textbook's warning: "we typically only test their overall accuracy; so a '5% error' statistic might hide the fact that a model performs terribly for a minority group."2 It also notes the usual cause: with fewer training points from a group that differs systematically on the task, the model works less well for it.2
Worked example (made up)
Big Widget's supplier-risk model flags suppliers for an audit that holds payment (the harm of a false positive). Outcomes are known six months later for 1,000 suppliers.
| Region | n | Y = 0 | FP | Y = 1 | TP | FPR (95% Wilson) | FNR | PPV | Flag rate |
|---|---|---|---|---|---|---|---|---|---|
| North | 600 | 540 | 27 | 60 | 48 | 0.050 [0.035, 0.072] | 0.20 | 0.640 | 0.125 |
| East | 300 | 270 | 14 | 30 | 24 | 0.052 [0.031, 0.085] | 0.20 | 0.632 | 0.127 |
| South | 100 | 90 | 27 | 10 | 8 | 0.300 [0.215, 0.401] | 0.20 | 0.229 | 0.350 |
| All | 1,000 | 900 | 68 | 100 | 80 | 0.076 | 0.20 | 0.541 | 0.148 |
Accuracy is 0.912 overall, 0.935 North, 0.933 East, 0.710 South. The Wilson score interval for k successes in n trials at z = 1.96 is
[ p̂ + z*²/2*n ± z √( p̂(1 − p̂)/n + z*²/4*n*² ) ] / (1 + *z*²/*n), with p̂ = k/n.
The South–North FPR intervals do not overlap, so the disparity is not small-sample noise; for a formal test use a two-proportion or χ² test. NIST notes that in regulated settings "t-tests, χ2-tests, analysis of regression coefficients, and other traditional statistical tests can be used to show a statistically significant difference between ML system outcomes across demographic groups."3
Note what the table does not show: an FNR disparity (all 0.20). A check that monitors only missed risks would pass this model.
Practice. Pre-register the groups, the metric tied to the harm (here FPR) and the minimum group size before looking; report every group's counts, not only rates; report intervals; and keep the per-group table in the evaluation record (400.6 covers evaluation design and intervals).
Misconception corrected: "High overall accuracy means the model is fair to everyone." Accuracy 0.912 overall coexists with FPR 0.300 for the South.
2. Direct use, proxies and redundant encoding
Direct use: A ∈ X. Proxy use: A ∉ X, but some function of X predicts A, and the model exploits that dependence. NIST SP 1270 defines a proxy as "a variable that can stand in for another, usually not directly observable or measurable, variable".3
Case: nationality as a risk-model indicator (direct use)
The Autoriteit Persoonsgegevens (AP) decision of 25 November 2021 fined the Minister of Finance €2,750,000 in total for three unlawful processing operations by Belastingdienst/Toeslagen: retaining the dual nationality of Dutch citizens (at least 1 January 2016 to 30 June 2020); processing applicants' nationality "voor een indicator in zijn zogeheten risico-classificatiemodel" (for an indicator in its so-called risk classification model; at least March 2016 to October 2018); and processing nationality to detect organised fraud (at least 1 January 2016 to February 2019).4
As recorded in the decision: the model assessed all applications in a month using a self-learning algorithm, estimated from a set of indicators the risk that an application was incorrect, and passed the 100 highest-scoring applications that month to an employee for manual review.4 One indicator was "Nederlanderschap/niet-Nederlanderschap"; dual nationality including Dutch counted as Dutch.4 Non-Dutch nationality, combined with other indicators, could produce a higher risk score and a manual check.4 The AP held the risk-model and fraud-detection processing discriminatory, as a distinction on nationality without objective justification, applying a four-part test (comparable cases; a distinction; disadvantage in treatment; no reasonable and objective justification).4 It identified a less intrusive, more accurate indicator tied to the eligibility rule: Dutch nationality, or EU nationality with registration in a Dutch municipality, or non-EU nationality with a valid residence permit.4 The minister stated that nationality had not been used in the childcare-benefit risk model since October 2018.4 The State Secretary for Finance sent the decision to Parliament under a letter of 7 December 2021: "Ik zal mij niet verzetten tegen de boete" (I will not oppose the fine).5
This is direct use (A ∈ X), not a proxy case. In engineering terms the AP's alternative replaces a feature whose information is mostly group membership with one that encodes the decision rule itself.
Why dropping A fails ("no fairness through unawareness")
Barocas, Hardt and Narayanan: "In large feature spaces sensitive attributes are generally redundant given the other features. If a classifier trained on the original data uses the sensitive attribute and we remove the attribute, the classifier will then find a redundant encoding in terms of the other features. This results in an essentially equivalent classifier".2 Many weakly informative features combine into an accurate predictor of A; their illustration is browsing history and gender, and their thought experiment is income predicted from the genome through ancestry, where removing "a few individual genetic markers" does not remove the encoding.2 They add the converse caution: in small, hand-curated feature spaces A may have no good redundant encoding, and even then removing it is not automatically right.2
NIST makes the same point for practice ("this does not remedy the problem, since the information can be inadvertently inferred in other ways through proxy or latent variables"; "race can be inferred through zip code") and warns that proxies used to hide protected attributes "may still reveal sensitive information about individuals or groups".3 It also notes that "depending on the relationship between a protected attribute and the data, certain observational definitions of fairness can increase discrimination."3
Kleinberg, Mullainathan and Raghavan's model builds this in. The risk assignment maps feature vectors σ to bins and "does not have access to the group t", yet "this does not mean that the rule is incapable of exhibiting bias between the two groups"; and since groups may have disjoint feature vectors, σ can implicitly convey "perfect information about a person's group."6
Testing for proxies (worked example, made up)
The owner-nationality attribute A (foreign / domestic) was never an input to Big Widget's supplier model. Compliance holds it aside, access-controlled, for testing only. For 200 suppliers:
| A = foreign | A = domestic | |
|---|---|---|
| bank abroad | 44 | 9 |
| bank at home | 6 | 141 |
P(A = foreign | bank abroad) = 44/53 ≈ 0.83; P(A = foreign | bank at home) = 6/147 ≈ 0.04. The one-feature rule "bank abroad ⇒ foreign" has accuracy 185/200 = 0.925 against a majority-class baseline of 150/200 = 0.75.
A proxy audit (the course's procedure, built on the sources above):
- Probe. Train a classifier g: X → A. The textbook's reasoning is that if A is treated as the target, "there is reason to believe that the remaining features would give a non-trivial classifier for A."2 Split the audit data three ways for the probe itself: a probe-training set to fit g, a validation set to choose its settings, and a probe-test set you do not touch until the final score. "Held out from the supplier model" is not the same as held out from g: a flexible probe scored on its own training rows can memorise identifiers and look like a proxy that is not there. Record how the probe data relates to the main model's training and test sets. Compare each metric with its own baseline: accuracy against majority-class accuracy (0.75 in the table above), ROC-AUC against 0.5 (a constant majority guess scores 0.5, not 0.75). Report the class counts and an interval for the probe score. A probe well above baseline is evidence that X encodes A. It is not by itself proof that the deployed model relies on that encoding, or that the use is unlawful. A probe near baseline is not proof that no proxy exists either: it tests only the features and probe class you tried. Write "no encoding found by this probe", not "no proxy".
- Attribute. Rank features by their contribution to g (single-feature probes, as above, or permutation importance), and list candidate proxies.
- Justify or remove. For each candidate, record whether it has a logical relationship to the prediction target. NIST's interim mitigation for proxy discrimination is "minimizing the number of input variables, and ensuring that there is no strong correlation amongst them and a logical relationship to the prediction target."3
- Re-measure. Re-run section 1's table by A with and without each candidate, and record the effect on both accuracy and group error rates. Removing a proxy can leave its information in the remaining features; re-run the probe.
Holding A for testing is a governance decision with legal constraints that vary by jurisdiction; this module does not teach them.
Misconception corrected: "If nationality is not a column, the model cannot discriminate by nationality." Unawareness is not a fairness criterion; the probe above shows why.
3. The impossibility result
Figure data as text
Three fairness conditions
| Condition | What it shows |
|---|---|
| (A) [s1] | Calibration within groups. In each group, among people given score v, a v fraction are in fact positive. |
| (B) [s1] | Balance for the negative class. People who are in fact negative get the same average score in each group. |
| (C) [s1] | Balance for the positive class. People who are in fact positive get the same average score in each group. |
| THEOREM 1.1 [s1] | All three only if. Prediction is perfect, or the groups have equal base rates. |
Kleinberg, Mullainathan and Raghavan (arXiv 2016; ITCS 2017)
Setup: each person has a feature vector σ and belongs to group t ∈ {1, 2}; p*σ is the fraction of people with σ in the positive class, and given σ the group carries no further information. A risk assignment places a fraction *X*σ*b of people with σ in bin b, labelled with score *v*ᵦ.6 The conditions:6
- (A) Calibration within groups: for each group t and bin b, the expected number of positives from t in b is a v*ᵦ fraction of the expected number of people from *t in b.
- (B) Balance for the negative class: the average score of group-1 negatives equals that of group-2 negatives.
- (C) Balance for the positive class: the same for positives.
(B) and (C) "can be viewed as generalizations of the notions that both groups should have equal false negative and false positive rates"; (A) says that, conditioned on the bin, the likelihood of being positive "is independent of the group to which the individual belongs".6 In the textbook's terms that is sufficiency, Y ⊥ A | R.2
Theorem 1.1. "Consider an instance of the problem in which there is a risk assignment satisfying fairness conditions (A), (B), and (C). Then the instance must either allow for perfect prediction (with *p*σ equal to 0 or 1 for all σ) or have equal base rates."6
Both escape cases are constructive. With perfect prediction, put every σ with *p*σ = 0 in a bin scored 0 and every σ with *p*σ = 1 in a bin scored 1. With equal base rates, put everyone in a single bin scored at the common base rate; this satisfies all three conditions while being "not a particularly informative risk assignment".6
Theorem 1.2 extends this to ε-approximate versions: there is a continuous f with f(x) → 0 as x → 0 such that any instance with a risk assignment satisfying ε-approximate (A), (B) and (C) must satisfy the f(ε)-approximate version of perfect prediction or of equal base rates.6 The result holds "regardless of how the risk assignment is computed", and the authors state they do not recommend how conflicts should be resolved.6 The paper treats two groups for exposition and says the definitions extend directly to more than two.6 Applied to COMPAS, the authors read the critique as showing violation of (B) and (C) and the counter-arguments as establishing (A).6
Related results at a threshold
Chouldechova (2017). For a thresholded flag, the confusion-matrix identity
FPR = [p / (1 − p)] · [(1 − PPV) / PPV] · (1 − FNR)
holds within each group. (Derivation: FP = TP(1 − PPV)/PPV, TP = p·n·(1 − FNR) and the negatives number (1 − p)·n; divide.) Chouldechova's conclusion: if PPV is equal across groups but p differs, FPR and FNR cannot both be equal.1 Read it with the identity's conditions, which are this module's reading of the algebra, not the paper's words: both outcome classes present in each group (0 < p < 1), and 0 < PPV < 1 (so some flags are wrong and some positives are caught). Then the factor (1 − PPV)/PPV is positive, and with equal PPV and equal FNR, FPR must scale with p/(1 − p). At the boundary PPV = 1 the factor is zero: FPR = 0 in every group whatever p, and FNR can match too. A made-up example: group 1 has TP = 160, FN = 40, FP = 0, TN = 800 (p = 0.20); group 2 has TP = 80, FN = 20, FP = 0, TN = 900 (p = 0.10). PPV is 1 and 1, FPR 0 and 0, FNR 0.20 and 0.20, and neither classifier is perfect. This is a thresholded result about precision; do not treat it as Kleinberg's score calibration (A), which has different conditions. On the Broward County data (n = 6,150), the recidivism base rates were 51% for Black and 39% for White defendants; COMPAS was approximately calibrated and met predictive parity for cut-offs of 4 or above, but failed error rate balance.1 Chouldechova also stresses that "fairness itself ... is a social and ethical concept, not a statistical one."1 This module's quiz keys Kleinberg's result, not this one.
Barocas, Hardt and Narayanan prove the pairwise versions for the three families: if A and Y are dependent, sufficiency and independence cannot both hold (Proposition 2); with binary Y, A dependent on Y, and an informative score (R dependent on Y), independence and separation cannot both hold (Proposition 3); and with all events of positive probability and A dependent on Y, separation and sufficiency cannot both hold (Proposition 4), from which "Optimal scores generally violate separation."2 Every one of the three needs A dependent on Y, that is, unequal base rates. With equal base rates the pairwise conflicts need not arise: if A is independent of a binary Y and R is Y flipped with some independent probability, R is informative and imperfect, yet both independence and separation hold.
Check on the worked example. Section 1's made-up table has the same base rate in every region (60/600 = 30/300 = 10/100 = 0.10). With equal base rates the impossibility does not bite, and the identity reproduces each group's FPR: for the South, (0.1/0.9)·(0.771/0.229)·0.8 ≈ 0.30, for the North (0.1/0.9)·(0.36/0.64)·0.8 = 0.05. The South's high FPR comes with low PPV (0.229) at an equal base rate: it is a model defect to fix, not a trade-off forced by mathematics. Use the theorem to explain unavoidable conflicts, never to excuse avoidable ones.
The recorded choice
When base rates differ and prediction is not perfect, the team must choose which condition to keep for this decision.
Score conditions are not threshold error rates. (B) and (C) constrain mean scores among negatives and positives. They generalise equal error rates, but they do not fix the probability of crossing the threshold that actually triggers the action. A made-up counterexample at threshold 0.5: every negative in group 1 scores 0.2; in group 2 a negative scores 1 with probability 0.2 and 0 otherwise. Both negative-class means are 0.2, so (B) holds, yet the FPRs at the threshold are 0 and 0.2. So keeping (B) does not by itself fix a payment-hold disparity. Whatever score condition is kept, measure FPRₐ (and FNRₐ) at the operational threshold too, and record both.
The course's rule (not a claim of the papers): before release, record
- the score condition kept, and the conditions given up and by how much (the per-group numbers);
- the operational per-group rates at the actual threshold, with intervals, and the tolerance the signer accepts for the gap (the signer sets it; this module does not supply a number);
- the reason, in terms of the harm;
- the name, role and date of the person who signed. A person signs, never an agent.
Theorem 1.1's exact escapes are equal base rates or perfect prediction. A small nonzero error does not make the exact conflict disappear when base rates differ. If you rely on Theorem 1.2's approximate version, record the measured departures from each condition and the tolerances, and do not mark the conflict "absent" because the error is small. If base rates are equal, record the per-group counts that show it.
Misconception corrected: "There is one fairness test, and a model either passes it or fails it." Outside the stated special cases, calibration within groups and the two balance conditions cannot all hold, and the pairwise results above conflict whenever A depends on Y. So "passing" is relative to a chosen criterion, and the record must say which.
4. The three yeses
A yes here is conditional: one model, one decision, one threshold, the groups named in advance.
- Get to Yes (before release): the evaluation record contains the pre-registered per-group table with intervals at the operational threshold, the proxy audit (probe splits, probe score against its baseline, candidate proxies, decisions), and the signed criterion choice with its tolerance. The gate's answers are go ahead, only this far (allow · constrain), wait for a person (hold) or no (deny). With any part missing, the answer is hold until it exists, or deny; only the named signer can release a hold, never an agent. (The protected-attribute gate as policy is written in 400.9.)
- Stay at Yes (while it runs): re-run the table and the probe on each retrain, threshold change or input shift (400.4 monitoring). A signed choice covers the data it was signed on, not every later version.
- Recover to Yes (after harm is found): stop the feature or model for the affected decision, name the owner, re-review affected cases, fix, re-test and record. Stopping is not the same as being back at yes: you are back when a re-run table shows the gap closed within the signed tolerance and a person signs again. In the Dutch case the minister stated that nationality had not been used in the risk model since October 2018.4 That is a stop, as the minister reported it. This module's sources are not used to show what happened to the people affected; the wider inquiry is 400.9.
Records and receipts. Keep two things apart. The evaluation record holds the per-group counts, probe results and reasons; it stays in the controlled test environment, because aggregate rates for a small group can still point to individuals. The shareable receipt holds no private content: the decision, the policy and record hashes, the signing key's id and the date. It never holds any individual's protected attribute, prompts, model outputs, payloads or other personal data.
What the receipt proves, and what it does not. A digest binds bytes; it does not show a signing event. A record hash and a key id can be assembled without any signature, so on their own they prove no signer, no signer identity and no authorisation. Only after the receipt's signature verifies against trusted key material does comparing its authenticated record hash with the retained record identify the bytes that signature covers. Canonical serialisation, signatures, pinned keys and checkpoints are 400.8 and offline verification is 400.9; this module does not implement them. Even a verified receipt shows what the key holder signed, not that the decision was right or the model fair.
Not checked here. All tables are synthetic. No model, probe, threshold sweep or receipt verification was run for this module, and its sources are not used to show what happened to the people affected in the Dutch case. In your own record, mark any gap "not checked" and name its owner and next check.
Further learning (not copied; background only)
- University of California, Berkeley, CS 294: Fairness in Machine Learning (Fall 2017), https://fairmlclass.github.io/ : seminar syllabus and readings. No content licence was found; link only.
- Massachusetts Institute of Technology, RES.EC-001: Exploring Fairness in Machine Learning for International Development (MIT OpenCourseWare, CC BY-NC-SA 4.0), https://ocw.mit.edu/courses/res-ec-001-exploring-fairness-in-machine-learning-for-international-development-spring-2020/
- Stanford University, CS 329T: Trustworthy Machine Learning, https://web.stanford.edu/class/cs329t/ : course page. No content licence was found; link only.