Try each exercise first, then open its model answer.
Testing for fairness and proxies · Workbook
Three exercises, one for each thing this module teaches. Every table here is made up, set at Big Widget Company. Do not use real names, suppliers or data from your own work. Show your working for each rate. "Not checked" or "I don't know yet" is an honest answer if you also say who you would ask.
Exercise 1 (400.7-a)
Split the numbers by group. Big Widget's procurement team uses a model to flag suppliers for an extra audit. A flag holds the supplier's next payment until the audit is done. Six months after release, the team checks 1,000 suppliers whose real risk is now known, grouped by supplier size.
| Size | Suppliers | In fact low-risk | Low-risk but flagged | In fact high-risk | High-risk and flagged |
|---|---|---|---|---|---|
| Large | 500 | 450 | 18 | 50 | 40 |
| Medium | 300 | 270 | 16 | 30 | 24 |
| Small | 200 | 180 | 45 | 20 | 16 |
- Work out the overall accuracy and the overall false-positive rate.
- For each size group, work out the false-positive rate, the false-negative rate and the flag rate.
- Which group does the overall picture hide, and how much more likely is a low-risk supplier in that group to have its payment held than one in the Large group?
- A colleague checked only missed risks (false negatives) and says the model is fair. What do you tell them?
- What goes into the evaluation record?
Model answer
- Correct cases: Large 432 + 40 = 472, Medium 254 + 24 = 278, Small 135 + 16 = 151, so 901 of 1,000 (90.1%). Overall false-positive rate: 18 + 16 + 45 = 79 flagged out of 900 low-risk suppliers, about 8.8%. Both look acceptable.
- Per group:
Size False-positive rate False-negative rate Flag rate Large 18/450 = 4.0% 10/50 = 20% 58/500 = 11.6% Medium 16/270 = 5.9% 6/30 = 20% 40/300 = 13.3% Small 45/180 = 25.0% 4/20 = 20% 61/200 = 30.5%
- The Small group. Its false-positive rate is 25.0% against 4.0% for Large, so a low-risk small supplier is about six times as likely to have its payment held. The overall 8.8% hides this because small suppliers are only a fifth of the total. A strong answer also notes that 180 low-risk suppliers is not a large sample, and checks the uncertainty (module 400.6): Wilson 95% intervals computed for this made-up table are about 19.2% to 31.8% for Small and 2.5% to 6.2% for Large, so the gap is far wider than the uncertainty.
- The false-negative rate is 20% in every group, so a check of missed risks alone would not see the problem. Equal on one rate does not mean equal on the others. The harm here is a wrongly held payment, so the rate that matters most is the false-positive rate. The textbook's point applies: an overall error figure can hide poor performance for one group, and reporting rates by group helps catch it.
- The per-group table, not just the overall score; which rate the team treats as the main harm and why (false positives, because of held payments); that the groups were named before the team looked; and the next step: ask why small suppliers differ (different data, a feature that behaves differently for them, or a feature standing in for something it should not).
Exercise 2 (400.7-b)
Direct use or proxy? Part A. Label each of these as direct use or proxy use, and say why.
- The Dutch childcare-benefit risk model in the Autoriteit Persoonsgegevens (AP) decision of 25 November 2021, with the indicator "Dutch nationality or not".
- A made-up Big Widget model that has no nationality column, but uses the country of the supplier's bank.
Part B. Big Widget's supplier model never had an owner-nationality column. For a fairness test, the compliance team holds owner nationality aside (it is not a model input) and compares it with three model features for 300 suppliers: 60 with a foreign owner and 240 with a domestic owner.
| Feature | Foreign-owned (of 60) | Domestic-owned (of 240) |
|---|---|---|
| Invoices in a language other than the local one | 48 | 12 |
| Banks abroad | 30 | 6 |
| Late delivery last year | 9 | 36 |
- For each feature, is it a candidate proxy for owner nationality? Show the numbers.
- The data team says: "We deleted nationality, so the model can't discriminate by it." What do you tell them?
- What should the team do next?
Model answer
- Direct use. Nationality itself was an indicator in the model, at least from March 2016 to October 2018. The AP found that not having Dutch nationality, combined with other indicators, could lead to a higher risk score and a manual check, and judged the processing discriminatory because it made a distinction on nationality without an objective justification. It is not a proxy case.
- Proxy use. Nationality is not an input, but bank country can stand in for it. NIST defines a proxy as "a variable that can stand in for another, usually not directly observable or measurable, variable."
Invoice language: 60 suppliers invoice in another language, and 48 of them (80%) have a foreign owner; of the 240 that invoice in the local language, 12 (5%) do. The rule "other language, so foreign owner" is right for 48 + 228 = 276 of 300 (92%), against 80% for guessing "domestic" every time. A strong candidate proxy.
- Bank abroad: 36 bank abroad, and 30 of them (83%) have a foreign owner; of the 264 that bank at home, 30 (11%) do. The rule is right for 30 + 234 = 264 of 300 (88%). Also a candidate proxy.
- Late delivery: 15% for both groups (9 of 60, 36 of 240). On these numbers it carries no information about nationality on its own, so it is not a candidate proxy by itself. It could still add information in combination with other features, which the joint check in answer 5 tests.
A strong answer adds that the two proxies together may predict nationality better than either alone. The textbook explains that in large feature spaces a sensitive attribute is generally redundant given the other features, so checking one feature at a time is not enough.
- Deleting the column hides nationality from the reader of the data, not from the model. NIST says excluding protected attributes "does not remedy the problem, since the information can be inadvertently inferred in other ways through proxy or latent variables." The textbook says a classifier that loses the attribute "will then find a redundant encoding in terms of the other features."
- Follow the lesson's steps: keep owner nationality held aside for testing under its own access rules (whether and how the company may hold it is a question for counsel); try to predict nationality from all the model's inputs together, scoring that predictor on data it was not fitted on and against the right baseline (here, always guessing "domestic" is right for 240 of 300, 80%; for ROC-AUC the baseline is 0.5). Well above baseline is evidence that the inputs encode nationality, not proof that the model relies on it; near baseline means "no encoding found by this test", not "no proxy". For each strong predictor, ask whether it has a logical link to supplier risk, as NIST's mitigation advises, and consider replacing it with a feature tied to the actual rule, as the AP's alternative indicator did in the Dutch case; then run the Exercise 1 table by owner nationality with and without the candidate proxies, and record what changed.
Exercise 3 (400.7-c)
Choose, write it down, name who signed. Big Widget now uses a supplier risk score (a probability that a supplier turns out high-risk) in two made-up regions. In Region 1, 20% of suppliers turn out high-risk; in Region 2, 5% do. The score is calibrated within each region, and it does not predict perfectly. A flag holds payment, so the procurement director worries most about low-risk suppliers being wrongly held.
- Name Kleinberg, Mullainathan and Raghavan's three conditions, and say which the score meets.
- What does their result say the score must give up, and why can't more training data fix it?
- Which condition is about the suppliers who are in fact low-risk, and what does the team give up if it keeps that one? Does keeping it mean low-risk suppliers have their payments held equally often in both regions?
- Write the record entry for the decision.
- Go back to the Exercise 1 table. Can the team cite this theorem to explain the Small group's 25% false-positive rate?
Model answer
- (A) Calibration within groups: in each region, a score means what it says. (B) Balance for the negative class: suppliers who are in fact low-risk get the same average score in each region. (C) Balance for the positive class: suppliers who are in fact high-risk get the same average score in each region. The score meets (A).
- Their Theorem 1.1: meeting (A), (B) and (C) together requires perfect prediction or equal base rates. Here base rates differ (20% against 5%) and prediction is not perfect, so (B) or (C), or both, must fail. More data does not change the base rates, and the result applies however the scores are computed, by an algorithm or by people. An approximate version (Theorem 1.2) shows that getting close to all three needs the situation to be close to one of the two special cases. A model that predicts nearly perfectly does not escape: with unequal base rates, a small error still leaves the conflict.
- Balance for the negative class (B): low-risk suppliers get the same average score in each region. The score already meets (A), so it cannot hold (B) and (C) too: keeping (B) means giving up (C), or giving up (A) so that a score no longer means the same probability in each region and buyers cannot read it the same way everywhere. (A team might instead keep (A) if buyers act on the score as a probability; either can be defended, but it must be chosen.) No, (B) does not mean equal payment holds. A payment is held when a score crosses the threshold, and (B) is only about averages. The authors call the balance conditions "generalizations" of equal false-positive and false-negative rates, not the same thing. The lesson's made-up example: in one region every low-risk supplier scores 0.2; in the other, one in five scores 1 and the rest score 0. Both averages are 0.2, but at a threshold of 0.5 none are held in the first region and one in five in the second. So whichever condition the team keeps, it still measures the false-positive rate at the actual threshold, region by region, as in Exercise 1.
- A good entry, written before release, holds: which score condition the team keeps, which it gives up and by how much; the false-positive and false-negative rates at the actual threshold for each region, with intervals, and the largest gap the signer will accept (the signer sets that number); why, in terms of the harm (a wrongly held payment); and the name, role and date of the person who signed it. A person signs, never an agent. The paper does not say which to choose (the authors say they do not recommend how to handle the conflict), so the record cannot claim "the maths decided".
- No. In Exercise 1, 50 of 500, 30 of 300 and 20 of 200 suppliers are high-risk: 10% in every group. Equal base rates are one of the two cases where the conflict does not apply, so the theorem cannot excuse the gap. It is a model defect to fix. Use the theorem to explain conflicts that cannot be avoided, never to excuse ones that can.