Chapter 11: Algorithmic Fairness and Ethics
Introduction
Warm-up: A Fair Division Game
Two players take turns picking indivisible items, one at a time. Each item card shows how much each player values it. The four items on the right are our running example.
- If your opponent always grabs their favorite item, should you always grab yours?
- Is there always a split in which everyone values their own share at least as much as the other's?
Introduction
Learning Outcomes
Introduction
Why Do Algorithmic Decisions Matter?
- Algorithms now make or steer high-stakes decisions: lending, hiring, medical diagnosis, social media feeds, criminal justice.
- Unfair or unethical decisions harm individuals and erode public trust in institutions and technology.
- The game-theory link: a decision rule is a mechanism, and its stakeholders (applicants, institutions, society) have conflicting interests.
Introduction
Where Do Bias and Discrimination Come From?
Fairness Criteria
The Confusion Matrix
Lending example: Y = does the applicant actually repay, Ŷ = does the model approve; n = TP + FP + FN + TN
| Rate | Definition | Question |
|---|---|---|
| Selection rate | (TP + FP) / n | What fraction is approved? |
| TPR (true positive rate) | TP / (TP + FN) | How many future repayers were approved? |
| FPR (false positive rate) | FP / (FP + TN) | How many future defaulters were approved? |
| PPV (precision) | TP / (TP + FP) | How many of the approved repay? |
| Base rate p | (TP + FN) / n | Share of repayers in the group |
Fairness Criteria
Example: Two Groups, One Model
| group | score | people | repay | default | decision (threshold 0.5) |
|---|---|---|---|---|---|
| A | 0.80 | 50 | 40 | 10 | approve |
| A | 0.20 | 50 | 10 | 40 | reject |
| B | 0.80 | 20 | 16 | 4 | approve |
| B | 0.20 | 80 | 16 | 64 | reject |
100 applicants per group; the model outputs two scores: 0.8 ("low risk") and 0.2 ("high risk")
The share of repayers (the base rate) is 50% in group A and 32% in group B. The model uses a 0.5 threshold.
Score 0.8 is approved: in A, 40 repay (TP) + 10 default (FP); in B, 16 TP + 4 FP.
Score 0.2 is rejected: in A, 10 would-be repayers are rejected (FN) and 40 are correctly rejected (TN); in B, 16 FN + 64 TN.
Both confusion matrices are ready. Accuracy is 80% in both groups ((40+40)/100 and (16+64)/100). Is the model "fair"? Let us check the criteria one by one.
Fairness Criteria
Demographic Parity Step by Step
Demographic parity: P(Ŷ=1 | A) = P(Ŷ=1 | B). Approved: 40 + 10 = 50 in A, 16 + 4 = 20 in B.
Selection rates: A = 50/100 = 0.50, B = 20/100 = 0.20. A gap of 0.30 ⇒ demographic parity fails.
Ratio: 0.20 / 0.50 = 0.40 < 0.8, below even the "four-fifths rule" used in US hiring. On its own this is not a legal finding of discrimination.
But the groups differ: 50 actual repayers in A, 32 in B. Forcing demographic parity means approving defaulters in B or rejecting repayers in A.
Fairness Criteria
Equalized Odds Step by Step
Equalized odds (Hardt et al., 2016): error rates must match conditional on the true outcome: TPRA = TPRB and FPRA = FPRB.
TPR (approval among repayers): A = 40/(40+10) = 0.80, B = 16/(16+16) = 0.50. Half of B's creditworthy applicants are rejected!
FPR (approval among defaulters): A = 10/(10+40) = 0.20, B = 4/(4+64) = 0.059.
TPR gap 0.30, FPR gap 0.141 ⇒ equalized odds fails. Requiring equal TPR alone is called equal opportunity.
Fairness Criteria
Predictive Parity Step by Step
Predictive parity: the repayment rate among approved applicants (PPV) must be equal across groups. This is the lender's point of view.
PPV: A = 40/(40+10) = 0.80, B = 16/(16+4) = 0.80 ⇒ predictive parity holds.
Rejections too (NPV: share of rejected who really default = TN/(TN + FN)): A = 40/50 = 0.80, B = 64/80 = 0.80. "Approve" and "reject" mean the same thing in both groups.
Fairness Criteria
Calibration Step by Step
| group | score s | people | repay | observed rate | equals s? |
|---|---|---|---|---|---|
| A | 0.80 | 50 | 40 | ? | |
| A | 0.20 | 50 | 10 | ? | |
| B | 0.80 | 20 | 16 | ? | |
| B | 0.20 | 80 | 16 | ? |
Calibration within groups: among people with score s, the repayment rate must be s in every group. A score should mean the same risk regardless of group.
| group | score s | people | repay | observed rate | equals s? |
|---|---|---|---|---|---|
| A | 0.80 | 50 | 40 | 40/50 = 0.80 | yes |
| A | 0.20 | 50 | 10 | ? | |
| B | 0.80 | 20 | 16 | ? | |
| B | 0.20 | 80 | 16 | ? |
Group A, score 0.8: 40 of 50 repay ⇒ 0.80.
| group | score s | people | repay | observed rate | equals s? |
|---|---|---|---|---|---|
| A | 0.80 | 50 | 40 | 40/50 = 0.80 | yes |
| A | 0.20 | 50 | 10 | 10/50 = 0.20 | yes |
| B | 0.80 | 20 | 16 | ? | |
| B | 0.20 | 80 | 16 | ? |
Group A, score 0.2: 10 of 50 repay ⇒ 0.20.
| group | score s | people | repay | observed rate | equals s? |
|---|---|---|---|---|---|
| A | 0.80 | 50 | 40 | 40/50 = 0.80 | yes |
| A | 0.20 | 50 | 10 | 10/50 = 0.20 | yes |
| B | 0.80 | 20 | 16 | 16/20 = 0.80 | yes |
| B | 0.20 | 80 | 16 | ? |
Group B, score 0.8: 16 of 20 repay ⇒ 0.80.
| group | score s | people | repay | observed rate | equals s? |
|---|---|---|---|---|---|
| A | 0.80 | 50 | 40 | 40/50 = 0.80 | yes |
| A | 0.20 | 50 | 10 | 10/50 = 0.20 | yes |
| B | 0.80 | 20 | 16 | 16/20 = 0.80 | yes |
| B | 0.20 | 80 | 16 | 16/80 = 0.20 | yes |
Group B, score 0.2: 16 of 80 repay ⇒ 0.20. The model is calibrated in both groups: the score is an honest probability.
Fairness Criteria
Code Trace: metrics(TP, FP, FN, TN)
The function is called with group A's matrix.
n = 40 + 10 + 10 + 40 = 100 applicants.
Selection rate = (40 + 10) / 100 = 0.5 (compared for demographic parity).
TPR = 40 / (40 + 10) = 0.8.
FPR = 10 / (10 + 40) = 0.2.
PPV = 40 / (40 + 10) = 0.8.
The four rates are returned as a Python tuple and assigned to A.
Same computation for B: selection 20/100, TPR 16/32, FPR 4/68 ≈ 0.059, PPV 16/20.
B = (0.2, 0.5, 0.0588…, 0.8).
No demographic parity: 0.5 ≠ 0.2.
i = 1, 2, 3: TPR differs ⇒ no equal opportunity; FPR differs too ⇒ no equalized odds; PPV equal ⇒ predictive parity holds.
def metrics(TP, FP, FN, TN): n = TP + FP + FN + TN sel = (TP + FP) / n # demographic parity tpr = TP / (TP + FN) # equalized odds fpr = FP / (FP + TN) ppv = TP / (TP + FP) # predictive parity return sel, tpr, fpr, ppv A = metrics(40, 10, 10, 40) B = metrics(16, 4, 16, 64) NAMES = ["sel", "TPR", "FPR", "PPV"] for i, name in enumerate(NAMES): print(name, round(A[i], 3), round(B[i], 3))
Fairness Criteria
All Criteria, One Model: The Scorecard
| Criterion | Must be equal | A | B | Holds? |
|---|---|---|---|---|
| Demographic parity | P(Ŷ=1) | 0.50 | 0.20 | no |
| Equal opportunity | TPR = P(Ŷ=1 | Y=1) | 0.80 | 0.50 | no |
| Equalized odds | TPR and FPR = P(Ŷ=1 | Y=0) | 0.80; 0.20 | 0.50; 0.059 | no |
| Predictive parity | PPV = P(Y=1 | Ŷ=1) | 0.80 | 0.80 | yes |
| Calibration | P(Y=1 | score = s) = s | 0.8→0.8; 0.2→0.2 | 0.8→0.8; 0.2→0.2 | yes |
Fairness Criteria
Beyond Group Fairness: Individual Fairness
distance between Ece and Nur d = 0.05; L·d = 0.1
- Dwork et al. (2012): "treat similar people similarly": |f(x) − f(y)| ≤ L · d(x, y) (the Lipschitz condition). f(x) is person x's score; d(x, y) measures how different two people are; L is the largest allowed slope. Two people's scores may differ by at most L times their distance.
- The red step model gives Ece and Nur, on either side of the threshold, 0.2 and 0.8: |Δf| = 0.6 > 0.1 ⇒ violation. The green model satisfies the condition.
- Group criteria equalize averages: approving random people from B achieves demographic parity but is arbitrary for individuals.
Incompatibility
Why Can't the Criteria Hold Together?
Chouldechova (2017): the criteria all come from the same four numbers, so they are linked. Start with the base rate p.
Write PPV, the subject of predictive parity, in terms of TPR, FPR and p.
Solving for FPR gives an identity linking the three rates (FPR, TPR, PPV) and the base rate.
Check the identity on group A.
Check it on group B: it matches the 4/68 in the table.
Conclusion: with different base rates, predictive parity and equalized odds cannot both hold (except in extreme cases such as FPR = 0 and PPV = 1).
Incompatibility
What If We Enforce Equalized Odds?
Imagine a (hypothetical) better model that achieves these error rates for B. Today's model (scores 0.8 and 0.2 only) cannot reach them by moving the threshold or by randomizing approvals.
Fixing the error rates fixes the top and bottom rows of the confusion matrix.
B's approval rate rises from 0.2 to 0.39.
The repayment rate among the approved falls.
Which criterion should we keep? Context (the kind of harm, the law) decides, not mathematics.
Incompatibility
Calibration and Balance: Kleinberg's Theorem
The score-level analogue of equalized odds: balance.
In our calibrated model, B's repayers get a lower average score.
No balance among defaulters either. Calibrated, yet unbalanced.
In our example the base rates differ (0.5 and 0.32) and prediction is imperfect: the criteria must conflict.
Incompatibility
Change the Assumption: Which Fairness Criterion?
| group | approve: repay / default | reject: repay / default | base rate |
|---|---|---|---|
| A | 40 / 10 | 10 / 40 | 0.50 |
| B | 16 / 4 | 16 / 64 | 0.32 |
| Criterion | A | B | Holds? |
|---|---|---|---|
| Demographic parity (approvals) | 0.50 | 0.20 | no |
| Equalized odds (TPR; FPR) | 0.80; 0.20 | 0.50; 0.059 | no |
| Predictive parity (PPV) | 0.80 | 0.80 | yes |
| group | approve: repay / default | reject: repay / default | base rate |
|---|---|---|---|
| A | 40 / 10 | 10 / 40 | 0.50 |
| B | 16 / 4 | 16 / 64 | 0.32 |
| Criterion | A | B | Holds? |
|---|---|---|---|
| Demographic parity (approvals) | 0.50 | 0.20 | no |
| Equalized odds (TPR; FPR) | 0.80; 0.20 | 0.50; 0.059 | no |
| Predictive parity (PPV) | 0.80 | 0.80 | yes |
| group | approve: repay / default | reject: repay / default | base rate |
|---|---|---|---|
| A | 40 / 10 | 10 / 40 | 0.50 |
| B | 16 / 4 | 16 / 64 | 0.32 |
| Criterion | A | B | Holds? |
|---|---|---|---|
| Demographic parity (approvals) | 0.50 | 0.20 | no |
| Equalized odds (TPR; FPR) | 0.80; 0.20 | 0.50; 0.059 | no |
| Predictive parity (PPV) | 0.80 | 0.80 | yes |
Incompatibility
A Real Case: The COMPAS Debate
ProPublica (2016), Broward County data: percentages
Incompatibility
The Fairness–Accuracy Trade-off
- Accuracy alone does not show on whom the errors fall: in our example accuracy is 80% in both groups, yet TPR is 0.8 vs 0.5.
- A fairness constraint may raise total error; the real choice is which errors are acceptable (wrongful rejection or wrongful approval?).
Incompatibility
Code Trace: Equal Opportunity by Randomization
q = 0: only score 0.8 is approved (today's model). TP = 16, FP = 4.
TPR = 16/32 = 0.5; FPR = 4/68 ≈ 0.06; accuracy (16 + 64)/100 = 0.80.
q = 0.2: 20% of the 80 people scored 0.2 are approved. TP = 16 + 16·q = 19.2, FP = 4 + 64·q = 16.8.
TPR rose to 0.6 but accuracy fell to 0.704: four of every five new approvals are defaulters.
q = 0.4: 40% of the 80 people scored 0.2 are approved. TP = 16 + 16·q = 22.4, FP = 4 + 64·q = 29.6.
TPR 0.7; FPR jumps to 0.44, accuracy 0.608.
q = 0.6: 60% of the 80 people scored 0.2 are approved. TP = 16 + 16·q = 25.6, FP = 4 + 64·q = 42.4.
TPR_B = 0.8 = TPR_A: equal opportunity achieved. The price: accuracy 0.512, FPR 0.62.
For equal opportunity, 60% of those scored 0.2 were approved at random: about 38 of the 64 defaulters get loans. FPR is more than three times A's (0.2), so equalized odds still fails.
def group_b(q): # approve score 0.2 w.p. q TP = 16 + 16 * q FP = 4 + 64 * q tpr = TP / 32 fpr = FP / 68 hits = TP + (68 - FP) # TP + TN return tpr, fpr, round(hits / 100, 3) for q in [0, 0.2, 0.4, 0.6]: tpr, fpr, d = group_b(q) print(q, round(tpr, 2), round(fpr, 2), d)
Fair Division
Fair Division: Cake Cutting
- A divisible resource (the cake = the interval [0, 1]: time, land, compute) is to be shared among n people.
- Each person has a valuation function vi: the whole cake is worth 1, but people value different parts differently.
- The algorithm may only ask queries (the Robertson–Webb model): "what is this piece worth to you?" and "where does a piece of a given value end?". Xi is the piece person i receives; vi(Xi) is its value to person i.
Fair Division
Cut and Choose Step by Step
Ali values the whole cake uniformly. Ayşe likes chocolate 3 times as much as vanilla: vAyşe(vanilla) = 1/4, vAyşe(chocolate) = 3/4.
1. Ali cuts into two pieces equal in his eyes ⇒ x = 1/2, each piece worth 1/2 to Ali.
2. Ayşe evaluates: left piece 1/4, right piece 3/4.
3. Ayşe chooses the more valuable right piece (chocolate).
Result: Ali 1/2 ≥ 1/2, Ayşe 3/4 ≥ 1/2 ⇒ proportional; Ali 1/2 = 1/2, Ayşe 3/4 > 1/4 ⇒ envy-free.
Swap roles: Ayşe cuts at x = 2/3 (1/2 – 1/2 in her eyes). Ali picks the left piece and gets 2/3; Ayşe gets 1/2. The chooser has the edge over the cutter.
Fair Division
Code Trace: A Cut Query by Bisection
A Robertson–Webb cut query: "where is x so that [0, x] is worth 1/2 to you?" Ayşe's valuation is increasing in x, so bisection finds it.
Query 1: mid = 0.5: all of the vanilla is worth 1/4 = 0.25 < 0.5 to Ayşe ⇒ lo = 0.5. The interval halves (width 1/2).
Query 2: v(0.75) = 0.625 ≥ 0.5 ⇒ the cut is here or to the left: hi = mid. The interval halves (width 1/4).
Query 3: v(0.625) = 0.4375 < 0.5 ⇒ the cut is further right: lo = mid. The interval halves (width 1/8).
Query 4: v(0.6875) = 0.5312 ≥ 0.5 ⇒ the cut is here or to the left: hi = mid. The interval halves (width 1/16).
Query 5: v(0.6562) = 0.4844 < 0.5 ⇒ the cut is further right: lo = mid. The interval halves (width 1/32).
Query 6: v(0.6719) = 0.5078 ≥ 0.5 ⇒ the cut is here or to the left: hi = mid. The interval halves (width 1/64).
Query 7: v(0.6641) = 0.4961 < 0.5 ⇒ the cut is further right: lo = mid. The interval halves (width 1/128).
Query 8: v(0.668) = 0.502 ≥ 0.5 ⇒ the cut is here or to the left: hi = mid. The interval halves (width 1/256).
After 8 queries the interval is 1/256 wide; its midpoint is 0.666. The true cut is 2/3: where Ayşe would cut if she were the cutter.
def v_ayse(x): # value of [0, x] if x <= 0.5: return x / 2 return 0.25 + 1.5 * (x - 0.5) def cut_point(v, target, steps): lo, hi = 0.0, 1.0 for _ in range(steps): mid = (lo + hi) / 2 if v(mid) < target: lo = mid else: hi = mid return (lo + hi) / 2 print(round(cut_point(v_ayse, 0.5, 8), 3))
Fair Division
Why Is Cut and Choose Fair?
- The cutter does not know which piece will be left, so the guarantee is the smaller piece. It peaks at the cutter's own halfway point (1/2): cutting honestly is the maximin strategy.
- The chooser takes the piece that is larger in her eyes: worth ≥ 1/2 and no less than the other, wherever the cut is.
- A cutter who knows the chooser's preferences can shift the cut and get more than 1/2, but the guarantee is 1/2.
Fair Division
Proportional but Not Envy-Free: Three People
| people | piece X | piece Y | piece Z | gets | ≥ 1/3? |
|---|---|---|---|---|---|
| Ali | 0.40 | 0.30 | 0.30 | ||
| Ayşe | 0.20 | 0.35 | 0.45 | ||
| Cem | 0.30 | 0.30 | 0.40 |
Three people, three pieces (X, Y, Z). The table shows each person's value for each piece; each row sums to 1.
| people | piece X | piece Y | piece Z | gets | ≥ 1/3? |
|---|---|---|---|---|---|
| Ali | 0.40 | 0.30 | 0.30 | X: 0.40 | yes |
| Ayşe | 0.20 | 0.35 | 0.45 | Y: 0.35 | yes |
| Cem | 0.30 | 0.30 | 0.40 | Z: 0.40 | yes |
Allocation: Ali → X, Ayşe → Y, Cem → Z. Everyone gets at least 1/3 ≈ 0.33 by their own measure ⇒ proportional.
| people | piece X | piece Y | piece Z | gets | ≥ 1/3? |
|---|---|---|---|---|---|
| Ali | 0.40 | 0.30 | 0.30 | X: 0.40 | yes |
| Ayşe | 0.20 | 0.35 | 0.45 | Y: 0.35 | yes |
| Cem | 0.30 | 0.30 | 0.40 | Z: 0.40 | yes |
But to Ayşe, Cem's piece Z (0.45) beats her own Y (0.35) ⇒ Ayşe envies Cem. Proportional, but not envy-free.
Fair Division
Last Diminisher Step by Step
Three people; the cake has four quarters. Bars show each person's value per quarter (total 1). Ali is uniform; Ayşe likes the right, Cem the left.
Ali cuts a piece worth 1/3 to him: [0, 1/3].
Ayşe values it at 1/6, less than 1/3, so she leaves it alone ("passes").
To Cem the piece is worth 1/2 > 1/3: he trims it, moving the cut to 5/24 (exactly 1/3 for Cem).
The last trimmer (Cem) takes the piece and leaves. For those who did not trim, the piece is worth ≤ 1/3, so the rest of the cake is worth ≥ 2/3 to them.
The remaining [5/24, 1] is split between two by cut and choose: Ali cuts at 29/48, Ayşe picks the more valuable right piece (23/40).
Everyone got ≥ 1/3 by their own measure ⇒ proportional. O(n²) queries for n people. Not envy-free, though: Cem values Ali's piece at 9/20 (his own 1/3).
Fair Division
Moving Knife (Dubins–Spanier) and Even–Paz
| people | round 1: v([0, x]) = 1/3 | round 2: v([5/24, x]) = 1/3 | gets |
|---|---|---|---|
| Cem | x = 5/24 ≈ 0.21 (stop!) | — | [0, 5/24]: 1/3 |
| Ali | x = 1/3 ≈ 0.33 | x = 13/24 ≈ 0.54 (stop!) | [5/24, 13/24]: 1/3 |
| Ayşe | x = 19/36 ≈ 0.53 | x = 43/72 ≈ 0.60 | [13/24, 1]: 13/20 |
Values as in the last diminisher example (per quarter: Ali 0.25; Ayşe 0.1–0.4; Cem 0.4–0.1). The first piece matches last diminisher: Cem stops the knife first. Everyone gets ≥ 1/3 ⇒ proportional. Not envy-free: Cem values Ali's piece at 2/5 > 1/3.
Fair Division
Indivisible Goods: Round-Robin and EF1
| people | m1 | m2 | m3 | m4 | own | other's |
|---|---|---|---|---|---|---|
| Ali | 7 | 5 | 3 | 1 | 0 | 0 |
| Ayşe | 8 | 6 | 1 | 1 | 0 | 0 |
4 indivisible goods, 2 people; the table shows values. Round-robin: on each turn, the next person takes the most valuable remaining good.
| people | m1 | m2 | m3 | m4 | own | other's |
|---|---|---|---|---|---|---|
| Ali | 7 | 5 | 3 | 1 | 7 | 0 |
| Ayşe | 8 | 6 | 1 | 1 | 0 | 8 |
Ali takes m1 (7).
| people | m1 | m2 | m3 | m4 | own | other's |
|---|---|---|---|---|---|---|
| Ali | 7 | 5 | 3 | 1 | 7 | 5 |
| Ayşe | 8 | 6 | 1 | 1 | 6 | 8 |
Ayşe takes m2 (6) from what is left.
| people | m1 | m2 | m3 | m4 | own | other's |
|---|---|---|---|---|---|---|
| Ali | 7 | 5 | 3 | 1 | 10 | 5 |
| Ayşe | 8 | 6 | 1 | 1 | 6 | 9 |
Ali takes m3 (3).
| people | m1 | m2 | m3 | m4 | own | other's |
|---|---|---|---|---|---|---|
| Ali | 7 | 5 | 3 | 1 | 10 | 6 |
| Ayşe | 8 | 6 | 1 | 1 | 7 | 9 |
Ayşe takes m4. Ayşe: her own bundle 7, Ali's bundle 9 in her eyes ⇒ envy! But removing m1 from Ali's bundle gives 1 ≤ 7: EF1 (envy-free up to one good).
With indivisible goods, envy-freeness is not always possible (one car, two siblings). With additive values, round-robin in which everyone picks their favorite remaining good always yields an EF1 allocation.
Fair Division
Back to the Game: Fair Division
The opening items (P1 / P2 value): Cake 5/1, Book 4/9, Headphones 3/8, Ball 1/1.
| P1's plan | Pick order | P1 | P2 |
|---|---|---|---|
| always the best | Cake, Book, Phones, Ball | 8 | 10 |
| Book first | Book, Phones, Cake, Ball | 9 | 9 |
P2 values the Cake at 1, so it will still be there later. In round-robin, taking your favorite item every turn is not always the best strategy.
Under the second plan, P2 values P1's bundle at 9 + 1 = 10 and its own at 9: envy. Remove the Book and 1 ≤ 9: the allocation is still EF1. The theory's guarantee: if both sides always pick their favorite, round-robin yields EF1.
In practice: data from Harvard Business School's draft-based course allocation showed students making exactly this kind of strategic pick, at a cost in welfare (Budish and Cantillon, 2012).
Fair Division
Which Allocation? Three Social Welfare Criteria
Maximize social welfare SW = Σ vi: efficiency.
Maximize the smallest vi (maximin).
Maximize vAli × vAyşe; scaling one person's values by 10 does not change the answer.
| Allocation | Ali | Ayşe | sum | min | product | envy-free? |
|---|---|---|---|---|---|---|
| Round-robin: Ali {m1, m3}, Ayşe {m2, m4} | 10 | 7 | 17 | 7 | 70 | no, EF1 |
| Utilitarian: Ali {m3, m4}, Ayşe {m1, m2} | 4 | 14 | 18 | 4 | 56 | no, not even EF1 |
| Nash / egalitarian (not unique): Ali {m2, m3, m4}, Ayşe {m1} | 9 | 8 | 17 | 8 | 72 | yes |
Fair Division
The Price of Fairness and Pareto Efficiency
| Allocation | Ali | Ayşe | sum | envy-free? |
|---|---|---|---|---|
| Ali {m3, m4}, Ayşe {m1, m2} | 4 | 14 | 18 | no |
| Ali {m3}, Ayşe {m1, m2, m4} | 3 | 15 | 18 | no |
| Ali {m2, m3, m4}, Ayşe {m1} | 9 | 8 | 17 | yes |
| Ali {m2, m3}, Ayşe {m1, m4} | 8 | 9 | 17 | yes |
Ethics and Governance
Ethical Principles and Standards
| GDPR (EU, 2018), Art. 22 | Right to contest, and to human intervention in, significant decisions based solely on automated processing |
| EU AI Act (2024) | Risk-based: credit scoring and hiring are "high risk" |
| KVKK (Turkey), OECD AI Principles | Rules for processing personal data; shared principles for trustworthy AI |
Ethics and Governance
Strategic Design and Incentives
- When an algorithm sets a rule, people respond to it: those being classified are players too (strategic classification).
- Goodhart's law: when a measure becomes a target, it ceases to be a good measure (click counts → clickbait).
- Giving users the right incentives and preventing malicious manipulation (fake reviews, bot accounts) is part of the design.
Ethics and Governance
Strategic Classification Step by Step
Threshold 0.6; filled circle = truly qualified (score ≥ 0.6). People can raise their score at a cost (e.g. number of cards). A's budget is 0.15, B's 0.03.
Gaming: anyone within their budget of the threshold moves up to it. 3 unqualified people from A and 1 from B get through (orange ring).
The institution responds by raising the threshold to 0.75 (accounting for A's budget). Now no unqualified person gets through.
But only 3 of B's 5 qualified people pass, given B's small budget (red); in A, 5 of 5. The defense against strategic behavior created a new inequality (Hu, Immorlica, Vaughan 2019).
Ethics and Governance
Algorithmic Auditing
Ethics and Governance
Discussion: Can Algorithms Be Neutral?
- Algorithms are not independent of the values of the institutions that choose their objectives, data, criteria and thresholds.
- Historical inequalities can seep into training data and target variables.
- Different fairness criteria give conflicting verdicts on the same decision (the impossibility theorems).
- After deployment a model can change behavior and create new feedback loops.
Ethics and Governance
Code Trace: Self-Confirming Data
The two regions have equal true incident counts; past records show one more in region 0 (6 vs 5), perhaps by chance.
Day 1: the patrol goes to the region with more records ⇒ target = 0.
Incidents are recorded only where the patrol goes: records = [9, 5]. The 3 incidents in region 1 stay invisible. Share 0.64.
Day 2: the patrol goes to the region with more records ⇒ target = 0.
Incidents are recorded only where the patrol goes: records = [12, 5]. The 3 incidents in region 1 stay invisible. Share 0.71.
Day 3: the patrol goes to the region with more records ⇒ target = 0.
Incidents are recorded only where the patrol goes: records = [15, 5]. The 3 incidents in region 1 stay invisible. Share 0.75.
Day 4: the patrol goes to the region with more records ⇒ target = 0.
Share 0.55 → 0.78: the data "confirm" the model because they were collected by the model's own decisions (Ensign et al., 2018).
records = [6, 5] # past records (region 0, 1) actual = [3, 3] # true daily incidents: equal! for day in range(1, 5): target = 0 if records[0] >= records[1] else 1 records[target] += actual[target] # only there share = records[0] / sum(records) print(day, target, records, round(share, 2))
Ethics and Governance
Real-World Applications and an Exercise
metrics function. (3) Vary the threshold and report which criterion you can satisfy and what you lose.Summary
Summary
Summary
Sample Problems
- Group C: TP = 30, FP = 20, FN = 10, TN = 40. Find the selection rate, TPR, FPR and PPV.
- Which criteria hold between group C from Problem 1 and group A from this chapter?
- If p = 0.2, PPV = 0.5 and TPR = 0.6, what is FPR?
- In cut and choose, Ali cuts and knows Ayşe likes chocolate 3 times as much. If Ayşe finds the two pieces equally valuable, she picks the chocolate (right) piece. Where does Ali cut, and what does he get in his own eyes?
- Show that with two people, proportionality implies envy-freeness.
- In round-robin, if Ayşe went first (same values), who would get what? A person indifferent between two goods takes the lower-numbered one. Is the allocation envy-free?
Show answers
- n = 100; selection 0.5; TPR 30/40 = 0.75; FPR 20/60 ≈ 0.333; PPV 30/50 = 0.6.
- Only demographic parity (0.5 = 0.5). TPR, FPR and PPV differ.
- FPR = 0.2/0.8 · 0.5/0.5 · 0.6 = 0.15.
- Ayşe picks the right piece as long as the left is worth ≤ 1/2 to her: 1/4 + 1.5(x − 1/2) ≤ 1/2 ⇒ x ≤ 2/3. Ali cuts at x = 2/3 and gets 2/3 (> 1/2): information favors the cutter.
- v(X1) ≥ 1/2 ⇒ v(X2) = 1 − v(X1) ≤ 1/2 ≤ v(X1).
- Ayşe m1, Ali m2, Ayşe m3 (1; tied with m4), Ali m4. Ali: 6 vs 10 ⇒ envy, but removing m1 gives 3 ≤ 6: EF1.
Summary
Exercises: New Topics
- In the
group_btrace, if A's TPR were 0.7, what q would give equal opportunity in B? What would B's accuracy be? - After running
cut_point(v_ayse, 0.5, 3), what is [lo, hi] at the end of the loop? How many queries are needed at least so that the returned midpoint is strictly within 0.001 of the true cut point? - In the last diminisher example, if Cem's values were like Ayşe's (0.1, 0.2, 0.3, 0.4), who would get the first piece?
- With L = 2, d(x, y) = 0.1 and f(x) = 0.8, what range does individual fairness allow for f(y)?
- Run the feedback code with
records = [5, 6]. What is region 0's share after 4 days?
Show answers
- TPR_B = 0.5 + 0.5·q = 0.7 ⇒ q = 0.4; accuracy (80 − 48·q)/100 = 0.608.
- 0.5 → 0.25 < 0.5 (lo = 0.5); 0.75 → 0.625 (hi = 0.75); 0.625 → 0.4375 (lo = 0.625) ⇒ [0.625, 0.75]. Midpoint error ≤ 2−(k+1) ⇒ k = 9.
- For Cem, the piece [0, 1/3] is worth 1/6 < 1/3: nobody trims, so the first cutter, Ali, takes it.
- |f(y) − 0.8| ≤ 0.2 ⇒ f(y) ∈ [0.6, 1] (a score cannot exceed 1).
- The patrol always goes to region 1: records = [5, 18], share = 5/23 ≈ 0.22. The one-record initial gap grows again.
Summary
References (1/2)
- S. Barocas, M. Hardt, A. Narayanan, Fairness and Machine Learning: Limitations and Opportunities, MIT Press, 2023.
- A. Chouldechova, "Fair prediction with disparate impact: a study of bias in recidivism prediction instruments", Big Data 5(2), 2017.
- J. Kleinberg, S. Mullainathan, M. Raghavan, "Inherent trade-offs in the fair determination of risk scores", ITCS, 2017.
- M. Hardt, E. Price, N. Srebro, "Equality of opportunity in supervised learning", NeurIPS, 2016.
- J. Angwin, J. Larson, S. Mattu, L. Kirchner, "Machine Bias", ProPublica, 2016.
- J. Buolamwini, T. Gebru, "Gender Shades", Proc. FAT*, 2018.
- S. J. Brams, A. D. Taylor, Fair Division: From Cake-Cutting to Dispute Resolution, Cambridge University Press, 1996.
- C. Dwork, M. Hardt, T. Pitassi, O. Reingold, R. Zemel, "Fairness through awareness", ITCS, 2012.
Summary
References (2/2)
- D. Ensign et al., "Runaway feedback loops in predictive policing", Proc. FAT*, 2018.
- L. Hu, N. Immorlica, J. W. Vaughan, "The disparate effects of strategic manipulation", Proc. FAT*, 2019.
- A. D. Procaccia, "Cake cutting algorithms", in Handbook of Computational Social Choice, Chapter 13, 2016.
- I. Caragiannis, D. Kurokawa, H. Moulin, A. D. Procaccia, N. Shah, J. Wang, "The unreasonable fairness of maximum Nash welfare", ACM EC, 2016.
- A. R. Karlin, Y. Peres, Game Theory, Alive, AMS, 2017 (fair division chapter). Course pages: Procaccia, CMU 15-896 · Shah, Toronto CSC304.
- E. Budish, E. Cantillon, "The multi-unit assignment problem: theory and evidence from course allocation at Harvard", American Economic Review 102(5), 2012.
- H. Aziz, S. Mackenzie, "A discrete and bounded envy-free cake cutting protocol for any number of agents", FOCS, 2016.
- L. E. Dubins, E. H. Spanier, "How to cut a cake fairly", American Mathematical Monthly 68(1), 1961; S. Even, A. Paz, "A note on cake cutting", Discrete Applied Mathematics 7(3), 1984.
- I. Caragiannis, C. Kaklamanis, P. Kanellopoulos, M. Kyropoulou, "The efficiency of fair division", Theory of Computing Systems 50(4), 2012.
Summary
References: Regulation and News
- J. Dastin, "Amazon scraps secret AI recruiting tool that showed bias against women", Reuters, 10 October 2018.
- European Parliament and Council, General Data Protection Regulation (GDPR), Regulation (EU) 2016/679, Art. 22 (in force 2018).
- European Parliament and Council, Artificial Intelligence Act, Regulation (EU) 2024/1689.
- Law No. 6698 on the Protection of Personal Data (KVKK), Official Gazette of Turkey, 7 April 2016.
- OECD, Recommendation of the Council on Artificial Intelligence (OECD AI Principles), 2019.