Chapter 11: Algorithmic Fairness and Ethics

Algorithmic Game Theory
Confusion matrixDemographic parityEqualized oddsPredictive parityCalibrationImpossibility theoremsCut and chooseProportionality and envy-freenessEthics and auditing
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

Warm-up: A Fair Division Game

The rules

Two players take turns picking indivisible items, one at a time. Each item card shows how much each player values it. The four items on the right are our running example.

Think about it
  • If your opponent always grabs their favorite item, should you always grab yours?
  • Is there always a split in which everyone values their own share at least as much as the other's?
CakeP1:5P2:1BookP1:4P2:9PhonesP1:3P2:8BallP1:1P2:1Order: P1, P2, P1, P2P2 (computer) always takes the itemit values most among those left.

Introduction

Learning Outcomes

Compute the criteriaCompute demographic parity, equalized odds, predictive parity and calibration from the confusion matrices of two groups.
Show the conflictShow numerically why the criteria cannot all hold when base rates differ.
Fair divisionApply cut and choose; distinguish proportionality from envy-freeness.
Ethics and incentivesDiscuss how transparency, accountability, auditing and strategic behavior shape design.

Introduction

Why Do Algorithmic Decisions Matter?

  • Algorithms now make or steer high-stakes decisions: lending, hiring, medical diagnosis, social media feeds, criminal justice.
  • Unfair or unethical decisions harm individuals and erode public trust in institutions and technology.
  • The game-theory link: a decision rule is a mechanism, and its stakeholders (applicants, institutions, society) have conflicting interests.
Amazon hiring (2018)A model trained on past résumés penalized applications containing the word "women's" and was scrapped.
COMPAS (2016)A recidivism risk score and the debate over unequal error rates: with different base rates, fairness criteria conflict.
Credit scoringDoes the same score mean the same risk for everyone? Who never gets credit at all?

Introduction

Where Do Bias and Discrimination Come From?

Historical dataPast inequalities end up in the labels, and the model learns them as "truth" (gender bias in hiring).
UnderrepresentationIf a group is rare in the data, the model errs more on it (face recognition fails most on darker-skinned women: Gender Shades, 2018).
Measurement and proxiesMeasure a proxy ("health spending") instead of the target ("health need") and groups with less access look healthier.
Design and feedbackThe wrong objective, proxies for a protected attribute (zip code), and decisions that reshape future data.
Dropping the protected attribute (gender, ethnicity) is not enough: other variables carry it indirectly. That is why outcomes must be measured per group.

Fairness Criteria

The Confusion Matrix

approve (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40

Lending example: Y = does the applicant actually repay, Ŷ = does the model approve; n = TP + FP + FN + TN

RateDefinitionQuestion
Selection rate(TP + FP) / nWhat fraction is approved?
TPR (true positive rate)TP / (TP + FN)How many future repayers were approved?
FPR (false positive rate)FP / (FP + TN)How many future defaulters were approved?
PPV (precision)TP / (TP + FP)How many of the approved repay?
Base rate p(TP + FN) / nShare of repayers in the group
Each fairness criterion asks for one of these rates to be equal across groups. Which one to equalize is a value judgment.

Fairness Criteria

Example: Two Groups, One Model

groupscorepeoplerepaydefaultdecision (threshold 0.5)
A0.80504010approve
A0.20501040reject
B0.8020164approve
B0.20801664reject

100 applicants per group; the model outputs two scores: 0.8 ("low risk") and 0.2 ("high risk")

The share of repayers (the base rate) is 50% in group A and 32% in group B. The model uses a 0.5 threshold.

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40
Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

Score 0.8 is approved: in A, 40 repay (TP) + 10 default (FP); in B, 16 TP + 4 FP.

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40
Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

Score 0.2 is rejected: in A, 10 would-be repayers are rejected (FN) and 40 are correctly rejected (TN); in B, 16 FN + 64 TN.

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40
Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

Both confusion matrices are ready. Accuracy is 80% in both groups ((40+40)/100 and (16+64)/100). Is the model "fair"? Let us check the criteria one by one.

Fairness Criteria

Demographic Parity Step by Step

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40
Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

Demographic parity: P(Ŷ=1 | A) = P(Ŷ=1 | B). Approved: 40 + 10 = 50 in A, 16 + 4 = 20 in B.

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40
Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

Selection rates: A = 50/100 = 0.50, B = 20/100 = 0.20. A gap of 0.30 ⇒ demographic parity fails.

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40
Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

Ratio: 0.20 / 0.50 = 0.40 < 0.8, below even the "four-fifths rule" used in US hiring. On its own this is not a legal finding of discrimination.

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40
Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

But the groups differ: 50 actual repayers in A, 32 in B. Forcing demographic parity means approving defaulters in B or rejecting repayers in A.

Fairness Criteria

Equalized Odds Step by Step

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40
Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

Equalized odds (Hardt et al., 2016): error rates must match conditional on the true outcome: TPRA = TPRB and FPRA = FPRB.

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40
Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

TPR (approval among repayers): A = 40/(40+10) = 0.80, B = 16/(16+16) = 0.50. Half of B's creditworthy applicants are rejected!

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40
Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

FPR (approval among defaulters): A = 10/(10+40) = 0.20, B = 4/(4+64) = 0.059.

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40
Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

TPR gap 0.30, FPR gap 0.141 ⇒ equalized odds fails. Requiring equal TPR alone is called equal opportunity.

Fairness Criteria

Predictive Parity Step by Step

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40
Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

Predictive parity: the repayment rate among approved applicants (PPV) must be equal across groups. This is the lender's point of view.

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40
Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

PPV: A = 40/(40+10) = 0.80, B = 16/(16+4) = 0.80 ⇒ predictive parity holds.

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40
Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

Rejections too (NPV: share of rejected who really default = TN/(TN + FN)): A = 40/50 = 0.80, B = 64/80 = 0.80. "Approve" and "reject" mean the same thing in both groups.

Fairness Criteria

Calibration Step by Step

groupscore speoplerepayobserved rateequals s?
A0.805040?
A0.205010?
B0.802016?
B0.208016?

Calibration within groups: among people with score s, the repayment rate must be s in every group. A score should mean the same risk regardless of group.

groupscore speoplerepayobserved rateequals s?
A0.80504040/50 = 0.80yes
A0.205010?
B0.802016?
B0.208016?

Group A, score 0.8: 40 of 50 repay ⇒ 0.80.

groupscore speoplerepayobserved rateequals s?
A0.80504040/50 = 0.80yes
A0.20501010/50 = 0.20yes
B0.802016?
B0.208016?

Group A, score 0.2: 10 of 50 repay ⇒ 0.20.

groupscore speoplerepayobserved rateequals s?
A0.80504040/50 = 0.80yes
A0.20501010/50 = 0.20yes
B0.80201616/20 = 0.80yes
B0.208016?

Group B, score 0.8: 16 of 20 repay ⇒ 0.80.

groupscore speoplerepayobserved rateequals s?
A0.80504040/50 = 0.80yes
A0.20501010/50 = 0.20yes
B0.80201616/20 = 0.80yes
B0.20801616/80 = 0.20yes

Group B, score 0.2: 16 of 80 repay ⇒ 0.20. The model is calibrated in both groups: the score is an honest probability.

Fairness Criteria

Code Trace: metrics(TP, FP, FN, TN)

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40

The function is called with group A's matrix.

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40

n = 40 + 10 + 10 + 40 = 100 applicants.

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40

Selection rate = (40 + 10) / 100 = 0.5 (compared for demographic parity).

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40

TPR = 40 / (40 + 10) = 0.8.

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40

FPR = 10 / (10 + 40) = 0.2.

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40

PPV = 40 / (40 + 10) = 0.8.

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40

The four rates are returned as a Python tuple and assigned to A.

Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

Same computation for B: selection 20/100, TPR 16/32, FPR 4/68 ≈ 0.059, PPV 16/20.

Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

B = (0.2, 0.5, 0.0588…, 0.8).

Group Aapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP40FN10FP10TN40

No demographic parity: 0.5 ≠ 0.2.

Group Bapprove (Ŷ=1)reject (Ŷ=0)repays (Y=1)defaults (Y=0)TP16FN16FP4TN64

i = 1, 2, 3: TPR differs ⇒ no equal opportunity; FPR differs too ⇒ no equalized odds; PPV equal ⇒ predictive parity holds.

def metrics(TP, FP, FN, TN):
  n = TP + FP + FN + TN
  sel = (TP + FP) / n     # demographic parity
  tpr = TP / (TP + FN)    # equalized odds
  fpr = FP / (FP + TN)
  ppv = TP / (TP + FP)    # predictive parity
  return sel, tpr, fpr, ppv

A = metrics(40, 10, 10, 40)
B = metrics(16, 4, 16, 64)
NAMES = ["sel", "TPR", "FPR", "PPV"]
for i, name in enumerate(NAMES):
  print(name, round(A[i], 3), round(B[i], 3))

Fairness Criteria

All Criteria, One Model: The Scorecard

CriterionMust be equalABHolds?
Demographic parityP(Ŷ=1)0.500.20no
Equal opportunityTPR = P(Ŷ=1 | Y=1)0.800.50no
Equalized oddsTPR and FPR = P(Ŷ=1 | Y=0)0.80; 0.200.50; 0.059no
Predictive parityPPV = P(Y=1 | Ŷ=1)0.800.80yes
CalibrationP(Y=1 | score = s) = s0.8→0.8; 0.2→0.20.8→0.8; 0.2→0.2yes
The applicant's view"If I would repay, does my chance of approval depend on my group?" ⇒ TPR / FPR.
The lender's view"Does the repayment chance of someone I approve depend on their group?" ⇒ PPV, calibration.

Fairness Criteria

Beyond Group Fairness: Individual Fairness

0.20.50.8similarity axis (e.g. debt/income)model score fEceNurstep: |Δf| = 0.6slope L = 2: |Δf| = 0.1

distance between Ece and Nur d = 0.05; L·d = 0.1

  • Dwork et al. (2012): "treat similar people similarly": |f(x) − f(y)| ≤ L · d(x, y) (the Lipschitz condition). f(x) is person x's score; d(x, y) measures how different two people are; L is the largest allowed slope. Two people's scores may differ by at most L times their distance.
  • The red step model gives Ece and Nur, on either side of the threshold, 0.2 and 0.8: |Δf| = 0.6 > 0.1 ⇒ violation. The green model satisfies the condition.
  • Group criteria equalize averages: approving random people from B achieves demographic parity but is arbitrary for individuals.
The catch: who chooses the metric d? "Who counts as similar?" is itself a value judgment. Individual fairness does not remove the problem; it moves it into the choice of d.

Incompatibility

Why Can't the Criteria Hold Together?

p = base rate, n people ⇒ TP = p·n·TPR, FP = (1 − p)·n·FPR
PPV = TP / (TP + FP) = p·TPR / (p·TPR + (1 − p)·FPR)
Clear the denominator: PPV·[p·TPR + (1 − p)·FPR] = p·TPR ⇒ (1 − p)·FPR·PPV = p·TPR·(1 − PPV)
Rearrange: FPR = p / (1 − p) · (1 − PPV) / PPV · TPR
A: p = 0.5 ⇒ 0.5/0.5 · 0.2/0.8 · 0.8 = 1 · 0.25 · 0.8 = 0.20 ✓
B: p = 0.32 ⇒ 0.32/0.68 · 0.25 · 0.5 = 0.059 ✓
PPV is equal (0.8) but p differs ⇒ the factor p/(1 − p) differs ⇒ TPR and FPR cannot both be equal.

Chouldechova (2017): the criteria all come from the same four numbers, so they are linked. Start with the base rate p.

p = base rate, n people ⇒ TP = p·n·TPR, FP = (1 − p)·n·FPR
PPV = TP / (TP + FP) = p·TPR / (p·TPR + (1 − p)·FPR)
Clear the denominator: PPV·[p·TPR + (1 − p)·FPR] = p·TPR ⇒ (1 − p)·FPR·PPV = p·TPR·(1 − PPV)
Rearrange: FPR = p / (1 − p) · (1 − PPV) / PPV · TPR
A: p = 0.5 ⇒ 0.5/0.5 · 0.2/0.8 · 0.8 = 1 · 0.25 · 0.8 = 0.20 ✓
B: p = 0.32 ⇒ 0.32/0.68 · 0.25 · 0.5 = 0.059 ✓
PPV is equal (0.8) but p differs ⇒ the factor p/(1 − p) differs ⇒ TPR and FPR cannot both be equal.

Write PPV, the subject of predictive parity, in terms of TPR, FPR and p.

p = base rate, n people ⇒ TP = p·n·TPR, FP = (1 − p)·n·FPR
PPV = TP / (TP + FP) = p·TPR / (p·TPR + (1 − p)·FPR)
Clear the denominator: PPV·[p·TPR + (1 − p)·FPR] = p·TPR ⇒ (1 − p)·FPR·PPV = p·TPR·(1 − PPV)
Rearrange: FPR = p / (1 − p) · (1 − PPV) / PPV · TPR
A: p = 0.5 ⇒ 0.5/0.5 · 0.2/0.8 · 0.8 = 1 · 0.25 · 0.8 = 0.20 ✓
B: p = 0.32 ⇒ 0.32/0.68 · 0.25 · 0.5 = 0.059 ✓
PPV is equal (0.8) but p differs ⇒ the factor p/(1 − p) differs ⇒ TPR and FPR cannot both be equal.

Solving for FPR gives an identity linking the three rates (FPR, TPR, PPV) and the base rate.

p = base rate, n people ⇒ TP = p·n·TPR, FP = (1 − p)·n·FPR
PPV = TP / (TP + FP) = p·TPR / (p·TPR + (1 − p)·FPR)
Clear the denominator: PPV·[p·TPR + (1 − p)·FPR] = p·TPR ⇒ (1 − p)·FPR·PPV = p·TPR·(1 − PPV)
Rearrange: FPR = p / (1 − p) · (1 − PPV) / PPV · TPR
A: p = 0.5 ⇒ 0.5/0.5 · 0.2/0.8 · 0.8 = 1 · 0.25 · 0.8 = 0.20 ✓
B: p = 0.32 ⇒ 0.32/0.68 · 0.25 · 0.5 = 0.059 ✓
PPV is equal (0.8) but p differs ⇒ the factor p/(1 − p) differs ⇒ TPR and FPR cannot both be equal.

Check the identity on group A.

p = base rate, n people ⇒ TP = p·n·TPR, FP = (1 − p)·n·FPR
PPV = TP / (TP + FP) = p·TPR / (p·TPR + (1 − p)·FPR)
Clear the denominator: PPV·[p·TPR + (1 − p)·FPR] = p·TPR ⇒ (1 − p)·FPR·PPV = p·TPR·(1 − PPV)
Rearrange: FPR = p / (1 − p) · (1 − PPV) / PPV · TPR
A: p = 0.5 ⇒ 0.5/0.5 · 0.2/0.8 · 0.8 = 1 · 0.25 · 0.8 = 0.20 ✓
B: p = 0.32 ⇒ 0.32/0.68 · 0.25 · 0.5 = 0.059 ✓
PPV is equal (0.8) but p differs ⇒ the factor p/(1 − p) differs ⇒ TPR and FPR cannot both be equal.

Check it on group B: it matches the 4/68 in the table.

p = base rate, n people ⇒ TP = p·n·TPR, FP = (1 − p)·n·FPR
PPV = TP / (TP + FP) = p·TPR / (p·TPR + (1 − p)·FPR)
Clear the denominator: PPV·[p·TPR + (1 − p)·FPR] = p·TPR ⇒ (1 − p)·FPR·PPV = p·TPR·(1 − PPV)
Rearrange: FPR = p / (1 − p) · (1 − PPV) / PPV · TPR
A: p = 0.5 ⇒ 0.5/0.5 · 0.2/0.8 · 0.8 = 1 · 0.25 · 0.8 = 0.20 ✓
B: p = 0.32 ⇒ 0.32/0.68 · 0.25 · 0.5 = 0.059 ✓
PPV is equal (0.8) but p differs ⇒ the factor p/(1 − p) differs ⇒ TPR and FPR cannot both be equal.

Conclusion: with different base rates, predictive parity and equalized odds cannot both hold (except in extreme cases such as FPR = 0 and PPV = 1).

Incompatibility

What If We Enforce Equalized Odds?

Goal: TPR = 0.8 and FPR = 0.2 in B as well (same as A). B has 32 repayers and 68 defaulters.
TP = 32 × 0.8 = 25.6, FP = 68 × 0.2 = 13.6 (expected counts)
Approved = 25.6 + 13.6 = 39.2 ⇒ selection rate 0.39
PPVB = 25.6 / 39.2 = 0.65 < PPVA = 0.80
Equalized odds now holds, but predictive parity breaks: "approve" in B now signals a lower repayment probability (with only approve/reject outputs, the criterion that breaks is PPV).

Imagine a (hypothetical) better model that achieves these error rates for B. Today's model (scores 0.8 and 0.2 only) cannot reach them by moving the threshold or by randomizing approvals.

Goal: TPR = 0.8 and FPR = 0.2 in B as well (same as A). B has 32 repayers and 68 defaulters.
TP = 32 × 0.8 = 25.6, FP = 68 × 0.2 = 13.6 (expected counts)
Approved = 25.6 + 13.6 = 39.2 ⇒ selection rate 0.39
PPVB = 25.6 / 39.2 = 0.65 < PPVA = 0.80
Equalized odds now holds, but predictive parity breaks: "approve" in B now signals a lower repayment probability (with only approve/reject outputs, the criterion that breaks is PPV).

Fixing the error rates fixes the top and bottom rows of the confusion matrix.

Goal: TPR = 0.8 and FPR = 0.2 in B as well (same as A). B has 32 repayers and 68 defaulters.
TP = 32 × 0.8 = 25.6, FP = 68 × 0.2 = 13.6 (expected counts)
Approved = 25.6 + 13.6 = 39.2 ⇒ selection rate 0.39
PPVB = 25.6 / 39.2 = 0.65 < PPVA = 0.80
Equalized odds now holds, but predictive parity breaks: "approve" in B now signals a lower repayment probability (with only approve/reject outputs, the criterion that breaks is PPV).

B's approval rate rises from 0.2 to 0.39.

Goal: TPR = 0.8 and FPR = 0.2 in B as well (same as A). B has 32 repayers and 68 defaulters.
TP = 32 × 0.8 = 25.6, FP = 68 × 0.2 = 13.6 (expected counts)
Approved = 25.6 + 13.6 = 39.2 ⇒ selection rate 0.39
PPVB = 25.6 / 39.2 = 0.65 < PPVA = 0.80
Equalized odds now holds, but predictive parity breaks: "approve" in B now signals a lower repayment probability (with only approve/reject outputs, the criterion that breaks is PPV).

The repayment rate among the approved falls.

Goal: TPR = 0.8 and FPR = 0.2 in B as well (same as A). B has 32 repayers and 68 defaulters.
TP = 32 × 0.8 = 25.6, FP = 68 × 0.2 = 13.6 (expected counts)
Approved = 25.6 + 13.6 = 39.2 ⇒ selection rate 0.39
PPVB = 25.6 / 39.2 = 0.65 < PPVA = 0.80
Equalized odds now holds, but predictive parity breaks: "approve" in B now signals a lower repayment probability (with only approve/reject outputs, the criterion that breaks is PPV).

Which criterion should we keep? Context (the kind of harm, the law) decides, not mathematics.

Incompatibility

Calibration and Balance: Kleinberg's Theorem

Balance for the positive class: repayers should get the same average score in every group.
A, Y=1: (40·0.8 + 10·0.2)/50 = 0.68; B, Y=1: (16·0.8 + 16·0.2)/32 = 0.50
A, Y=0: (10·0.8 + 40·0.2)/50 = 0.32; B, Y=0: (4·0.8 + 64·0.2)/68 = 0.235
Theorem (Kleinberg, Mullainathan, Raghavan 2017): calibration and balance for both classes can hold together only with perfect prediction or equal base rates.

The score-level analogue of equalized odds: balance.

Balance for the positive class: repayers should get the same average score in every group.
A, Y=1: (40·0.8 + 10·0.2)/50 = 0.68; B, Y=1: (16·0.8 + 16·0.2)/32 = 0.50
A, Y=0: (10·0.8 + 40·0.2)/50 = 0.32; B, Y=0: (4·0.8 + 64·0.2)/68 = 0.235
Theorem (Kleinberg, Mullainathan, Raghavan 2017): calibration and balance for both classes can hold together only with perfect prediction or equal base rates.

In our calibrated model, B's repayers get a lower average score.

Balance for the positive class: repayers should get the same average score in every group.
A, Y=1: (40·0.8 + 10·0.2)/50 = 0.68; B, Y=1: (16·0.8 + 16·0.2)/32 = 0.50
A, Y=0: (10·0.8 + 40·0.2)/50 = 0.32; B, Y=0: (4·0.8 + 64·0.2)/68 = 0.235
Theorem (Kleinberg, Mullainathan, Raghavan 2017): calibration and balance for both classes can hold together only with perfect prediction or equal base rates.

No balance among defaulters either. Calibrated, yet unbalanced.

Balance for the positive class: repayers should get the same average score in every group.
A, Y=1: (40·0.8 + 10·0.2)/50 = 0.68; B, Y=1: (16·0.8 + 16·0.2)/32 = 0.50
A, Y=0: (10·0.8 + 40·0.2)/50 = 0.32; B, Y=0: (4·0.8 + 64·0.2)/68 = 0.235
Theorem (Kleinberg, Mullainathan, Raghavan 2017): calibration and balance for both classes can hold together only with perfect prediction or equal base rates.

In our example the base rates differ (0.5 and 0.32) and prediction is imperfect: the criteria must conflict.

Incompatibility

Change the Assumption: Which Fairness Criterion?

groupapprove: repay / defaultreject: repay / defaultbase rate
A40 / 1010 / 400.50
B16 / 416 / 640.32
100 applicants per group, same model and threshold (0.5). Is it fair?
groupapprove: repay / defaultreject: repay / defaultbase rate
A40 / 1010 / 400.50
B16 / 416 / 640.32
100 applicants per group, same model and threshold (0.5). Is it fair?
CriterionABHolds?
Demographic parity (approvals)0.500.20no
Equalized odds (TPR; FPR)0.80; 0.200.50; 0.059no
Predictive parity (PPV)0.800.80yes
groupapprove: repay / defaultreject: repay / defaultbase rate
A40 / 1010 / 400.50
B16 / 416 / 640.32
100 applicants per group, same model and threshold (0.5). Is it fair?
CriterionABHolds?
Demographic parity (approvals)0.500.20no
Equalized odds (TPR; FPR)0.80; 0.200.50; 0.059no
Predictive parity (PPV)0.800.80yes
It depends on the criterion: fair for the lender, unfair for the applicant. With unequal base rates, equalized odds and predictive parity cannot both hold (Chouldechova, 2017; Kleinberg et al., 2017). Choosing one is a value judgment.

Incompatibility

A Real Case: The COMPAS Debate

0%25%50%44.923.5FPRwrongly "high risk"2847.7FNRwrongly "low risk"Blackwhite

ProPublica (2016), Broward County data: percentages

ProPublicaAmong defendants who did not reoffend, 44.9% of Black and 23.5% of white defendants were labeled "high risk": unequal FPR. For FNR (the share of reoffenders labeled “low risk”) the picture reverses.
Northpointe (the vendor)The same risk score corresponds to similar reoffense rates in both groups: calibration / predictive parity holds.
Both sides computed correctly. With different base rates, the two criteria cannot hold together (Chouldechova's identity): the real dispute is over which definition of fairness to adopt. Report error rates per group, not a single accuracy number.

Incompatibility

The Fairness–Accuracy Trade-off

  • Accuracy alone does not show on whom the errors fall: in our example accuracy is 80% in both groups, yet TPR is 0.8 vs 0.5.
  • A fairness constraint may raise total error; the real choice is which errors are acceptable (wrongful rejection or wrongful approval?).
Pre-processingReweight the data and fix underrepresentation.
In-processingAdd a fairness constraint to the training loss (giving up some accuracy).
Post-processingGroup-specific thresholds (Hardt et al.): may be legally contested.
The context of use, legal requirements, regulation and the views of affected groups should guide the choice of criterion. Example: credit scoring that treats low-income applicants fairly.

Incompatibility

Code Trace: Equal Opportunity by Randomization

00.50.81q=0q=0.2q=0.4q=0.6TPR_Aq = 0TP = 16 FP = 4

q = 0: only score 0.8 is approved (today's model). TP = 16, FP = 4.

00.50.81q=0q=0.2q=0.4q=0.6TPR_Aq = 0TP = 16 FP = 4TPR_B = 0.5FPR_B = 0.06accuracy_B = 0.8

TPR = 16/32 = 0.5; FPR = 4/68 ≈ 0.06; accuracy (16 + 64)/100 = 0.80.

00.50.81q=0q=0.2q=0.4q=0.6TPR_Aq = 0.2TP = 19.2 FP = 16.8

q = 0.2: 20% of the 80 people scored 0.2 are approved. TP = 16 + 16·q = 19.2, FP = 4 + 64·q = 16.8.

00.50.81q=0q=0.2q=0.4q=0.6TPR_Aq = 0.2TP = 19.2 FP = 16.8TPR_B = 0.6FPR_B = 0.25accuracy_B = 0.704

TPR rose to 0.6 but accuracy fell to 0.704: four of every five new approvals are defaulters.

00.50.81q=0q=0.2q=0.4q=0.6TPR_Aq = 0.4TP = 22.4 FP = 29.6

q = 0.4: 40% of the 80 people scored 0.2 are approved. TP = 16 + 16·q = 22.4, FP = 4 + 64·q = 29.6.

00.50.81q=0q=0.2q=0.4q=0.6TPR_Aq = 0.4TP = 22.4 FP = 29.6TPR_B = 0.7FPR_B = 0.44accuracy_B = 0.608

TPR 0.7; FPR jumps to 0.44, accuracy 0.608.

00.50.81q=0q=0.2q=0.4q=0.6TPR_Aq = 0.6TP = 25.6 FP = 42.4

q = 0.6: 60% of the 80 people scored 0.2 are approved. TP = 16 + 16·q = 25.6, FP = 4 + 64·q = 42.4.

00.50.81q=0q=0.2q=0.4q=0.6TPR_Aq = 0.6TP = 25.6 FP = 42.4TPR_B = 0.8FPR_B = 0.62accuracy_B = 0.512

TPR_B = 0.8 = TPR_A: equal opportunity achieved. The price: accuracy 0.512, FPR 0.62.

00.50.81q=0q=0.2q=0.4q=0.6TPR_Aq = 0.6TP = 25.6 FP = 42.4TPR_B = 0.8FPR_B = 0.62accuracy_B = 0.512

For equal opportunity, 60% of those scored 0.2 were approved at random: about 38 of the 64 defaulters get loans. FPR is more than three times A's (0.2), so equalized odds still fails.

def group_b(q):   # approve score 0.2 w.p. q
  TP = 16 + 16 * q
  FP = 4 + 64 * q
  tpr = TP / 32
  fpr = FP / 68
  hits = TP + (68 - FP)    # TP + TN
  return tpr, fpr, round(hits / 100, 3)

for q in [0, 0.2, 0.4, 0.6]:
  tpr, fpr, d = group_b(q)
  print(q, round(tpr, 2), round(fpr, 2), d)

Fair Division

Fair Division: Cake Cutting

  • A divisible resource (the cake = the interval [0, 1]: time, land, compute) is to be shared among n people.
  • Each person has a valuation function vi: the whole cake is worth 1, but people value different parts differently.
  • The algorithm may only ask queries (the Robertson–Webb model): "what is this piece worth to you?" and "where does a piece of a given value end?". Xi is the piece person i receives; vi(Xi) is its value to person i.
ProportionalityEveryone gets at least 1/n by their own measure: vi(Xi) ≥ 1/n.
Envy-freenessNo one prefers someone else's piece to their own: vi(Xi) ≥ vi(Xj).
The game-theoretic view: people may be strategic. A good protocol gives every honest participant a guarantee, whatever the others do.

Fair Division

Cut and Choose Step by Step

vanilla [0, 0.5]chocolate [0.5, 1]

Ali values the whole cake uniformly. Ayşe likes chocolate 3 times as much as vanilla: vAyşe(vanilla) = 1/4, vAyşe(chocolate) = 3/4.

vanilla [0, 0.5]chocolate [0.5, 1]cut x = 0.5Ali: 1/2Ali: 1/2

1. Ali cuts into two pieces equal in his eyes ⇒ x = 1/2, each piece worth 1/2 to Ali.

vanilla [0, 0.5]chocolate [0.5, 1]cut x = 0.5Ali: 1/2Ali: 1/2Ayşe: 1/4Ayşe: 3/4

2. Ayşe evaluates: left piece 1/4, right piece 3/4.

vanilla [0, 0.5]chocolate [0.5, 1]cut x = 0.5Ali: 1/2Ali: 1/2Ayşe: 1/4Ayşe: 3/4Ayşe

3. Ayşe chooses the more valuable right piece (chocolate).

vanilla [0, 0.5]chocolate [0.5, 1]cut x = 0.5Ali: 1/2Ali: 1/2Ayşe: 1/4Ayşe: 3/4AliAyşe

Result: Ali 1/2 ≥ 1/2, Ayşe 3/4 ≥ 1/2 ⇒ proportional; Ali 1/2 = 1/2, Ayşe 3/4 > 1/4 ⇒ envy-free.

vanilla [0, 0.5]chocolate [0.5, 1]cut x = 0.667Ali: 2/3Ali: 1/3Ayşe: 1/2Ayşe: 1/2AliAyşe

Swap roles: Ayşe cuts at x = 2/3 (1/2 – 1/2 in her eyes). Ali picks the left piece and gets 2/3; Ayşe gets 1/2. The chooser has the edge over the cutter.

Fair Division

Code Trace: A Cut Query by Bisection

vanilla (Ayşe: 1/4)chocolate (Ayşe: 3/4)00.51lo=0hi=1

A Robertson–Webb cut query: "where is x so that [0, x] is worth 1/2 to you?" Ayşe's valuation is increasing in x, so bisection finds it.

vanilla (Ayşe: 1/4)chocolate (Ayşe: 3/4)00.51lo=0.5hi=1mid = 0.5 → v = 0.25

Query 1: mid = 0.5: all of the vanilla is worth 1/4 = 0.25 < 0.5 to Ayşe ⇒ lo = 0.5. The interval halves (width 1/2).

vanilla (Ayşe: 1/4)chocolate (Ayşe: 3/4)00.51lo=0.5hi=0.75mid = 0.75 → v = 0.625

Query 2: v(0.75) = 0.625 ≥ 0.5 ⇒ the cut is here or to the left: hi = mid. The interval halves (width 1/4).

vanilla (Ayşe: 1/4)chocolate (Ayşe: 3/4)00.51lo=0.625hi=0.75mid = 0.625 → v = 0.4375

Query 3: v(0.625) = 0.4375 < 0.5 ⇒ the cut is further right: lo = mid. The interval halves (width 1/8).

vanilla (Ayşe: 1/4)chocolate (Ayşe: 3/4)00.51lo=0.625hi=0.6875mid = 0.6875 → v = 0.5312

Query 4: v(0.6875) = 0.5312 ≥ 0.5 ⇒ the cut is here or to the left: hi = mid. The interval halves (width 1/16).

vanilla (Ayşe: 1/4)chocolate (Ayşe: 3/4)00.51lo=0.6562hi=0.6875mid = 0.6562 → v = 0.4844

Query 5: v(0.6562) = 0.4844 < 0.5 ⇒ the cut is further right: lo = mid. The interval halves (width 1/32).

vanilla (Ayşe: 1/4)chocolate (Ayşe: 3/4)00.51lo=0.6562hi=0.6719mid = 0.6719 → v = 0.5078

Query 6: v(0.6719) = 0.5078 ≥ 0.5 ⇒ the cut is here or to the left: hi = mid. The interval halves (width 1/64).

vanilla (Ayşe: 1/4)chocolate (Ayşe: 3/4)00.51lo=0.6641hi=0.6719mid = 0.6641 → v = 0.4961

Query 7: v(0.6641) = 0.4961 < 0.5 ⇒ the cut is further right: lo = mid. The interval halves (width 1/128).

vanilla (Ayşe: 1/4)chocolate (Ayşe: 3/4)00.51lo=0.6641hi=0.668mid = 0.668 → v = 0.502

Query 8: v(0.668) = 0.502 ≥ 0.5 ⇒ the cut is here or to the left: hi = mid. The interval halves (width 1/256).

vanilla (Ayşe: 1/4)chocolate (Ayşe: 3/4)00.51lo=0.6641hi=0.668cut ≈ 0.666 (exactly 2/3)

After 8 queries the interval is 1/256 wide; its midpoint is 0.666. The true cut is 2/3: where Ayşe would cut if she were the cutter.

def v_ayse(x):          # value of [0, x]
  if x <= 0.5: return x / 2
  return 0.25 + 1.5 * (x - 0.5)
def cut_point(v, target, steps):
  lo, hi = 0.0, 1.0
  for _ in range(steps):
    mid = (lo + hi) / 2
    if v(mid) < target: lo = mid
    else: hi = mid
  return (lo + hi) / 2
print(round(cut_point(v_ayse, 0.5, 8), 3))

Fair Division

Why Is Cut and Choose Fair?

00.250.501/22/31cut point xcutter's guarantee: min(v(left), v(right))Ali: x = 1/2Ayşe: x = 2/3
  • The cutter does not know which piece will be left, so the guarantee is the smaller piece. It peaks at the cutter's own halfway point (1/2): cutting honestly is the maximin strategy.
  • The chooser takes the piece that is larger in her eyes: worth ≥ 1/2 and no less than the other, wherever the cut is.
  • A cutter who knows the chooser's preferences can shift the cut and get more than 1/2, but the guarantee is 1/2.
With two people, proportionality ⇔ envy-freeness: if v(X1) ≥ 1/2, then v(X2) = 1 − v(X1) ≤ 1/2 ≤ v(X1). With three or more, envy-freeness is stronger.

Fair Division

Proportional but Not Envy-Free: Three People

peoplepiece Xpiece Ypiece Zgets≥ 1/3?
Ali0.400.300.30
Ayşe0.200.350.45
Cem0.300.300.40

Three people, three pieces (X, Y, Z). The table shows each person's value for each piece; each row sums to 1.

peoplepiece Xpiece Ypiece Zgets≥ 1/3?
Ali0.400.300.30X: 0.40yes
Ayşe0.200.350.45Y: 0.35yes
Cem0.300.300.40Z: 0.40yes

Allocation: Ali → X, Ayşe → Y, Cem → Z. Everyone gets at least 1/3 ≈ 0.33 by their own measure ⇒ proportional.

peoplepiece Xpiece Ypiece Zgets≥ 1/3?
Ali0.400.300.30X: 0.40yes
Ayşe0.200.350.45Y: 0.35yes
Cem0.300.300.40Z: 0.40yes

But to Ayşe, Cem's piece Z (0.45) beats her own Y (0.35) ⇒ Ayşe envies Cem. Proportional, but not envy-free.

Last diminisherA proportional division for n people.
Selfridge–ConwayEnvy-free for 3 people, at most 5 cuts.
Aziz–Mackenzie (2016)Envy-free for n people; finitely many but astronomically many queries.

Fair Division

Last Diminisher Step by Step

Ali0.250.250.250.25Ayşe0.10.20.30.4Cem0.40.30.20.1

Three people; the cake has four quarters. Bars show each person's value per quarter (total 1). Ali is uniform; Ayşe likes the right, Cem the left.

piece [0, 1/3]Ali0.250.250.250.25Ayşe0.10.20.30.4Cem0.40.30.20.11/3 ✓

Ali cuts a piece worth 1/3 to him: [0, 1/3].

piece [0, 1/3]Ali0.250.250.250.25Ayşe0.10.20.30.4Cem0.40.30.20.11/31/6 < 1/3

Ayşe values it at 1/6, less than 1/3, so she leaves it alone ("passes").

piece [0, 1/3]Ali0.250.250.250.25Ayşe0.10.20.30.4Cem0.40.30.20.11/31/61/2 > 1/3

To Cem the piece is worth 1/2 > 1/3: he trims it, moving the cut to 5/24 (exactly 1/3 for Cem).

Cem's pieceAli0.250.250.250.25Ayşe0.10.20.30.4Cem0.40.30.20.1gets: 1/3

The last trimmer (Cem) takes the piece and leaves. For those who did not trim, the piece is worth ≤ 1/3, so the rest of the cake is worth ≥ 2/3 to them.

Cem's pieceAliAyşeAli0.250.250.250.25Ayşe0.10.20.30.4Cem0.40.30.20.119/4823/401/3

The remaining [5/24, 1] is split between two by cut and choose: Ali cuts at 29/48, Ayşe picks the more valuable right piece (23/40).

Cem's pieceAliAyşeAli0.250.250.250.25Ayşe0.10.20.30.4Cem0.40.30.20.119/48 ≥ 1/323/40 ≥ 1/31/3 ≥ 1/3

Everyone got ≥ 1/3 by their own measure ⇒ proportional. O(n²) queries for n people. Not envy-free, though: Cem values Ali's piece at 9/20 (his own 1/3).

Fair Division

Moving Knife (Dubins–Spanier) and Even–Paz

The ruleA knife slides slowly from left to right across the cake. When the piece left of the knife is worth 1/n to someone, that person calls "stop!", takes the piece and leaves; the knife continues with the rest, and the last person takes what remains (Dubins and Spanier, 1961).
peopleround 1: v([0, x]) = 1/3round 2: v([5/24, x]) = 1/3gets
Cemx = 5/24 ≈ 0.21 (stop!)—[0, 5/24]: 1/3
Alix = 1/3 ≈ 0.33x = 13/24 ≈ 0.54 (stop!)[5/24, 13/24]: 1/3
Ayşex = 19/36 ≈ 0.53x = 43/72 ≈ 0.60[13/24, 1]: 13/20

Values as in the last diminisher example (per quarter: Ali 0.25; Ayşe 0.1–0.4; Cem 0.4–0.1). The first piece matches last diminisher: Cem stops the knife first. Everyone gets ≥ 1/3 ⇒ proportional. Not envy-free: Cem values Ali's piece at 2/5 > 1/3.

Even–Paz (1984): finds a proportional division with far fewer queries by halving the cake at each step.

Fair Division

Indivisible Goods: Round-Robin and EF1

peoplem1m2m3m4ownother's
Ali753100
Ayşe861100

4 indivisible goods, 2 people; the table shows values. Round-robin: on each turn, the next person takes the most valuable remaining good.

peoplem1m2m3m4ownother's
Ali753170
Ayşe861108

Ali takes m1 (7).

peoplem1m2m3m4ownother's
Ali753175
Ayşe861168

Ayşe takes m2 (6) from what is left.

peoplem1m2m3m4ownother's
Ali7531105
Ayşe861169

Ali takes m3 (3).

peoplem1m2m3m4ownother's
Ali7531106
Ayşe861179

Ayşe takes m4. Ayşe: her own bundle 7, Ali's bundle 9 in her eyes ⇒ envy! But removing m1 from Ali's bundle gives 1 ≤ 7: EF1 (envy-free up to one good).

With indivisible goods, envy-freeness is not always possible (one car, two siblings). With additive values, round-robin in which everyone picks their favorite remaining good always yields an EF1 allocation.

Fair Division

Back to the Game: Fair Division

Greedy picking is not a best response

The opening items (P1 / P2 value): Cake 5/1, Book 4/9, Headphones 3/8, Ball 1/1.

P1's planPick orderP1P2
always the bestCake, Book, Phones, Ball810
Book firstBook, Phones, Cake, Ball99

P2 values the Cake at 1, so it will still be there later. In round-robin, taking your favorite item every turn is not always the best strategy.

What about fairness?

Under the second plan, P2 values P1's bundle at 9 + 1 = 10 and its own at 9: envy. Remove the Book and 1 ≤ 9: the allocation is still EF1. The theory's guarantee: if both sides always pick their favorite, round-robin yields EF1.

In practice: data from Harvard Business School's draft-based course allocation showed students making exactly this kind of strategic pick, at a cost in welfare (Budish and Cantillon, 2012).

Even a fair protocol is open to strategic players: the EF1 guarantee holds when everyone picks their favorite on their turn.

Fair Division

Which Allocation? Three Social Welfare Criteria

Utilitarian

Maximize social welfare SW = Σ vi: efficiency.

Egalitarian

Maximize the smallest vi (maximin).

Nash social welfare

Maximize vAli × vAyşe; scaling one person's values by 10 does not change the answer.

AllocationAliAyşesumminproductenvy-free?
Round-robin: Ali {m1, m3}, Ayşe {m2, m4}10717770no, EF1
Utilitarian: Ali {m3, m4}, Ayşe {m1, m2}41418456no, not even EF1
Nash / egalitarian (not unique): Ali {m2, m3, m4}, Ayşe {m1}9817872yes
Under the utilitarian allocation Ali values Ayşe's bundle at 12 (his own 4); even without m1 it is 5 > 4. Theorem (Caragiannis et al., 2016): with nonnegative additive values, a Nash-welfare-maximizing allocation is both Pareto optimal and EF1 (with a tie-breaking rule if the product is always 0).

Fair Division

The Price of Fairness and Pareto Efficiency

Pareto efficiencyNo other allocation makes someone better off without hurting anyone. 11 of the 16 allocations here are, including round-robin's (10, 7).
Price of fairnessMaximum welfare over maximum welfare among fair allocations. Near 1: fairness is cheap.
AllocationAliAyşesumenvy-free?
Ali {m3, m4}, Ayşe {m1, m2}41418no
Ali {m3}, Ayşe {m1, m2, m4}31518no
Ali {m2, m3, m4}, Ayşe {m1}9817yes
Ali {m2, m3}, Ayşe {m1, m4}8917yes
Only two of 16 allocations are envy-free (total 17; with the tie on m4, both are Nash and egalitarian optima). Price of fairness = 18 / 17 ≈ 1.059: about 5.6% of welfare. Like the PoA, but the loss comes from the fairness constraint, not selfishness. For a divisible cake and n people, the price of proportionality grows like √n (Caragiannis et al., 2012).

Ethics and Governance

Ethical Principles and Standards

Transparency and explainabilityDecision processes should be understandable; people should be able to learn why a decision was made.
AccountabilitySomeone must be responsible for a wrong decision; appeal and human review must be possible.
Fairness and equalitySocial impact and group-level outcomes must be taken into account.
PrivacyCollect only the data needed, and only for its stated purpose.
GDPR (EU, 2018), Art. 22Right to contest, and to human intervention in, significant decisions based solely on automated processing
EU AI Act (2024)Risk-based: credit scoring and hiring are "high risk"
KVKK (Turkey), OECD AI PrinciplesRules for processing personal data; shared principles for trustworthy AI

Ethics and Governance

Strategic Design and Incentives

  • When an algorithm sets a rule, people respond to it: those being classified are players too (strategic classification).
  • Goodhart's law: when a measure becomes a target, it ceases to be a good measure (click counts → clickbait).
  • Giving users the right incentives and preventing malicious manipulation (fake reviews, bot accounts) is part of the design.
Gameable featuresEasy-to-change features such as "number of cards" in a credit score move the score without changing the real risk.
Recommender systemsAn objective that rewards the user's long-term benefit, not engagement.
Unequal costsIf gaming costs differ across groups, strategic behavior creates new unfairness.
The mechanism design question applies here too: honest, rule-following behavior should be in each person's own interest.

Ethics and Governance

Strategic Classification Step by Step

0.20.40.60.81score (true qualification)Abudget 0.15Bbudget 0.03threshold 0.6

Threshold 0.6; filled circle = truly qualified (score ≥ 0.6). People can raise their score at a cost (e.g. number of cards). A's budget is 0.15, B's 0.03.

0.20.40.60.81score (true qualification)Abudget 0.15Bbudget 0.03threshold 0.6

Gaming: anyone within their budget of the threshold moves up to it. 3 unqualified people from A and 1 from B get through (orange ring).

0.20.40.60.81score (true qualification)Abudget 0.15Bbudget 0.03threshold 0.75

The institution responds by raising the threshold to 0.75 (accounting for A's budget). Now no unqualified person gets through.

0.20.40.60.81score (true qualification)Abudget 0.15Bbudget 0.03threshold 0.75

But only 3 of B's 5 qualified people pass, given B's small budget (red); in A, 5 of 5. The defense against strategic behavior created a new inequality (Hu, Immorlica, Vaughan 2019).

Ethics and Governance

Algorithmic Auditing

What is audited?Data sources, the target variable, per-group performance and human oversight, examined together.
DocumentationModel cards and datasheets make assumptions, intended use and known limitations visible.
After deploymentData drift, appeals and unexpected harms are monitored regularly.
ParticipationAffected groups are consulted; a right of appeal and a clear chain of responsibility are defined.
Audit checklist: base rates, selection rate, TPR, FPR, PPV and calibration are reported separately for each group, with a written justification of which criterion was prioritized and why.

Ethics and Governance

Discussion: Can Algorithms Be Neutral?

  • Algorithms are not independent of the values of the institutions that choose their objectives, data, criteria and thresholds.
  • Historical inequalities can seep into training data and target variables.
  • Different fairness criteria give conflicting verdicts on the same decision (the impossibility theorems).
  • After deployment a model can change behavior and create new feedback loops.
Core challengesFairness depends on context; trade-offs among accuracy, privacy and group fairness; cultural differences.
The responseInstead of claiming perfect neutrality: measure impact, document choices, provide an appeals process.

Ethics and Governance

Code Trace: Self-Confirming Data

6region 0actual: 3/day5region 1actual: 3/day0.50.650.801234dayregion 0's share of recordstrue share 0.5

The two regions have equal true incident counts; past records show one more in region 0 (6 vs 5), perhaps by chance.

6region 0actual: 3/day5region 1actual: 3/day◀ patrol0.50.650.801234dayregion 0's share of recordstrue share 0.5

Day 1: the patrol goes to the region with more records ⇒ target = 0.

9region 0actual: 3/day5region 1actual: 3/day◀ patrol0.50.650.801234dayregion 0's share of recordstrue share 0.5

Incidents are recorded only where the patrol goes: records = [9, 5]. The 3 incidents in region 1 stay invisible. Share 0.64.

9region 0actual: 3/day5region 1actual: 3/day◀ patrol0.50.650.801234dayregion 0's share of recordstrue share 0.5

Day 2: the patrol goes to the region with more records ⇒ target = 0.

12region 0actual: 3/day5region 1actual: 3/day◀ patrol0.50.650.801234dayregion 0's share of recordstrue share 0.5

Incidents are recorded only where the patrol goes: records = [12, 5]. The 3 incidents in region 1 stay invisible. Share 0.71.

12region 0actual: 3/day5region 1actual: 3/day◀ patrol0.50.650.801234dayregion 0's share of recordstrue share 0.5

Day 3: the patrol goes to the region with more records ⇒ target = 0.

15region 0actual: 3/day5region 1actual: 3/day◀ patrol0.50.650.801234dayregion 0's share of recordstrue share 0.5

Incidents are recorded only where the patrol goes: records = [15, 5]. The 3 incidents in region 1 stay invisible. Share 0.75.

15region 0actual: 3/day5region 1actual: 3/day◀ patrol0.50.650.801234dayregion 0's share of recordstrue share 0.5

Day 4: the patrol goes to the region with more records ⇒ target = 0.

18region 0actual: 3/day5region 1actual: 3/day◀ patrol0.50.650.801234dayregion 0's share of recordstrue share 0.5

Share 0.55 → 0.78: the data "confirm" the model because they were collected by the model's own decisions (Ensign et al., 2018).

records = [6, 5]   # past records (region 0, 1)
actual = [3, 3]    # true daily incidents: equal!
for day in range(1, 5):
  target = 0 if records[0] >= records[1] else 1
  records[target] += actual[target]  # only there
  share = records[0] / sum(records)
  print(day, target, records, round(share, 2))

Ethics and Governance

Real-World Applications and an Exercise

Credit and bankingScoring; choosing between calibration and equal opportunity.
Hiring and HRRésumé screening; bias in historical data, the four-fifths rule.
HealthcareDiagnosis and treatment recommendations; proxy targets (spending ≠ need).
Exercise: Design a threshold classifier for a small dataset. (1) Build the confusion matrices of two groups. (2) Compute the four group rates (selection rate, TPR, FPR, PPV) with the metrics function. (3) Vary the threshold and report which criterion you can satisfy and what you lose.

Summary

Summary

CriteriaDemographic parity (selection rate), equalized odds (TPR, FPR), predictive parity (PPV), calibration.
Our exampleA calibrated model with equal PPV; still, selection rate 0.5/0.2 and TPR 0.8/0.5.
IncompatibilityFPR = p/(1−p)·(1−PPV)/PPV·TPR: with different base rates the criteria cannot all hold (Chouldechova; Kleinberg et al.).
Fair divisionCut and choose: proportional and envy-free for 2 people. For 3+, proportional ≠ envy-free; EF1 for indivisible goods.
EthicsTransparency, accountability, privacy; GDPR, the EU AI Act.
IncentivesPeople respond to algorithms: Goodhart's law, strategic classification.

Summary

Sample Problems

  1. Group C: TP = 30, FP = 20, FN = 10, TN = 40. Find the selection rate, TPR, FPR and PPV.
  2. Which criteria hold between group C from Problem 1 and group A from this chapter?
  3. If p = 0.2, PPV = 0.5 and TPR = 0.6, what is FPR?
  4. In cut and choose, Ali cuts and knows Ayşe likes chocolate 3 times as much. If Ayşe finds the two pieces equally valuable, she picks the chocolate (right) piece. Where does Ali cut, and what does he get in his own eyes?
  5. Show that with two people, proportionality implies envy-freeness.
  6. In round-robin, if Ayşe went first (same values), who would get what? A person indifferent between two goods takes the lower-numbered one. Is the allocation envy-free?
Show answers
  1. n = 100; selection 0.5; TPR 30/40 = 0.75; FPR 20/60 ≈ 0.333; PPV 30/50 = 0.6.
  2. Only demographic parity (0.5 = 0.5). TPR, FPR and PPV differ.
  3. FPR = 0.2/0.8 · 0.5/0.5 · 0.6 = 0.15.
  4. Ayşe picks the right piece as long as the left is worth ≤ 1/2 to her: 1/4 + 1.5(x − 1/2) ≤ 1/2 ⇒ x ≤ 2/3. Ali cuts at x = 2/3 and gets 2/3 (> 1/2): information favors the cutter.
  5. v(X1) ≥ 1/2 ⇒ v(X2) = 1 − v(X1) ≤ 1/2 ≤ v(X1).
  6. Ayşe m1, Ali m2, Ayşe m3 (1; tied with m4), Ali m4. Ali: 6 vs 10 ⇒ envy, but removing m1 gives 3 ≤ 6: EF1.

Summary

Exercises: New Topics

  1. In the group_b trace, if A's TPR were 0.7, what q would give equal opportunity in B? What would B's accuracy be?
  2. After running cut_point(v_ayse, 0.5, 3), what is [lo, hi] at the end of the loop? How many queries are needed at least so that the returned midpoint is strictly within 0.001 of the true cut point?
  3. In the last diminisher example, if Cem's values were like Ayşe's (0.1, 0.2, 0.3, 0.4), who would get the first piece?
  4. With L = 2, d(x, y) = 0.1 and f(x) = 0.8, what range does individual fairness allow for f(y)?
  5. Run the feedback code with records = [5, 6]. What is region 0's share after 4 days?
Show answers
  1. TPR_B = 0.5 + 0.5·q = 0.7 ⇒ q = 0.4; accuracy (80 − 48·q)/100 = 0.608.
  2. 0.5 → 0.25 < 0.5 (lo = 0.5); 0.75 → 0.625 (hi = 0.75); 0.625 → 0.4375 (lo = 0.625) ⇒ [0.625, 0.75]. Midpoint error ≤ 2−(k+1) ⇒ k = 9.
  3. For Cem, the piece [0, 1/3] is worth 1/6 < 1/3: nobody trims, so the first cutter, Ali, takes it.
  4. |f(y) − 0.8| ≤ 0.2 ⇒ f(y) ∈ [0.6, 1] (a score cannot exceed 1).
  5. The patrol always goes to region 1: records = [5, 18], share = 5/23 ≈ 0.22. The one-record initial gap grows again.

Summary

References (1/2)

  • S. Barocas, M. Hardt, A. Narayanan, Fairness and Machine Learning: Limitations and Opportunities, MIT Press, 2023.
  • A. Chouldechova, "Fair prediction with disparate impact: a study of bias in recidivism prediction instruments", Big Data 5(2), 2017.
  • J. Kleinberg, S. Mullainathan, M. Raghavan, "Inherent trade-offs in the fair determination of risk scores", ITCS, 2017.
  • M. Hardt, E. Price, N. Srebro, "Equality of opportunity in supervised learning", NeurIPS, 2016.
  • J. Angwin, J. Larson, S. Mattu, L. Kirchner, "Machine Bias", ProPublica, 2016.
  • J. Buolamwini, T. Gebru, "Gender Shades", Proc. FAT*, 2018.
  • S. J. Brams, A. D. Taylor, Fair Division: From Cake-Cutting to Dispute Resolution, Cambridge University Press, 1996.
  • C. Dwork, M. Hardt, T. Pitassi, O. Reingold, R. Zemel, "Fairness through awareness", ITCS, 2012.

Summary

References (2/2)

  • D. Ensign et al., "Runaway feedback loops in predictive policing", Proc. FAT*, 2018.
  • L. Hu, N. Immorlica, J. W. Vaughan, "The disparate effects of strategic manipulation", Proc. FAT*, 2019.
  • A. D. Procaccia, "Cake cutting algorithms", in Handbook of Computational Social Choice, Chapter 13, 2016.
  • I. Caragiannis, D. Kurokawa, H. Moulin, A. D. Procaccia, N. Shah, J. Wang, "The unreasonable fairness of maximum Nash welfare", ACM EC, 2016.
  • A. R. Karlin, Y. Peres, Game Theory, Alive, AMS, 2017 (fair division chapter). Course pages: Procaccia, CMU 15-896 · Shah, Toronto CSC304.
  • E. Budish, E. Cantillon, "The multi-unit assignment problem: theory and evidence from course allocation at Harvard", American Economic Review 102(5), 2012.
  • H. Aziz, S. Mackenzie, "A discrete and bounded envy-free cake cutting protocol for any number of agents", FOCS, 2016.
  • L. E. Dubins, E. H. Spanier, "How to cut a cake fairly", American Mathematical Monthly 68(1), 1961; S. Even, A. Paz, "A note on cake cutting", Discrete Applied Mathematics 7(3), 1984.
  • I. Caragiannis, C. Kaklamanis, P. Kanellopoulos, M. Kyropoulou, "The efficiency of fair division", Theory of Computing Systems 50(4), 2012.

Summary

References: Regulation and News

  • J. Dastin, "Amazon scraps secret AI recruiting tool that showed bias against women", Reuters, 10 October 2018.
  • European Parliament and Council, General Data Protection Regulation (GDPR), Regulation (EU) 2016/679, Art. 22 (in force 2018).
  • European Parliament and Council, Artificial Intelligence Act, Regulation (EU) 2024/1689.
  • Law No. 6698 on the Protection of Personal Data (KVKK), Official Gazette of Turkey, 7 April 2016.
  • OECD, Recommendation of the Council on Artificial Intelligence (OECD AI Principles), 2019.

The End

Chapter 11: Algorithmic Fairness and Ethics
Previous: Chapter 10: Multi-Agent Systems · Next: Chapter 12: Behavioral and Collective Decisions
1 / 1 Sercan KÜLCÜ. All rights reserved.