Chapter 13: Learning and Adaptation

Algorithmic Game Theory
Best-response dynamicsFictitious playRegretHedgeCorrelated equilibriumε-greedy and UCBReinforcement learningReplicator dynamicsTit-for-Tat
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

Warm-up: The Multi-Armed Bandit

The rules

Each arm pays 1 point with a hidden probability; pulling an arm shows only its own outcome, and the goal is to maximize total reward.

Think about it
  • Would you stick with Arm 1, which has the best average so far, or try Arm 3 again?
  • Out of a hundred pulls, how many would you spend on trying arms (exploration)?
Arm 13 pulls2 successesavg 0.67p = ?Arm 24 pulls1 successavg 0.25p = ?Arm 31 pull0 successesavg 0p = ?True probabilities are hidden and redrawn in every new trial.Arm 3 has been tried only once.

Introduction

Learning Outcomes

DynamicsRun best-response dynamics and fictitious play step by step; explain when they converge to equilibrium.
RegretCompute external regret; explain why multiplicative weights (Hedge) has no regret.
Exploration–exploitationTrace and compare ε-greedy and UCB on the multi-armed bandit problem.
Evolution and repetitionExplain replicator dynamics and the success of Tit-for-Tat in the repeated Prisoner's Dilemma.

Introduction

Why Learning and Adaptation?

Choose actionstrategy / probabilitiesObserve outcomereward, opponent's actionUpdate ruleweights, averages, QGame+ opponents

Learning: improving a strategy from past experience. Adaptation: updating the strategy as opponents and the environment change.

  • Equilibria can be hard to compute: Nash equilibrium in two-player games is PPAD-complete (so is approximate equilibrium with many players), a class with no known fast algorithm for large games. How do players "find" an equilibrium?
  • They may not know the game: the payoff matrix is unknown; they only play and see the outcome.
  • The question: do simple, local update rules eventually lead to equilibrium, or at least to good performance?
Examples: chess and poker AIs learn through self-play; bidding algorithms in ad auctions adapt daily; navigation apps reroute with traffic.

Introduction

Learning Models: An Overview

ModelWhat does the player observe?How does it update?
Best-response dynamicsThe opponent's last actionPlays a best response to it
Fictitious playThe opponent's whole historyBest response to the frequencies
Regret minimization (Hedge)The payoff of every actionMultiplicatively boosts good actions
Multi-armed bandit (ε-greedy, UCB)Only the chosen action's rewardMean estimate + exploration
Reinforcement learning (Q-learning)Reward and new stateUpdates the Q(s, a) values
Replicator dynamicsAverage success in the populationSuccessful strategies spread

Top to bottom: the player stops watching the opponent and watches its own reward; in the last row the view shifts from the individual to the population.

Best Response and Fictitious Play

Best-Response Dynamics (Step by Step)

Column playerRow playerXYZABC(4,3)(0,2)(1,1)(2,1)(3,3)(0,2)(1,4)(2,0)(3,2)

Starting profile (B, Z). Players take turns best-responding to the opponent's current action.

Column playerRow playerXYZABC(4,3)(0,2)(1,1)(2,1)(3,3)(0,2)(1,4)(2,0)(3,2)

Row looks for a best response to Z: u1 = 1, 0, 3 → C. Profile (C, Z).

Column playerRow playerXYZABC(4,3)(0,2)(1,1)(2,1)(3,3)(0,2)(1,4)(2,0)(3,2)

Column responds to C: u2 = 4, 0, 2 → X. Profile (C, X).

Column playerRow playerXYZABC(4,3)(0,2)(1,1)(2,1)(3,3)(0,2)(1,4)(2,0)(3,2)

Row responds to X: u1 = 4, 2, 1 → A. Profile (A, X).

Column playerRow playerXYZABC(4,3)(0,2)(1,1)(2,1)(3,3)(0,2)(1,4)(2,0)(3,2)

Column already picks X against A (3 > 2, 1), and Row does not leave A → the dynamics stop at the Nash equilibrium (A, X).

Starting from (C, Y) would lead to the equilibrium (B, Y): where you end up depends on where you start.

Best Response and Fictitious Play

When Does Best-Response Dynamics Converge?

MismatcherMatcherHTHT(1,−1)(−1,1)(−1,1)(1,−1)

Matching Pennies: (H,H) → (H,T) → (T,T) → (T,H) → (H,H) … an endless cycle

Potential games: convergesEvery improving deviation increases a common potential function, which cannot increase forever in a finite game (Monderer–Shapley 1996; congestion games).
No pure equilibrium: cyclesIn Matching Pennies every step makes one player happy and the other unhappy.
Remedy: remember the pastRespond to the whole history, not just the last action → fictitious play.

Best Response and Fictitious Play

Fictitious Play: Matching Pennies

tMism. history H/T
(earlier rounds)
MatcherMatch. history H/T
(earlier rounds)
MismatcherMatch. H rate
(incl. this round)
Mism. H rate
(incl. this round)
10 / 0H0 / 0H1.001.00
2
3
4
5
6
7
8
9
10

Round 1 is arbitrary: both play H (no history yet).

tMism. history H/T
(earlier rounds)
MatcherMatch. history H/T
(earlier rounds)
MismatcherMatch. H rate
(incl. this round)
Mism. H rate
(incl. this round)
10 / 0H0 / 0H1.001.00
21 / 0H1 / 0T1.000.50
3
4
5
6
7
8
9
10

The matcher sees 1 / 0 in the opponent's history → H (match the majority, ties go to H). The mismatcher sees 1 / 0 → T (opposite of the majority, ties go to H).

tMism. history H/T
(earlier rounds)
MatcherMatch. history H/T
(earlier rounds)
MismatcherMatch. H rate
(incl. this round)
Mism. H rate
(incl. this round)
10 / 0H0 / 0H1.001.00
21 / 0H1 / 0T1.000.50
31 / 1H2 / 0T1.000.33
4
5
6
7
8
9
10

The matcher sees 1 / 1 in the opponent's history → H (match the majority, ties go to H). The mismatcher sees 2 / 0 → T (opposite of the majority, ties go to H).

tMism. history H/T
(earlier rounds)
MatcherMatch. history H/T
(earlier rounds)
MismatcherMatch. H rate
(incl. this round)
Mism. H rate
(incl. this round)
10 / 0H0 / 0H1.001.00
21 / 0H1 / 0T1.000.50
31 / 1H2 / 0T1.000.33
41 / 2T3 / 0T0.750.25
5
6
7
8
9
10

The matcher sees 1 / 2 in the opponent's history → T (match the majority, ties go to H). The mismatcher sees 3 / 0 → T (opposite of the majority, ties go to H).

tMism. history H/T
(earlier rounds)
MatcherMatch. history H/T
(earlier rounds)
MismatcherMatch. H rate
(incl. this round)
Mism. H rate
(incl. this round)
10 / 0H0 / 0H1.001.00
21 / 0H1 / 0T1.000.50
31 / 1H2 / 0T1.000.33
41 / 2T3 / 0T0.750.25
51 / 3T3 / 1T0.600.20
6
7
8
9
10

The matcher sees 1 / 3 in the opponent's history → T (match the majority, ties go to H). The mismatcher sees 3 / 1 → T (opposite of the majority, ties go to H).

tMism. history H/T
(earlier rounds)
MatcherMatch. history H/T
(earlier rounds)
MismatcherMatch. H rate
(incl. this round)
Mism. H rate
(incl. this round)
10 / 0H0 / 0H1.001.00
21 / 0H1 / 0T1.000.50
31 / 1H2 / 0T1.000.33
41 / 2T3 / 0T0.750.25
51 / 3T3 / 1T0.600.20
61 / 4T3 / 2T0.500.17
7
8
9
10

The matcher sees 1 / 4 in the opponent's history → T (match the majority, ties go to H). The mismatcher sees 3 / 2 → T (opposite of the majority, ties go to H).

tMism. history H/T
(earlier rounds)
MatcherMatch. history H/T
(earlier rounds)
MismatcherMatch. H rate
(incl. this round)
Mism. H rate
(incl. this round)
10 / 0H0 / 0H1.001.00
21 / 0H1 / 0T1.000.50
31 / 1H2 / 0T1.000.33
41 / 2T3 / 0T0.750.25
51 / 3T3 / 1T0.600.20
61 / 4T3 / 2T0.500.17
71 / 5T3 / 3H0.430.29
8
9
10

The matcher sees 1 / 5 in the opponent's history → T (match the majority, ties go to H). The mismatcher sees 3 / 3 → H (opposite of the majority, ties go to H).

tMism. history H/T
(earlier rounds)
MatcherMatch. history H/T
(earlier rounds)
MismatcherMatch. H rate
(incl. this round)
Mism. H rate
(incl. this round)
10 / 0H0 / 0H1.001.00
21 / 0H1 / 0T1.000.50
31 / 1H2 / 0T1.000.33
41 / 2T3 / 0T0.750.25
51 / 3T3 / 1T0.600.20
61 / 4T3 / 2T0.500.17
71 / 5T3 / 3H0.430.29
82 / 5T3 / 4H0.380.38
9
10

The matcher sees 2 / 5 in the opponent's history → T (match the majority, ties go to H). The mismatcher sees 3 / 4 → H (opposite of the majority, ties go to H).

tMism. history H/T
(earlier rounds)
MatcherMatch. history H/T
(earlier rounds)
MismatcherMatch. H rate
(incl. this round)
Mism. H rate
(incl. this round)
10 / 0H0 / 0H1.001.00
21 / 0H1 / 0T1.000.50
31 / 1H2 / 0T1.000.33
41 / 2T3 / 0T0.750.25
51 / 3T3 / 1T0.600.20
61 / 4T3 / 2T0.500.17
71 / 5T3 / 3H0.430.29
82 / 5T3 / 4H0.380.38
93 / 5T3 / 5H0.330.44
10

The matcher sees 3 / 5 in the opponent's history → T (match the majority, ties go to H). The mismatcher sees 3 / 5 → H (opposite of the majority, ties go to H).

tMism. history H/T
(earlier rounds)
MatcherMatch. history H/T
(earlier rounds)
MismatcherMatch. H rate
(incl. this round)
Mism. H rate
(incl. this round)
10 / 0H0 / 0H1.001.00
21 / 0H1 / 0T1.000.50
31 / 1H2 / 0T1.000.33
41 / 2T3 / 0T0.750.25
51 / 3T3 / 1T0.600.20
61 / 4T3 / 2T0.500.17
71 / 5T3 / 3H0.430.29
82 / 5T3 / 4H0.380.38
93 / 5T3 / 5H0.330.44
104 / 5T3 / 6H0.300.50

The matcher sees 4 / 5 in the opponent's history → T (match the majority, ties go to H). The mismatcher sees 3 / 6 → H (opposite of the majority, ties go to H).

Best Response and Fictitious Play

Code Trace: Fictitious Play (Matching Pennies, 6 rounds)

MismatcherHTH(1, −1)(−1, 1)T(−1, 1)(1, −1)Matchernext round's profileMatcher's historyH0T0Mismatcher's historyH0T0

Call fictitious(6). Counters are zero; with no history, both play H in round 1.

MismatcherHTH(1, −1)(−1, 1)T(−1, 1)(1, −1)Matcherplayed this roundMatcher's historyH rate 1.00H1T0Mismatcher's historyH rate 1.00H1T0

Round 1: (H, H) is played and the matcher wins. Counters: matcher H/T = 1/0, mismatcher H/T = 1/0.

MismatcherHTH(1, −1)(−1, 1)T(−1, 1)(1, −1)Matchernext round's profileMatcher's historyH rate 1.00H1T0Mismatcher's historyH rate 1.00H1T0

Next round: the mismatcher mostly played H → matcher H; the matcher mostly played H → mismatcher T.

MismatcherHTH(1, −1)(−1, 1)T(−1, 1)(1, −1)Matcherplayed this roundMatcher's historyH rate 1.00H2T0Mismatcher's historyH rate 0.50H1T1

Round 2: (H, T) is played and the mismatcher wins. Counters: matcher H/T = 2/0, mismatcher H/T = 1/1.

MismatcherHTH(1, −1)(−1, 1)T(−1, 1)(1, −1)Matchernext round's profileMatcher's historyH rate 1.00H2T0Mismatcher's historyH rate 0.50H1T1

Next round: the mismatcher's counters are tied (ties go to H) → matcher H; the matcher mostly played H → mismatcher T.

MismatcherHTH(1, −1)(−1, 1)T(−1, 1)(1, −1)Matcherplayed this roundMatcher's historyH rate 1.00H3T0Mismatcher's historyH rate 0.33H1T2

Round 3: (H, T) is played and the mismatcher wins. Counters: matcher H/T = 3/0, mismatcher H/T = 1/2.

MismatcherHTH(1, −1)(−1, 1)T(−1, 1)(1, −1)Matchernext round's profileMatcher's historyH rate 1.00H3T0Mismatcher's historyH rate 0.33H1T2

Next round: the mismatcher mostly played T → matcher T; the matcher mostly played H → mismatcher T.

MismatcherHTH(1, −1)(−1, 1)T(−1, 1)(1, −1)Matcherplayed this roundMatcher's historyH rate 0.75H3T1Mismatcher's historyH rate 0.25H1T3

Round 4: (T, T) is played and the matcher wins. Counters: matcher H/T = 3/1, mismatcher H/T = 1/3.

MismatcherHTH(1, −1)(−1, 1)T(−1, 1)(1, −1)Matchernext round's profileMatcher's historyH rate 0.75H3T1Mismatcher's historyH rate 0.25H1T3

Next round: the mismatcher mostly played T → matcher T; the matcher mostly played H → mismatcher T.

MismatcherHTH(1, −1)(−1, 1)T(−1, 1)(1, −1)Matcherplayed this roundMatcher's historyH rate 0.60H3T2Mismatcher's historyH rate 0.20H1T4

Round 5: (T, T) is played and the matcher wins. Counters: matcher H/T = 3/2, mismatcher H/T = 1/4.

MismatcherHTH(1, −1)(−1, 1)T(−1, 1)(1, −1)Matchernext round's profileMatcher's historyH rate 0.60H3T2Mismatcher's historyH rate 0.20H1T4

Next round: the mismatcher mostly played T → matcher T; the matcher mostly played H → mismatcher T.

MismatcherHTH(1, −1)(−1, 1)T(−1, 1)(1, −1)Matcherplayed this roundMatcher's historyH rate 0.50H3T3Mismatcher's historyH rate 0.17H1T5

Round 6: (T, T) is played and the matcher wins. Counters: matcher H/T = 3/3, mismatcher H/T = 1/5.

MismatcherHTH(1, −1)(−1, 1)T(−1, 1)(1, −1)Matcherplayed this roundMatcher's historyH rate 0.50H3T3Mismatcher's historyH rate 0.17H1T5

return nm = [3, 3], nx = [1, 5]. The actions cycle (HH → HT → TT → TH …); as the cycles lengthen, the frequencies approach ½ (0.50 and 0.17 after 6 rounds; 0.472 and 0.492 after 400).

def fictitious(T):
    nm, nx = [0, 0], [0, 0]   # H/T counters
    a, b = 0, 0               # round 1: both H
    for t in range(1, T + 1):
        nm[a] += 1            # matcher's history
        nx[b] += 1            # mismatcher's history
        # matcher: match the mismatcher's majority
        a = 0 if nx[0] >= nx[1] else 1
        # mismatcher: avoid the matcher's majority
        b = 1 if nm[0] > nm[1] else 0
    return nm, nx

Best Response and Fictitious Play

Fictitious Play: Frequencies Converge to Equilibrium

110020030040000.51round tfrequency of HMatcherMismatcherNash: 0.5
  • Actions change in ever-longer cycles, but the frequencies approach the mixed equilibrium (½, ½): 0.472 and 0.492 after 400 rounds.
  • Guaranteed convergence: two-player zero-sum games (Robinson 1951), 2×2 games, potential games. Shapley (1964): in a 3×3 game the frequencies cycle and never converge.

Regret and Hedge

External Regret: Why Does "Follow the Leader" Fail?

Regret = (the total you would have earned by always playing the best single action in hindsight) − (the total you actually earned). External regret RT = maxi Σt gt(i) − Σt gt(at) (using expected payoff for randomized choices): the gap to the best fixed action in hindsight. If RT/T → 0, the algorithm has no regret.

tTotals so far A / Bg(A)g(B)Follow the leaderPayoff
10.0 / 0.00.50.0A0.5
2
3
4
5
6

Leader A (totals 0.0 / 0.0) → A is chosen; round payoff 0.5. Algorithm's total: 0.5.

tTotals so far A / Bg(A)g(B)Follow the leaderPayoff
10.0 / 0.00.50.0A0.5
20.5 / 0.00.01.0A0.0
3
4
5
6

Leader A (totals 0.5 / 0.0) → A is chosen; round payoff 0.0. Algorithm's total: 0.5.

tTotals so far A / Bg(A)g(B)Follow the leaderPayoff
10.0 / 0.00.50.0A0.5
20.5 / 0.00.01.0A0.0
30.5 / 1.01.00.0B0.0
4
5
6

Leader B (totals 0.5 / 1.0) → B is chosen; round payoff 0.0. Algorithm's total: 0.5.

tTotals so far A / Bg(A)g(B)Follow the leaderPayoff
10.0 / 0.00.50.0A0.5
20.5 / 0.00.01.0A0.0
30.5 / 1.01.00.0B0.0
41.5 / 1.00.01.0A0.0
5
6

Leader A (totals 1.5 / 1.0) → A is chosen; round payoff 0.0. Algorithm's total: 0.5.

tTotals so far A / Bg(A)g(B)Follow the leaderPayoff
10.0 / 0.00.50.0A0.5
20.5 / 0.00.01.0A0.0
30.5 / 1.01.00.0B0.0
41.5 / 1.00.01.0A0.0
51.5 / 2.01.00.0B0.0
6

Leader B (totals 1.5 / 2.0) → B is chosen; round payoff 0.0. Algorithm's total: 0.5.

tTotals so far A / Bg(A)g(B)Follow the leaderPayoff
10.0 / 0.00.50.0A0.5
20.5 / 0.00.01.0A0.0
30.5 / 1.01.00.0B0.0
41.5 / 1.00.01.0A0.0
51.5 / 2.01.00.0B0.0
62.5 / 2.00.01.0A0.0

Leader A (totals 2.5 / 2.0) → A is chosen; round payoff 0.0. Algorithm's total: 0.5. Best fixed action B: 3. Regret = 3 − 0.5 = 2.5 ≈ T/2.

Regret and Hedge

Multiplicative Weights / Hedge

  1. Give each action a weight: wi = 1.
  2. Each round, play with probabilities pi = wi / Σj wj.
  3. Observe every action's payoff and update:
    wi ← wi · eη·gt(i)An action with payoff 1 has its weight multiplied by eη (exactly doubled if η = ln 2).
Theorem. With payoffs in [0, 1] and n actions, RT ≤ (ln n)/η + ηT (ln n is the natural log of n; ≈ 1.1 for n = 3). Choosing η = √(ln n / T) gives RT ≤ 2√(T ln n), so RT/T → 0.
Randomness is essentialFollow the leader is deterministic, so an adversary exploits it; Hedge plays mixed.
η: the learning rateA large η reacts fast but is sensitive to noise; a small η is slow but stable.
In gamesIf everyone plays a no-regret algorithm, the time average approaches a coarse correlated equilibrium, and minimax in zero-sum games.

Regret and Hedge

Code Trace: Multiplicative Weights

p=0.33Aw=1.00p=0.33Bw=1.00p=0.33Cw=1.00

Three experts (actions) A, B, C; η = 0.5. Initial weights are equal: w = [1, 1, 1].

p=0.33Aw=1.00p=0.33Bw=1.00p=0.33Cw=1.00

Round 1: W = 3.00, probabilities p = [0.33, 0.33, 0.33].

p=0.33Aw=1.00 g=1.0p=0.33Bw=1.00 g=0.0p=0.33Cw=1.00 g=0.5

Payoffs revealed: g = [1, 0, 0.5]. Expected payoff Σ p·g = 0.50 → total = 0.50.

p=0.42Aw=1.65 g=1.0p=0.25Bw=1.00 g=0.0p=0.33Cw=1.28 g=0.5

Update w ← w·e0.5·g: w = [1.65, 1.00, 1.28]. Experts that did well this round gain weight.

p=0.42Aw=1.65p=0.25Bw=1.00p=0.33Cw=1.28

Round 2: W = 3.93, probabilities p = [0.42, 0.25, 0.33].

p=0.42Aw=1.65 g=1.0p=0.25Bw=1.00 g=0.0p=0.33Cw=1.28 g=0.5

Payoffs revealed: g = [1, 0, 0.5]. Expected payoff Σ p·g = 0.58 → total = 1.08.

p=0.51Aw=2.72 g=1.0p=0.19Bw=1.00 g=0.0p=0.31Cw=1.65 g=0.5

Update w ← w·e0.5·g: w = [2.72, 1.00, 1.65]. Experts that did well this round gain weight.

p=0.51Aw=2.72p=0.19Bw=1.00p=0.31Cw=1.65

Round 3: W = 5.37, probabilities p = [0.51, 0.19, 0.31].

p=0.51Aw=2.72 g=0.0p=0.19Bw=1.00 g=1.0p=0.31Cw=1.65 g=0.5

Payoffs revealed: g = [0, 1, 0.5]. Expected payoff Σ p·g = 0.34 → total = 1.42.

p=0.42Aw=2.72 g=0.0p=0.25Bw=1.65 g=1.0p=0.33Cw=2.12 g=0.5

Update w ← w·e0.5·g: w = [2.72, 1.65, 2.12]. Experts that did well this round gain weight.

p=0.42Aw=2.72p=0.25Bw=1.65p=0.33Cw=2.12

Round 4: W = 6.48, probabilities p = [0.42, 0.25, 0.33].

p=0.42Aw=2.72 g=0.0p=0.25Bw=1.65 g=1.0p=0.33Cw=2.12 g=0.5

Payoffs revealed: g = [0, 1, 0.5]. Expected payoff Σ p·g = 0.42 → total = 1.84.

p=0.33Aw=2.72 g=0.0p=0.33Bw=2.72 g=1.0p=0.33Cw=2.72 g=0.5

Update w ← w·e0.5·g: w = [2.72, 2.72, 2.72]. Experts that did well this round gain weight.

p=0.33Aw=2.72p=0.33Bw=2.72p=0.33Cw=2.72

Round 5: W = 8.15, probabilities p = [0.33, 0.33, 0.33].

p=0.33Aw=2.72 g=0.0p=0.33Bw=2.72 g=1.0p=0.33Cw=2.72 g=0.5

Payoffs revealed: g = [0, 1, 0.5]. Expected payoff Σ p·g = 0.50 → total = 2.34.

p=0.25Aw=2.72 g=0.0p=0.42Bw=4.48 g=1.0p=0.33Cw=3.49 g=0.5

Update w ← w·e0.5·g: w = [2.72, 4.48, 3.49]. Experts that did well this round gain weight.

p=0.25Aw=2.72p=0.42Bw=4.48p=0.33Cw=3.49

return 2.34. Best fixed expert B: 3. Regret = 3 − 2.34 = 0.66 (follow the leader can do worse in 5 rounds).

import math
def hedge(G, eta):
    w = [1.0] * len(G[0])
    total = 0.0
    for g in G:                       # each round
        W = sum(w)
        p = [x / W for x in w]
        total += sum(pi * gi for pi, gi in zip(p, g))
        w = [x*math.exp(eta*gi) for x, gi in zip(w, g)]
    return total

Regret and Hedge

Change the Assumption: What If the Opponent Learns Too?

Rock–Paper–Scissors: a win pays 1, a tie ½, a loss 0. Player 1 learns with Hedge (fixed η = 0.1; results are simulated). If the opponent always plays (Rock 0.5, Paper 0.3, Scissors 0.2), where does Hedge go? What changes if the opponent also learns with Hedge?

Fixed opponent: Hedge goes to the best response. Expected payoffs: Rock 0.45, Paper 0.65, Scissors 0.40. Hedge's probability of Paper is 0.82 at round 100 and 0.997 at round 300.

Learning opponent: the target moves. Both update at once; player 1 starts at (0.5, 0.3, 0.2), player 2 uniform.

Rock–Paper–Scissors: a win pays 1, a tie ½, a loss 0. Player 1 learns with Hedge (fixed η = 0.1; results are simulated). If the opponent always plays (Rock 0.5, Paper 0.3, Scissors 0.2), where does Hedge go? What changes if the opponent also learns with Hedge?

Fixed opponent: Hedge goes to the best response. Expected payoffs: Rock 0.45, Paper 0.65, Scissors 0.40. Hedge's probability of Paper is 0.82 at round 100 and 0.997 at round 300.

Learning opponent: the target moves. Both update at once; player 1 starts at (0.5, 0.3, 0.2), player 2 uniform.

Rock–Paper–Scissors: a win pays 1, a tie ½, a loss 0. Player 1 learns with Hedge (fixed η = 0.1; results are simulated). If the opponent always plays (Rock 0.5, Paper 0.3, Scissors 0.2), where does Hedge go? What changes if the opponent also learns with Hedge?

Fixed opponent: Hedge goes to the best response. Expected payoffs: Rock 0.45, Paper 0.65, Scissors 0.40. Hedge's probability of Paper is 0.82 at round 100 and 0.997 at round 300.

Learning opponent: the target moves. Both update at once; player 1 starts at (0.5, 0.3, 0.2), player 2 uniform.

round1002008001000
P(Rock) in that round0.190.460.140.19
average P(Rock)0.350.310.330.34
The last-round strategy keeps cycling; the time average approaches the equilibrium (⅓, ⅓, ⅓).

Regret and Hedge

Hedge vs Itself: Matching Pennies

115030045060000.51round tMatcher's P(H)last iteratetime average
  • Both players use Hedge (η = 0.1). Initial weights: matcher (H, T) = (0.8, 0.2), so P(H) = 0.8; mismatcher (0.5, 0.5). A win pays 1, a loss 0; each round both players update simultaneously with the expected payoffs of their actions.
  • The probability in each round (the last iterate) never settles at equilibrium; it swings widely around it. The time average approaches the minimax strategy (½, ½) (0.498 after 600 rounds). This is the basis of the learning proof of the minimax theorem.

Regret and Hedge

Which Equilibrium Does No-Regret Learning Reach?

Coarse correlated equilibrium (CCE)← no-regret dynamics (Hedge)Correlated equilibrium (CE)← no-internal-regret dynamicsMixed Nash equilibriumalways exists, but PPAD-completePure Nash equilibriummay not exist
  • A mediator tells everyone what to play (like a traffic light).
  • Correlated equilibrium (CE): following your recommendation is best even after seeing it. With algorithms that minimize a stronger kind of regret (internal regret), the time average approaches the set of CE.
  • Coarse correlated equilibrium (CCE) is weaker: committing to "I'll always play this" before seeing the recommendation does not pay. If everyone plays no-regret, the time average approaches a CCE.
  • CE and CCE are easy to compute; mixed Nash equilibrium is PPAD-complete.
A traffic light at a crossing implements a correlated equilibrium of the game of Chicken: no one gains by ignoring the light. No-regret learning finds such equilibria without a central planner.

Multi-Armed Bandits

The Multi-Armed Bandit

  • A K-armed slot machine: arm i pays a random reward with (unknown) mean μi.
  • Each round one arm is pulled and only its reward is seen (unlike Hedge: partial feedback).
  • Regret: RT = T·μ* − Σt μat, where μ* is the best arm's mean.
The exploration–exploitation dilemma: exploration gathers information about new arms; exploitation uses the arm that looks best so far. Pure exploitation gets stuck on the wrong arm; pure exploration wastes reward.
00.511.5UCB 1.13arm 0 · N=9UCB 1.09arm 1 · N=8UCB 1.18arm 2 · N=23● Q ▮ Q ± bonus - - true μ

UCB1, μ = (0.3, 0.5, 0.7), 40 rounds: a rarely tried arm has a wide interval, so it gets tried again.

Applications: ad and news recommendation, A/B testing, clinical trials, price experimentation.

Multi-Armed Bandits

Code Trace: ε-Greedy (ε = 0.2)

Q=0.00arm 0 · N=0Q=0.00arm 1 · N=0Q=0.00arm 2 · N=0- - true μ = 0.3 · 0.5 · 0.7

Each round ε-greedy pulls a random arm with probability ε (explore), otherwise the arm with the largest Q (exploit). Call: egreedy([0.3, 0.5, 0.7], 8, 0.2, random.Random(11)). Estimates Q = 0, counts N = 0. The code numbers arms from 0: arms 0, 1, 2 = Arms 1, 2, 3.

Q=0.00arm 0 · N=0Q=0.00arm 1 · N=0Q=0.00arm 2 · N=0- - true μ = 0.3 · 0.5 · 0.7

t = 1: 0.452 ≥ 0.2 → exploit: the arm with the largest Q is 0 (ties go to the first index).

Q=0.00arm 0 · N=1 · r=0Q=0.00arm 1 · N=0Q=0.00arm 2 · N=0- - true μ = 0.3 · 0.5 · 0.7

Reward r = 0; N[0] = 1, Q[0] = 0.00 (incremental mean).

Q=0.00arm 0 · N=1Q=0.00arm 1 · N=0Q=0.00arm 2 · N=0- - true μ = 0.3 · 0.5 · 0.7

t = 2: 0.924 ≥ 0.2 → exploit: the arm with the largest Q is 0 (ties go to the first index).

Q=0.00arm 0 · N=2 · r=0Q=0.00arm 1 · N=0Q=0.00arm 2 · N=0- - true μ = 0.3 · 0.5 · 0.7

Reward r = 0; N[0] = 2, Q[0] = 0.00 (incremental mean).

Q=0.00arm 0 · N=2Q=0.00arm 1 · N=0Q=0.00arm 2 · N=0- - true μ = 0.3 · 0.5 · 0.7

t = 3: 0.508 ≥ 0.2 → exploit: the arm with the largest Q is 0 (ties go to the first index).

Q=0.00arm 0 · N=3 · r=0Q=0.00arm 1 · N=0Q=0.00arm 2 · N=0- - true μ = 0.3 · 0.5 · 0.7

Reward r = 0; N[0] = 3, Q[0] = 0.00 (incremental mean).

Q=0.00arm 0 · N=3Q=0.00arm 1 · N=0Q=0.00arm 2 · N=0- - true μ = 0.3 · 0.5 · 0.7

t = 4: random number 0.185 < 0.2 → explore, random arm 2.

Q=0.00arm 0 · N=3Q=0.00arm 1 · N=0Q=1.00arm 2 · N=1 · r=1- - true μ = 0.3 · 0.5 · 0.7

Reward r = 1; N[2] = 1, Q[2] = 1.00 (incremental mean).

Q=0.00arm 0 · N=3Q=0.00arm 1 · N=0Q=1.00arm 2 · N=1- - true μ = 0.3 · 0.5 · 0.7

t = 5: 0.614 ≥ 0.2 → exploit: the arm with the largest Q is 2.

Q=0.00arm 0 · N=3Q=0.00arm 1 · N=0Q=1.00arm 2 · N=2 · r=1- - true μ = 0.3 · 0.5 · 0.7

Reward r = 1; N[2] = 2, Q[2] = 1.00 (incremental mean).

Q=0.00arm 0 · N=3Q=0.00arm 1 · N=0Q=1.00arm 2 · N=2- - true μ = 0.3 · 0.5 · 0.7

t = 6: 0.447 ≥ 0.2 → exploit: the arm with the largest Q is 2.

Q=0.00arm 0 · N=3Q=0.00arm 1 · N=0Q=1.00arm 2 · N=3 · r=1- - true μ = 0.3 · 0.5 · 0.7

Reward r = 1; N[2] = 3, Q[2] = 1.00 (incremental mean).

Q=0.00arm 0 · N=3Q=0.00arm 1 · N=0Q=1.00arm 2 · N=3- - true μ = 0.3 · 0.5 · 0.7

t = 7: 0.539 ≥ 0.2 → exploit: the arm with the largest Q is 2.

Q=0.00arm 0 · N=3Q=0.00arm 1 · N=0Q=0.75arm 2 · N=4 · r=0- - true μ = 0.3 · 0.5 · 0.7

Reward r = 0; N[2] = 4, Q[2] = 0.75 (incremental mean).

Q=0.00arm 0 · N=3Q=0.00arm 1 · N=0Q=0.75arm 2 · N=4- - true μ = 0.3 · 0.5 · 0.7

t = 8: 0.634 ≥ 0.2 → exploit: the arm with the largest Q is 2.

Q=0.00arm 0 · N=3Q=0.00arm 1 · N=0Q=0.80arm 2 · N=5 · r=1- - true μ = 0.3 · 0.5 · 0.7

Reward r = 1; N[2] = 5, Q[2] = 0.80 (incremental mean).

Q=0.00arm 0 · N=3Q=0.00arm 1 · N=0Q=0.80arm 2 · N=5- - true μ = 0.3 · 0.5 · 0.7

return N = [3, 0, 5]. The first 3 rounds got stuck on arm 0 (all rewards 0, ties go to the first arm); a single exploration step found the best arm (2).

def egreedy(mu, T, eps, rng):
    N, Q = [0] * len(mu), [0.0] * len(mu)
    for t in range(1, T + 1):
        if rng.random() < eps:           # explore
            a = rng.randrange(len(mu))
        else:                            # exploit
            a = Q.index(max(Q))
        r = 1 if rng.random() < mu[a] else 0
        N[a] += 1
        Q[a] += (r - Q[a]) / N[a]
    return N, Q

Multi-Armed Bandits

Code Trace: UCB1 (Upper Confidence Bound)

Q=0.00arm 0 · N=0Q=0.00arm 1 · N=0Q=0.00arm 2 · N=0- - true μ = 0.3 · 0.5 · 0.7

Call: ucb([0.3, 0.5, 0.7], 8, random.Random(3)). Estimates and counts start at zero. Score Q + √(2 ln t / N): t is the current round, N the arm's pull count. Rarely pulled arms get a large bonus.

Q=0.00arm 0 · N=0Q=0.00arm 1 · N=0Q=0.00arm 2 · N=0- - true μ = 0.3 · 0.5 · 0.7

t = 1: an untried arm has bonus ∞ → the first untried arm 0 is chosen (each arm once).

Q=1.00arm 0 · N=1 · r=1Q=0.00arm 1 · N=0Q=0.00arm 2 · N=0- - true μ = 0.3 · 0.5 · 0.7

Reward r = 1; Q[0] = 1.00, N[0] = 1.

Q=1.00arm 0 · N=1Q=0.00arm 1 · N=0Q=0.00arm 2 · N=0- - true μ = 0.3 · 0.5 · 0.7

t = 2: an untried arm has bonus ∞ → the first untried arm 1 is chosen (each arm once).

Q=1.00arm 0 · N=1Q=0.00arm 1 · N=1 · r=0Q=0.00arm 2 · N=0- - true μ = 0.3 · 0.5 · 0.7

Reward r = 0; Q[1] = 0.00, N[1] = 1.

Q=1.00arm 0 · N=1Q=0.00arm 1 · N=1Q=0.00arm 2 · N=0- - true μ = 0.3 · 0.5 · 0.7

t = 3: an untried arm has bonus ∞ → the first untried arm 2 is chosen (each arm once).

Q=1.00arm 0 · N=1Q=0.00arm 1 · N=1Q=1.00arm 2 · N=1 · r=1- - true μ = 0.3 · 0.5 · 0.7

Reward r = 1; Q[2] = 1.00, N[2] = 1.

2.67Q=1.00arm 0 · N=11.67Q=0.00arm 1 · N=12.67Q=1.00arm 2 · N=1- - true μ = 0.3 · 0.5 · 0.7

t = 4: bonus √(2 ln t / N) = [1.67, 1.67, 1.67]; UCB = Q + b = [2.67, 1.67, 2.67] → largest: arm 0.

Q=0.50arm 0 · N=2 · r=0Q=0.00arm 1 · N=1Q=1.00arm 2 · N=1- - true μ = 0.3 · 0.5 · 0.7

Reward r = 0; Q[0] = 0.50, N[0] = 2.

1.77Q=0.50arm 0 · N=21.79Q=0.00arm 1 · N=12.79Q=1.00arm 2 · N=1- - true μ = 0.3 · 0.5 · 0.7

t = 5: bonus √(2 ln t / N) = [1.27, 1.79, 1.79]; UCB = Q + b = [1.77, 1.79, 2.79] → largest: arm 2.

Q=0.50arm 0 · N=2Q=0.00arm 1 · N=1Q=1.00arm 2 · N=2 · r=1- - true μ = 0.3 · 0.5 · 0.7

Reward r = 1; Q[2] = 1.00, N[2] = 2.

1.84Q=0.50arm 0 · N=21.89Q=0.00arm 1 · N=12.34Q=1.00arm 2 · N=2- - true μ = 0.3 · 0.5 · 0.7

t = 6: bonus √(2 ln t / N) = [1.34, 1.89, 1.34]; UCB = Q + b = [1.84, 1.89, 2.34] → largest: arm 2.

Q=0.50arm 0 · N=2Q=0.00arm 1 · N=1Q=1.00arm 2 · N=3 · r=1- - true μ = 0.3 · 0.5 · 0.7

Reward r = 1; Q[2] = 1.00, N[2] = 3.

1.89Q=0.50arm 0 · N=21.97Q=0.00arm 1 · N=12.14Q=1.00arm 2 · N=3- - true μ = 0.3 · 0.5 · 0.7

t = 7: bonus √(2 ln t / N) = [1.39, 1.97, 1.14]; UCB = Q + b = [1.89, 1.97, 2.14] → largest: arm 2.

Q=0.50arm 0 · N=2Q=0.00arm 1 · N=1Q=1.00arm 2 · N=4 · r=1- - true μ = 0.3 · 0.5 · 0.7

Reward r = 1; Q[2] = 1.00, N[2] = 4.

1.94Q=0.50arm 0 · N=22.04Q=0.00arm 1 · N=12.02Q=1.00arm 2 · N=4- - true μ = 0.3 · 0.5 · 0.7

t = 8: bonus √(2 ln t / N) = [1.44, 2.04, 1.02]; UCB = Q + b = [1.94, 2.04, 2.02] → largest: arm 1.

Q=0.50arm 0 · N=2Q=0.00arm 1 · N=2 · r=0Q=1.00arm 2 · N=4- - true μ = 0.3 · 0.5 · 0.7

Reward r = 0; Q[1] = 0.00, N[1] = 2.

Q=0.50arm 0 · N=2Q=0.00arm 1 · N=2Q=1.00arm 2 · N=4- - true μ = 0.3 · 0.5 · 0.7

return N = [2, 2, 4]. A rarely tried arm keeps a large bonus: at t = 8, arm 1 (one try, Q = 0) was tried again — exploration comes for free.

import math
def ucb(mu, T, rng):
    K = len(mu)
    N, Q = [0] * K, [0.0] * K
    for t in range(1, T + 1):
        b = [math.inf if n == 0 else
             math.sqrt(2*math.log(t)/n) for n in N]
        a = max(range(K), key=lambda i: Q[i] + b[i])
        r = 1 if rng.random() < mu[a] else 0
        N[a] += 1
        Q[a] += (r - Q[a]) / N[a]
    return N, Q

Multi-Armed Bandits

Comparison: 5000 Rounds, 100 Runs

0100020003000400050000306090120150round taverage cumulative regret← pure exploitation: 0.4·T = 2000 (off the chart)ε-greedy (0.1): 118UCB1: 84
  • Arms μ = (0.3, 0.5, 0.7), seeds 0–99. Pure exploitation gets stuck on arm 0 every time: regret 0.4·T, linear.
  • ε-greedy keeps exploring 10% of the time, so its regret grows linearly. UCB1's regret is above ε-greedy's from roughly round 270 to round 2,370 and stays below afterwards, because UCB1's regret is logarithmic, i.e. it grows very slowly (Auer et al., 2002).
  • ε is a tuning parameter: 0.2 in the short trace so that exploration shows up, 0.1 in the long comparison, 0.15 in the bandit game. A smaller ε means less exploration and later correction.

Multi-Armed Bandits

Back to the Game: The Multi-Armed Bandit

Theory's answer: exploration has a cost

Arms μ = (0.3, 0.5, 0.7), 100 rounds:

StrategyRewardRegret
always random (explore only)5020
stick with the 0.5 arm5020
ε-greedy, ε = 0.15 (best found)673
always the best arm700

ε-greedy explores 15% of rounds forever; UCB1 explores rarely tried arms, with logarithmic regret.

The opening question

One failed pull of Arm 3 says little. At t = 9 UCB1 picks Arm 3: 0 + √(2 ln 9 / 1) ≈ 2.10 vs. 0.67 + √(2 ln 9 / 3) ≈ 1.88 for Arm 1.

In practice: in a four-armed bandit task, human choices fit softmax on value differences better than a UCB-like uncertainty bonus (Daw et al., 2006).

One run is misleading: rewards are random, so compare averages over many runs.

Reinforcement Learning

From Bandits to Reinforcement Learning

  • A bandit is a one-state problem. In reinforcement learning an action determines not only the reward but also the next state (where the agent is, e.g. the position on a board).
  • Q-learning (Watkins, 1989): updates the state–action value from an observed transition
Q(s, a) ← Q(s, a) + α [ r + γ · maxa′ Q(s′, a′) − Q(s, a) ]

α: the learning rate (how much to trust new information); γ: the discount factor for future reward.

Numerical exampleQ(s, a) = 2, r = 1, γ = 0.9, max Q(s′, ·) = 4, α = 0.5:Q ← 2 + 0.5·(1 + 3.6 − 2) = 3.3
With multiple agentsOther agents are learning too, so from each agent's view the rules of the game keep changing (the environment is non-stationary); single-agent convergence guarantees no longer hold. Self-play (AlphaZero) is one way out.

Reinforcement Learning

Code Trace: Q-Learning (α = 0.5, γ = 0.9)

s0← 0.000→ 0.000s1← 0.000→ 0.000s2← 0.000→ 0.000s3 · goalr = 1, episode ends

Call q_learn(ep, 0.5, 0.9): 3 episodes starting at s0. All Q values are 0; s3 is the goal (+1).

s0← 0.000→ 0.000s1← 0.000→ 0.000s2← 0.000→ 0.000s3 · goalr = 1, episode endsright, r = 0

Episode 1: s0 → right → s1, r = 0. best = max Q(s1, ·) = 0.000; target = 0.000; Q(s0,right) = 0.000 + 0.5·(0.000 − 0.000) = 0.000.

s0← 0.000→ 0.000s1← 0.000→ 0.000s2← 0.000→ 0.000s3 · goalr = 1, episode endsright, r = 0

Episode 1: s1 → right → s2, r = 0. best = max Q(s2, ·) = 0.000; target = 0.000; Q(s1,right) = 0.000 + 0.5·(0.000 − 0.000) = 0.000.

s0← 0.000→ 0.000s1← 0.000→ 0.000s2← 0.000→ 0.500s3 · goalr = 1, episode endsright, r = 1

Episode 1: s2 → right → s3, r = 1. s3 is the goal → best = 0; target = 1.000; Q(s2,right) = 0.000 + 0.5·(1.000 − 0.000) = 0.500.

s0← 0.000→ 0.000s1← 0.000→ 0.000s2← 0.000→ 0.500s3 · goalr = 1, episode endsright, r = 0

Episode 2: s0 → right → s1, r = 0. best = max Q(s1, ·) = 0.000; target = 0.000; Q(s0,right) = 0.000 + 0.5·(0.000 − 0.000) = 0.000.

s0← 0.000→ 0.000s1← 0.000→ 0.225s2← 0.000→ 0.500s3 · goalr = 1, episode endsright, r = 0

Episode 2: s1 → right → s2, r = 0. best = max Q(s2, ·) = 0.500; target = 0.450; Q(s1,right) = 0.000 + 0.5·(0.450 − 0.000) = 0.225.

s0← 0.000→ 0.000s1← 0.000→ 0.225s2← 0.000→ 0.750s3 · goalr = 1, episode endsright, r = 1

Episode 2: s2 → right → s3, r = 1. s3 is the goal → best = 0; target = 1.000; Q(s2,right) = 0.500 + 0.5·(1.000 − 0.500) = 0.750.

s0← 0.000→ 0.101s1← 0.000→ 0.225s2← 0.000→ 0.750s3 · goalr = 1, episode endsright, r = 0

Episode 3: s0 → right → s1, r = 0. best = max Q(s1, ·) = 0.225; target = 0.203; Q(s0,right) = 0.000 + 0.5·(0.203 − 0.000) = 0.101.

s0← 0.000→ 0.101s1← 0.046→ 0.225s2← 0.000→ 0.750s3 · goalr = 1, episode endsleft, r = 0

Episode 3: s1 → left → s0, r = 0. best = max Q(s0, ·) = 0.101; target = 0.091; Q(s1,left) = 0.000 + 0.5·(0.091 − 0.000) = 0.046.

s0← 0.000→ 0.152s1← 0.046→ 0.225s2← 0.000→ 0.750s3 · goalr = 1, episode endsright, r = 0

Episode 3: s0 → right → s1, r = 0. best = max Q(s1, ·) = 0.225; target = 0.203; Q(s0,right) = 0.101 + 0.5·(0.203 − 0.101) = 0.152.

s0← 0.000→ 0.152s1← 0.046→ 0.450s2← 0.000→ 0.750s3 · goalr = 1, episode endsright, r = 0

Episode 3: s1 → right → s2, r = 0. best = max Q(s2, ·) = 0.750; target = 0.675; Q(s1,right) = 0.225 + 0.5·(0.675 − 0.225) = 0.450.

s0← 0.000→ 0.152s1← 0.046→ 0.450s2← 0.000→ 0.875s3 · goalr = 1, episode endsright, r = 1

Episode 3: s2 → right → s3, r = 1. s3 is the goal → best = 0; target = 1.000; Q(s2,right) = 0.750 + 0.5·(1.000 − 0.750) = 0.875.

s0← 0.000→ 0.152s1← 0.046→ 0.450s2← 0.000→ 0.875s3 · goalr = 1, episode ends

return Q. Reward propagated back from the goal: Q(s2,right) = 0.875 → Q(s1,right) = 0.450 → Q(s0,right) = 0.152. In every cell Q(right) > Q(left): the greedy policy is "always go right".

def q_learn(episodes, alpha, gamma):
    Q = [[0.0, 0.0] for s in range(4)]  # [left, right]
    for ep in episodes:
        for s, a, r, s2 in ep:
            best = 0 if s2 == 3 else max(Q[s2])
            target = r + gamma * best
            Q[s][a] += alpha * (target - Q[s][a])
    return Q

Reinforcement Learning

Beyond Q-Learning: Methods and Milestones

  • Thompson sampling (1933): keeps a distribution over each arm's possible mean (the posterior), draws a sample from it each round and pulls the arm with the largest sample.
  • Win-Stay, Lose-Shift (WSLS; Nowak–Sigmund, 1993): after a 3 or 5 repeat the move, after a 0 or 1 switch; in the Prisoner's Dilemma it recovers cooperation two rounds after a single error.
  • Others: EXP3 (Hedge that sees only the pulled arm's reward; its regret grows slower than T even if an adversary picks the rewards), SARSA and DQN (a neural network instead of a Q table), CFR (a regret-based method for poker).
YearSystemMethod
1992TD-Gammon (backgammon)self-play
2017–19Libratus, Pluribus (poker)CFR (regret)
2018AlphaZero (chess, Go)self-play
Games are a good laboratory for RL: clear rules, a clear reward, and millions of repetitions; in poker, regret-based methods came out on top.

Evolutionary Dynamics

Replicator Dynamics

  • In a large population, individuals are matched at random; each plays a single strategy and passes it on to its offspring. No rationality, just selection.
  • fi: the average payoff of strategy i (its fitness); f̄: the population's average payoff. Strategies that earn above average spread.
  • The discrete form is enough for this course: new share = old share × fi / f̄ (a baseline of 3 is added to every payoff to keep fitness positive).
  • Evolutionarily stable strategy (ESS): a small group of mutants cannot invade (Maynard Smith–Price, 1973).
OpponentIndividualHawkDoveHawkDove(−1,−1)(4,0)(0,4)(2,2)

Hawk–Dove: resource V = 4, cost of fighting C = 6. Two hawks fight: (V − C)/2 = −1; a hawk against a dove gets V = 4 and the dove 0; two doves share: V/2 = 2.

Evolutionary Dynamics

Replicator Dynamics Step by Step: Hawk–Dove

generationx (hawk share)fHfDf̄new x = x·fH/f̄
00.1006.504.804.970.131
1
2
3
4
5
6
7

x = 0.100: fH = 3 + (−1)·x + 4·(1 − x) = 6.50, fD = 3 + 0·x + 2·(1 − x) = 4.80. fH > f̄ → hawks increase.

generationx (hawk share)fHfDf̄new x = x·fH/f̄
00.1006.504.804.970.131
10.1316.354.744.950.168
2
3
4
5
6
7

x = 0.131: fH = 3 + (−1)·x + 4·(1 − x) = 6.35, fD = 3 + 0·x + 2·(1 − x) = 4.74. fH > f̄ → hawks increase.

generationx (hawk share)fHfDf̄new x = x·fH/f̄
00.1006.504.804.970.131
10.1316.354.744.950.168
20.1686.164.664.920.210
3
4
5
6
7

x = 0.168: fH = 3 + (−1)·x + 4·(1 − x) = 6.16, fD = 3 + 0·x + 2·(1 − x) = 4.66. fH > f̄ → hawks increase.

generationx (hawk share)fHfDf̄new x = x·fH/f̄
00.1006.504.804.970.131
10.1316.354.744.950.168
20.1686.164.664.920.210
30.2105.954.584.870.257
4
5
6
7

x = 0.210: fH = 3 + (−1)·x + 4·(1 − x) = 5.95, fD = 3 + 0·x + 2·(1 − x) = 4.58. fH > f̄ → hawks increase.

generationx (hawk share)fHfDf̄new x = x·fH/f̄
00.1006.504.804.970.131
10.1316.354.744.950.168
20.1686.164.664.920.210
30.2105.954.584.870.257
40.2575.724.494.800.306
5
6
7

x = 0.257: fH = 3 + (−1)·x + 4·(1 − x) = 5.72, fD = 3 + 0·x + 2·(1 − x) = 4.49. fH > f̄ → hawks increase.

generationx (hawk share)fHfDf̄new x = x·fH/f̄
00.1006.504.804.970.131
10.1316.354.744.950.168
20.1686.164.664.920.210
30.2105.954.584.870.257
40.2575.724.494.800.306
50.3065.474.394.720.354
6
7

x = 0.306: fH = 3 + (−1)·x + 4·(1 − x) = 5.47, fD = 3 + 0·x + 2·(1 − x) = 4.39. fH > f̄ → hawks increase.

generationx (hawk share)fHfDf̄new x = x·fH/f̄
00.1006.504.804.970.131
10.1316.354.744.950.168
20.1686.164.664.920.210
30.2105.954.584.870.257
40.2575.724.494.800.306
50.3065.474.394.720.354
60.3545.234.294.620.401
7

x = 0.354: fH = 3 + (−1)·x + 4·(1 − x) = 5.23, fD = 3 + 0·x + 2·(1 − x) = 4.29. fH > f̄ → hawks increase.

generationx (hawk share)fHfDf̄new x = x·fH/f̄
00.1006.504.804.970.131
10.1316.354.744.950.168
20.1686.164.664.920.210
30.2105.954.584.870.257
40.2575.724.494.800.306
50.3065.474.394.720.354
60.3545.234.294.620.401
70.4015.004.204.520.443

x = 0.401: fH = 3 + (−1)·x + 4·(1 − x) = 5.00, fD = 3 + 0·x + 2·(1 − x) = 4.20. fH > f̄ → hawks increase.

Evolutionary Dynamics

From Every Interior Start (0 < x₀ < 1) to the ESS: x* = 2/3

010203002/31generationhawk share xx₀ = 0.05x₀ = 0.40x₀ = 0.95ESS: V/C = 2/3
  • Rest point fH = fD: 7 − 5x = 5 − 2x ⇒ x* = 2/3, the same as the game's symmetric mixed Nash equilibrium.
  • In Rock–Paper–Scissors the replicator dynamics cycle around the equilibrium; in the Stag Hunt they reach one of the two pure equilibria, depending on the start.

Repeated Games

The Repeated Prisoner's Dilemma

Player 2Player 1C (cooperate)D (defect)C (cooperate)D (defect)(3,3)(0,5)(5,0)(1,1)

Points: temptation 5 > mutual cooperation 3 > mutual defection 1 > sucker 0. C: stay silent.

  • One-shot, or a known finite number of rounds (backward induction): defect every round.
  • If play continues with probability δ (the shadow of the future), grim trigger can sustain cooperation:

Comply: 3 every round ⇒ 3 + 3δ + … = 3/(1 − δ).

Deviate: 5 now, then 1 forever ⇒ 5 + δ/(1 − δ).

Comply ≥ deviate; times (1 − δ): 3 ≥ 5(1 − δ) + δ ⇔ δ ≥ (5 − 3)/(5 − 1) = 1/2.

δ = 0.9: comply 30, deviate 14 ⇒ cooperate. δ = 0.3: comply ≈ 4.29, deviate ≈ 5.43 ⇒ defect.

The punishment is credible: (D, D) is a stage-game equilibrium. Folk theorem: for large δ, any average payoffs above what defecting guarantees can be sustained.

Repeated Games

Grim Trigger: For Which δ Does Cooperation Last?

Comply: 3/(1 − δ)  ·  deviate: temptation + δ/(1 − δ)  ·  comply ≥ deviate ⇔ δ ≥ δ*  ·  δ* = (temptation − 3)/(temptation − 1)

The more the future is worth (the larger δ), the stronger the deterrent of the grim trigger; as temptation rises, so does the threshold δ* needed for cooperation.

Repeated Games

10-Round Matches: Who Plays What?

Tit-for-TatCCCCCCCCCC= 30Always CooperateCCCCCCCCCC= 3012345678910

TFT answers cooperation with cooperation: 30 points each. (Total: 30 – 30)

Tit-for-TatCDDDDDDDDD= 9Always DefectDDDDDDDDDD= 1412345678910

TFT is exploited in round 1, then always defects: 9 to 14. TFT can never win a match (at best it ties). (Total: 9 – 14)

Tit-for-TatCDCDCDCDCD= 25Suspicious TFTDCDCDCDCDC= 2512345678910

Suspicious TFT opens with defection; the two TFTs "echo" each other → alternating C/D, 25 each. (Total: 25 – 25)

Grim TriggerCDDDDDDDDD= 13Suspicious TFTDCDDDDDDDD= 1312345678910

Grim never forgives a single defection: from round 2 on, mutual defection. (Total: 13 – 13)

Always DefectDDDDDDDDDD= 50Always CooperateCCCCCCCCCC= 012345678910

Always Defect fully exploits the unconditional cooperator: 50 to 0. (Total: 50 – 0)

C: staying silent, i.e. cooperating (green); D: defecting (red). TFT: cooperate in round 1, then repeat the opponent's last move.

Repeated Games

Code Trace: match(tft, stft, 6)

TFT= 0Suspicious TFT= 0

Call match(tft, stft, 6): histories empty, scores 0.

TFT= 0Suspicious TFT= 0

t = 0: With empty histories, TFT opens with C and Suspicious TFT with D.

TFTC= 0Suspicious TFTD= 5

PAY["CD"] = (0, 5) → scores 0 – 5.

TFTC= 0Suspicious TFTD= 5

t = 1: Both copy the opponent's last move: TFT → D, STFT → C.

TFTCD= 5Suspicious TFTDC= 5

PAY["DC"] = (5, 0) → scores 5 – 5.

TFTCD= 5Suspicious TFTDC= 5

t = 2: Both copy the opponent's last move: TFT → C, STFT → D.

TFTCDC= 5Suspicious TFTDCD= 10

PAY["CD"] = (0, 5) → scores 5 – 10.

TFTCDC= 5Suspicious TFTDCD= 10

t = 3: Both copy the opponent's last move: TFT → D, STFT → C.

TFTCDCD= 10Suspicious TFTDCDC= 10

PAY["DC"] = (5, 0) → scores 10 – 10.

TFTCDCD= 10Suspicious TFTDCDC= 10

t = 4: Both copy the opponent's last move: TFT → C, STFT → D.

TFTCDCDC= 10Suspicious TFTDCDCD= 15

PAY["CD"] = (0, 5) → scores 10 – 15.

TFTCDCDC= 10Suspicious TFTDCDCD= 15

t = 5: Both copy the opponent's last move: TFT → D, STFT → C.

TFTCDCDCD= 15Suspicious TFTDCDCDC= 15

PAY["DC"] = (5, 0) → scores 15 – 15.

TFTCDCDCD= 15Suspicious TFTDCDCDC= 15

return (15, 15). A single opening defection starts an endless "echo": CD/DC alternating, 2.5 points per round on average (mutual cooperation would give 3).

PAY = {"CC": (3, 3), "CD": (0, 5),
       "DC": (5, 0), "DD": (1, 1)}
tft = lambda me, opp: opp[-1] if opp else "C"
stft = lambda me, opp: opp[-1] if opp else "D"
def match(s1, s2, T):
    h1, h2, p1, p2 = [], [], 0, 0
    for t in range(T):
        a, b = s1(h1, h2), s2(h2, h1)
        u, v = PAY[a + b]
        h1.append(a); h2.append(b)
        p1, p2 = p1 + u, p2 + v
    return p1, p2

Repeated Games

A Mini Axelrod Tournament: Step by Step

Strategy \ opponentAll-CAll-DTFTGrimSTFTTotalRank
Always Cooperate300303027117
Always Defect
Tit-for-Tat
Grim Trigger
Suspicious TFT

Always Cooperate plays 10 rounds against everyone (including its twin): total 117; it scores most against Always Cooperate (30).

Strategy \ opponentAll-CAll-DTFTGrimSTFTTotalRank
Always Cooperate300303027117
Always Defect501014141098
Tit-for-Tat
Grim Trigger
Suspicious TFT

Always Defect plays 10 rounds against everyone (including its twin): total 98; it scores most against Always Cooperate (50).

Strategy \ opponentAll-CAll-DTFTGrimSTFTTotalRank
Always Cooperate300303027117
Always Defect501014141098
Tit-for-Tat309303025124
Grim Trigger
Suspicious TFT

Tit-for-Tat plays 10 rounds against everyone (including its twin): total 124; it scores most against Always Cooperate (30).

Strategy \ opponentAll-CAll-DTFTGrimSTFTTotalRank
Always Cooperate300303027117
Always Defect501014141098
Tit-for-Tat309303025124
Grim Trigger309303013112
Suspicious TFT

Grim Trigger plays 10 rounds against everyone (including its twin): total 112; it scores most against Always Cooperate (30).

Strategy \ opponentAll-CAll-DTFTGrimSTFTTotalRank
Always Cooperate300303027117
Always Defect501014141098
Tit-for-Tat309303025124
Grim Trigger309303013112
Suspicious TFT321025131090

Suspicious TFT plays 10 rounds against everyone (including its twin): total 90; it scores most against Always Cooperate (32).

Strategy \ opponentAll-CAll-DTFTGrimSTFTTotalRank
Always Cooperate3003030271172
Always Defect5010141410984
Tit-for-Tat3093030251241
Grim Trigger3093030131123
Suspicious TFT3210251310905

Ranking: Tit-for-Tat (124), Always Cooperate (117), Grim Trigger (112), Always Defect (98), Suspicious TFT (90). Tit-for-Tat comes first without winning a single match! Always Defect is second to last.

Points scored in 10 rounds by the row strategy against the column opponent. All-C: Always Cooperate, All-D: Always Defect, STFT: Suspicious TFT.

Repeated Games

Axelrod's Tournaments (1980) and Their Lessons

Axelrod's two computer tournaments (14 and 62 programs) were both won by Rapoport's four-line Tit-for-Tat.

NiceNever be the first to defect.
RetaliatoryPunish defection.
ForgivingCooperate again when they do.
ClearBe easy to read.
  • With noise (mistaken moves), two TFTs retaliate endlessly; generous TFT sometimes forgives and does better.
  • Success depends on the pool of opponents; in an evolutionary (replicator) tournament, winners multiply.
TFT (A)CCCDerrorCDCDCD= 29TFT (B)CCCCDCDCDC= 24

Two TFTs, one error by A in round 4: an endless C/D echo. 29 – 24 over 10 rounds (30 – 30 without it).

Summary

Discussion: Learning, People and Ethics

Does learning always lead to the optimum?No. Some dynamics cycle (Matching Pennies, Shapley's example); the environment and opponents change; a wrong model leads to bad convergence. The reliable target is usually no regret.
Do people learn like algorithms?Similar: trial and error, learning from the past. Different: people act on risk perception, emotions and social norms; algorithms follow a defined reward.
Ethical risks of adaptive systemsData bias, recommendations that manipulate users (clickbait), privacy, algorithmic price coordination (tacit collusion among learning pricing bots).

Summary

Summary

MethodDescriptionOutcome
Best-response dynamicsBest response to the last actionEquilibrium in potential games; otherwise cycles
Fictitious playBest response to past frequenciesFrequencies reach equilibrium in zero-sum games
Hedgew ← w·eηgRT = O(√(T ln n)); time average → CCE
ε-greedy / UCBBalancing exploration and exploitationUCB: logarithmic regret
Q-learningQ ← Q + α(r + γ·max Q′ − Q)Reward propagates back from the goal
ReplicatorSuccessful strategies spreadHawk–Dove: x* = V/C
Repeated PDThe shadow of the futureCooperation if δ ≥ 1/2; TFT is strong

Summary

Sample Problems

  1. Hedge with η = ln 2, two actions, payoffs (1, 0) and (1, 0) in the first two rounds. What is p1 in round 3?
  2. UCB1 at t = 10: arm 1 has Q = 0.6, N = 5; arm 2 has Q = 0.2, N = 1. Which arm is chosen?
  3. In Hawk–Dove (V = 4, C = 6, baseline 3) with x = 0.5, does the hawk share increase in the next generation?
  4. With temptation 5, mutual cooperation 3 and mutual defection 1, what is the smallest δ for which Grim sustains cooperation? What if the temptation were 7?
  5. How does fictitious play differ from best-response dynamics? Which one "approaches" equilibrium in Matching Pennies?
  6. In Q-learning, Q(s, a) = 1, r = 0, γ = 0.9, max Q(s′, ·) = 2, α = 0.1. What is Q(s, a) after the update?
Show answers
  1. w = (e2 ln 2, 1) = (4, 1) ⇒ p1 = 4/5 = 0.8.
  2. UCB = 0.6 + √(2 ln 10 / 5) ≈ 1.56; 0.2 + √(2 ln 10) ≈ 2.35 ⇒ arm 2 (because it was rarely tried).
  3. fH = 4.50, fD = 4.00, f̄ = 4.25 ⇒ x′ = 0.529 > 0.5: it increases (toward 2/3).
  4. δ ≥ (5 − 3)/(5 − 1) = 1/2; with temptation 7, δ ≥ (7 − 3)/(7 − 1) = 2/3.
  5. Best response looks only at the last action and cycles; fictitious play looks at the frequencies of the whole history, and the frequencies approach (½, ½).
  6. Q ← 1 + 0.1·(0 + 0.9·2 − 1) = 1.08: the target 1.8 exceeds the old value, so Q increases.

Summary

Try It Yourself

Nashpy (Python library)Fictitious play (fictitious_play) and replicator dynamics (replicator_dynamics) on any payoff matrix.

Summary

References

  • D. Fudenberg, D. Levine, The Theory of Learning in Games, MIT Press, 1998.
  • N. Cesa-Bianchi, G. Lugosi, Prediction, Learning, and Games, Cambridge, 2006.
  • T. Lattimore, C. Szepesvári, Bandit Algorithms, Cambridge, 2020.
  • P. Auer, N. Cesa-Bianchi, P. Fischer, “Finite-time analysis of the multiarmed bandit problem”, Machine Learning 47, 2002.
  • C. Watkins, P. Dayan, “Q-learning”, Machine Learning 8, 1992; A. Blum, Y. Mansour, “From external to internal regret”, JMLR 8, 2007.
  • R. Axelrod, The Evolution of Cooperation, Basic Books, 1984.
  • J. Hofbauer, K. Sigmund, Evolutionary Games and Population Dynamics, Cambridge, 1998.
  • T. Roughgarden, Twenty Lectures on Algorithmic Game Theory, 2016 (Lectures 16–18); course page: CS364A (Stanford, 2013).
  • N. Nisan, T. Roughgarden, É. Tardos, V. Vazirani (eds.), Algorithmic Game Theory, Cambridge, 2007 (Chapter 4: learning, regret minimization and equilibria).
  • A. R. Karlin, Y. Peres, Game Theory, Alive, AMS, 2017 (Chapter 7: evolutionary equilibria; Chapter 18: adaptive decision making); D. Easley, J. Kleinberg, Networks, Crowds, and Markets, 2010 (Chapter 7: evolutionary game theory).
  • Tool: Nashpy (fictitious play, replicator dynamics).

Summary

References (2/2)

  • J. Robinson, “An iterative method of solving a game”, Annals of Mathematics 54(2), 1951; K. Miyasawa, “On the convergence of the learning process in a 2×2 non-zero-sum two-person game”, Princeton Research Memo 33, 1961; L. S. Shapley, “Some topics in two-person games”, Advances in Game Theory, 1964.
  • D. Monderer, L. S. Shapley, “Potential games”, Games and Economic Behavior 14(1), 1996.
  • D. Foster, R. Vohra, “Calibrated learning and correlated equilibrium”, Games and Economic Behavior 21, 1997; S. Hart, A. Mas-Colell, “A simple adaptive procedure leading to correlated equilibrium”, Econometrica 68(5), 2000.
  • J. Maynard Smith, G. R. Price, “The logic of animal conflict”, Nature 246, 1973.
  • W. R. Thompson, “On the likelihood that one unknown probability exceeds another in view of the evidence of two samples”, Biometrika 25, 1933.
  • P. Auer, N. Cesa-Bianchi, Y. Freund, R. E. Schapire, “The nonstochastic multiarmed bandit problem”, SIAM Journal on Computing 32(1), 2002.
  • N. D. Daw, J. P. O’Doherty, P. Dayan, B. Seymour, R. J. Dolan, “Cortical substrates for exploratory decisions in humans”, Nature 441, 2006.
  • C. Watkins, Learning from Delayed Rewards, PhD thesis, University of Cambridge, 1989; V. Mnih et al., “Human-level control through deep reinforcement learning”, Nature 518, 2015.
  • M. Zinkevich, M. Johanson, M. Bowling, C. Piccione, “Regret minimization in games with incomplete information”, NeurIPS, 2007.
  • M. Nowak, K. Sigmund, “A strategy of win-stay, lose-shift that outperforms tit-for-tat in the Prisoner’s Dilemma game”, Nature 364, 1993.

The End

Chapter 13: Learning and Adaptation
Previous: Chapter 12: Behavioral and Collective Decisions · Next: Chapter 14: Future Directions
1 / 1 Sercan KÜLCÜ. All rights reserved.