Chapter 13: Learning and Adaptation
Introduction
Warm-up: The Multi-Armed Bandit
Each arm pays 1 point with a hidden probability; pulling an arm shows only its own outcome, and the goal is to maximize total reward.
- Would you stick with Arm 1, which has the best average so far, or try Arm 3 again?
- Out of a hundred pulls, how many would you spend on trying arms (exploration)?
Introduction
Learning Outcomes
Introduction
Why Learning and Adaptation?
Learning: improving a strategy from past experience. Adaptation: updating the strategy as opponents and the environment change.
- Equilibria can be hard to compute: Nash equilibrium in two-player games is PPAD-complete (so is approximate equilibrium with many players), a class with no known fast algorithm for large games. How do players "find" an equilibrium?
- They may not know the game: the payoff matrix is unknown; they only play and see the outcome.
- The question: do simple, local update rules eventually lead to equilibrium, or at least to good performance?
Introduction
Learning Models: An Overview
| Model | What does the player observe? | How does it update? |
|---|---|---|
| Best-response dynamics | The opponent's last action | Plays a best response to it |
| Fictitious play | The opponent's whole history | Best response to the frequencies |
| Regret minimization (Hedge) | The payoff of every action | Multiplicatively boosts good actions |
| Multi-armed bandit (ε-greedy, UCB) | Only the chosen action's reward | Mean estimate + exploration |
| Reinforcement learning (Q-learning) | Reward and new state | Updates the Q(s, a) values |
| Replicator dynamics | Average success in the population | Successful strategies spread |
Top to bottom: the player stops watching the opponent and watches its own reward; in the last row the view shifts from the individual to the population.
Best Response and Fictitious Play
Best-Response Dynamics (Step by Step)
Starting profile (B, Z). Players take turns best-responding to the opponent's current action.
Row looks for a best response to Z: u1 = 1, 0, 3 → C. Profile (C, Z).
Column responds to C: u2 = 4, 0, 2 → X. Profile (C, X).
Row responds to X: u1 = 4, 2, 1 → A. Profile (A, X).
Column already picks X against A (3 > 2, 1), and Row does not leave A → the dynamics stop at the Nash equilibrium (A, X).
Starting from (C, Y) would lead to the equilibrium (B, Y): where you end up depends on where you start.
Best Response and Fictitious Play
When Does Best-Response Dynamics Converge?
Matching Pennies: (H,H) → (H,T) → (T,T) → (T,H) → (H,H) … an endless cycle
Best Response and Fictitious Play
Fictitious Play: Matching Pennies
| t | Mism. history H/T (earlier rounds) | Matcher | Match. history H/T (earlier rounds) | Mismatcher | Match. H rate (incl. this round) | Mism. H rate (incl. this round) |
|---|---|---|---|---|---|---|
| 1 | 0 / 0 | H | 0 / 0 | H | 1.00 | 1.00 |
| 2 | ||||||
| 3 | ||||||
| 4 | ||||||
| 5 | ||||||
| 6 | ||||||
| 7 | ||||||
| 8 | ||||||
| 9 | ||||||
| 10 |
Round 1 is arbitrary: both play H (no history yet).
| t | Mism. history H/T (earlier rounds) | Matcher | Match. history H/T (earlier rounds) | Mismatcher | Match. H rate (incl. this round) | Mism. H rate (incl. this round) |
|---|---|---|---|---|---|---|
| 1 | 0 / 0 | H | 0 / 0 | H | 1.00 | 1.00 |
| 2 | 1 / 0 | H | 1 / 0 | T | 1.00 | 0.50 |
| 3 | ||||||
| 4 | ||||||
| 5 | ||||||
| 6 | ||||||
| 7 | ||||||
| 8 | ||||||
| 9 | ||||||
| 10 |
The matcher sees 1 / 0 in the opponent's history → H (match the majority, ties go to H). The mismatcher sees 1 / 0 → T (opposite of the majority, ties go to H).
| t | Mism. history H/T (earlier rounds) | Matcher | Match. history H/T (earlier rounds) | Mismatcher | Match. H rate (incl. this round) | Mism. H rate (incl. this round) |
|---|---|---|---|---|---|---|
| 1 | 0 / 0 | H | 0 / 0 | H | 1.00 | 1.00 |
| 2 | 1 / 0 | H | 1 / 0 | T | 1.00 | 0.50 |
| 3 | 1 / 1 | H | 2 / 0 | T | 1.00 | 0.33 |
| 4 | ||||||
| 5 | ||||||
| 6 | ||||||
| 7 | ||||||
| 8 | ||||||
| 9 | ||||||
| 10 |
The matcher sees 1 / 1 in the opponent's history → H (match the majority, ties go to H). The mismatcher sees 2 / 0 → T (opposite of the majority, ties go to H).
| t | Mism. history H/T (earlier rounds) | Matcher | Match. history H/T (earlier rounds) | Mismatcher | Match. H rate (incl. this round) | Mism. H rate (incl. this round) |
|---|---|---|---|---|---|---|
| 1 | 0 / 0 | H | 0 / 0 | H | 1.00 | 1.00 |
| 2 | 1 / 0 | H | 1 / 0 | T | 1.00 | 0.50 |
| 3 | 1 / 1 | H | 2 / 0 | T | 1.00 | 0.33 |
| 4 | 1 / 2 | T | 3 / 0 | T | 0.75 | 0.25 |
| 5 | ||||||
| 6 | ||||||
| 7 | ||||||
| 8 | ||||||
| 9 | ||||||
| 10 |
The matcher sees 1 / 2 in the opponent's history → T (match the majority, ties go to H). The mismatcher sees 3 / 0 → T (opposite of the majority, ties go to H).
| t | Mism. history H/T (earlier rounds) | Matcher | Match. history H/T (earlier rounds) | Mismatcher | Match. H rate (incl. this round) | Mism. H rate (incl. this round) |
|---|---|---|---|---|---|---|
| 1 | 0 / 0 | H | 0 / 0 | H | 1.00 | 1.00 |
| 2 | 1 / 0 | H | 1 / 0 | T | 1.00 | 0.50 |
| 3 | 1 / 1 | H | 2 / 0 | T | 1.00 | 0.33 |
| 4 | 1 / 2 | T | 3 / 0 | T | 0.75 | 0.25 |
| 5 | 1 / 3 | T | 3 / 1 | T | 0.60 | 0.20 |
| 6 | ||||||
| 7 | ||||||
| 8 | ||||||
| 9 | ||||||
| 10 |
The matcher sees 1 / 3 in the opponent's history → T (match the majority, ties go to H). The mismatcher sees 3 / 1 → T (opposite of the majority, ties go to H).
| t | Mism. history H/T (earlier rounds) | Matcher | Match. history H/T (earlier rounds) | Mismatcher | Match. H rate (incl. this round) | Mism. H rate (incl. this round) |
|---|---|---|---|---|---|---|
| 1 | 0 / 0 | H | 0 / 0 | H | 1.00 | 1.00 |
| 2 | 1 / 0 | H | 1 / 0 | T | 1.00 | 0.50 |
| 3 | 1 / 1 | H | 2 / 0 | T | 1.00 | 0.33 |
| 4 | 1 / 2 | T | 3 / 0 | T | 0.75 | 0.25 |
| 5 | 1 / 3 | T | 3 / 1 | T | 0.60 | 0.20 |
| 6 | 1 / 4 | T | 3 / 2 | T | 0.50 | 0.17 |
| 7 | ||||||
| 8 | ||||||
| 9 | ||||||
| 10 |
The matcher sees 1 / 4 in the opponent's history → T (match the majority, ties go to H). The mismatcher sees 3 / 2 → T (opposite of the majority, ties go to H).
| t | Mism. history H/T (earlier rounds) | Matcher | Match. history H/T (earlier rounds) | Mismatcher | Match. H rate (incl. this round) | Mism. H rate (incl. this round) |
|---|---|---|---|---|---|---|
| 1 | 0 / 0 | H | 0 / 0 | H | 1.00 | 1.00 |
| 2 | 1 / 0 | H | 1 / 0 | T | 1.00 | 0.50 |
| 3 | 1 / 1 | H | 2 / 0 | T | 1.00 | 0.33 |
| 4 | 1 / 2 | T | 3 / 0 | T | 0.75 | 0.25 |
| 5 | 1 / 3 | T | 3 / 1 | T | 0.60 | 0.20 |
| 6 | 1 / 4 | T | 3 / 2 | T | 0.50 | 0.17 |
| 7 | 1 / 5 | T | 3 / 3 | H | 0.43 | 0.29 |
| 8 | ||||||
| 9 | ||||||
| 10 |
The matcher sees 1 / 5 in the opponent's history → T (match the majority, ties go to H). The mismatcher sees 3 / 3 → H (opposite of the majority, ties go to H).
| t | Mism. history H/T (earlier rounds) | Matcher | Match. history H/T (earlier rounds) | Mismatcher | Match. H rate (incl. this round) | Mism. H rate (incl. this round) |
|---|---|---|---|---|---|---|
| 1 | 0 / 0 | H | 0 / 0 | H | 1.00 | 1.00 |
| 2 | 1 / 0 | H | 1 / 0 | T | 1.00 | 0.50 |
| 3 | 1 / 1 | H | 2 / 0 | T | 1.00 | 0.33 |
| 4 | 1 / 2 | T | 3 / 0 | T | 0.75 | 0.25 |
| 5 | 1 / 3 | T | 3 / 1 | T | 0.60 | 0.20 |
| 6 | 1 / 4 | T | 3 / 2 | T | 0.50 | 0.17 |
| 7 | 1 / 5 | T | 3 / 3 | H | 0.43 | 0.29 |
| 8 | 2 / 5 | T | 3 / 4 | H | 0.38 | 0.38 |
| 9 | ||||||
| 10 |
The matcher sees 2 / 5 in the opponent's history → T (match the majority, ties go to H). The mismatcher sees 3 / 4 → H (opposite of the majority, ties go to H).
| t | Mism. history H/T (earlier rounds) | Matcher | Match. history H/T (earlier rounds) | Mismatcher | Match. H rate (incl. this round) | Mism. H rate (incl. this round) |
|---|---|---|---|---|---|---|
| 1 | 0 / 0 | H | 0 / 0 | H | 1.00 | 1.00 |
| 2 | 1 / 0 | H | 1 / 0 | T | 1.00 | 0.50 |
| 3 | 1 / 1 | H | 2 / 0 | T | 1.00 | 0.33 |
| 4 | 1 / 2 | T | 3 / 0 | T | 0.75 | 0.25 |
| 5 | 1 / 3 | T | 3 / 1 | T | 0.60 | 0.20 |
| 6 | 1 / 4 | T | 3 / 2 | T | 0.50 | 0.17 |
| 7 | 1 / 5 | T | 3 / 3 | H | 0.43 | 0.29 |
| 8 | 2 / 5 | T | 3 / 4 | H | 0.38 | 0.38 |
| 9 | 3 / 5 | T | 3 / 5 | H | 0.33 | 0.44 |
| 10 |
The matcher sees 3 / 5 in the opponent's history → T (match the majority, ties go to H). The mismatcher sees 3 / 5 → H (opposite of the majority, ties go to H).
| t | Mism. history H/T (earlier rounds) | Matcher | Match. history H/T (earlier rounds) | Mismatcher | Match. H rate (incl. this round) | Mism. H rate (incl. this round) |
|---|---|---|---|---|---|---|
| 1 | 0 / 0 | H | 0 / 0 | H | 1.00 | 1.00 |
| 2 | 1 / 0 | H | 1 / 0 | T | 1.00 | 0.50 |
| 3 | 1 / 1 | H | 2 / 0 | T | 1.00 | 0.33 |
| 4 | 1 / 2 | T | 3 / 0 | T | 0.75 | 0.25 |
| 5 | 1 / 3 | T | 3 / 1 | T | 0.60 | 0.20 |
| 6 | 1 / 4 | T | 3 / 2 | T | 0.50 | 0.17 |
| 7 | 1 / 5 | T | 3 / 3 | H | 0.43 | 0.29 |
| 8 | 2 / 5 | T | 3 / 4 | H | 0.38 | 0.38 |
| 9 | 3 / 5 | T | 3 / 5 | H | 0.33 | 0.44 |
| 10 | 4 / 5 | T | 3 / 6 | H | 0.30 | 0.50 |
The matcher sees 4 / 5 in the opponent's history → T (match the majority, ties go to H). The mismatcher sees 3 / 6 → H (opposite of the majority, ties go to H).
Best Response and Fictitious Play
Code Trace: Fictitious Play (Matching Pennies, 6 rounds)
Call fictitious(6). Counters are zero; with no history, both play H in round 1.
Round 1: (H, H) is played and the matcher wins. Counters: matcher H/T = 1/0, mismatcher H/T = 1/0.
Next round: the mismatcher mostly played H → matcher H; the matcher mostly played H → mismatcher T.
Round 2: (H, T) is played and the mismatcher wins. Counters: matcher H/T = 2/0, mismatcher H/T = 1/1.
Next round: the mismatcher's counters are tied (ties go to H) → matcher H; the matcher mostly played H → mismatcher T.
Round 3: (H, T) is played and the mismatcher wins. Counters: matcher H/T = 3/0, mismatcher H/T = 1/2.
Next round: the mismatcher mostly played T → matcher T; the matcher mostly played H → mismatcher T.
Round 4: (T, T) is played and the matcher wins. Counters: matcher H/T = 3/1, mismatcher H/T = 1/3.
Next round: the mismatcher mostly played T → matcher T; the matcher mostly played H → mismatcher T.
Round 5: (T, T) is played and the matcher wins. Counters: matcher H/T = 3/2, mismatcher H/T = 1/4.
Next round: the mismatcher mostly played T → matcher T; the matcher mostly played H → mismatcher T.
Round 6: (T, T) is played and the matcher wins. Counters: matcher H/T = 3/3, mismatcher H/T = 1/5.
return nm = [3, 3], nx = [1, 5]. The actions cycle (HH → HT → TT → TH …); as the cycles lengthen, the frequencies approach ½ (0.50 and 0.17 after 6 rounds; 0.472 and 0.492 after 400).
def fictitious(T): nm, nx = [0, 0], [0, 0] # H/T counters a, b = 0, 0 # round 1: both H for t in range(1, T + 1): nm[a] += 1 # matcher's history nx[b] += 1 # mismatcher's history # matcher: match the mismatcher's majority a = 0 if nx[0] >= nx[1] else 1 # mismatcher: avoid the matcher's majority b = 1 if nm[0] > nm[1] else 0 return nm, nx
Best Response and Fictitious Play
Fictitious Play: Frequencies Converge to Equilibrium
- Actions change in ever-longer cycles, but the frequencies approach the mixed equilibrium (½, ½): 0.472 and 0.492 after 400 rounds.
- Guaranteed convergence: two-player zero-sum games (Robinson 1951), 2×2 games, potential games. Shapley (1964): in a 3×3 game the frequencies cycle and never converge.
Regret and Hedge
External Regret: Why Does "Follow the Leader" Fail?
Regret = (the total you would have earned by always playing the best single action in hindsight) − (the total you actually earned). External regret RT = maxi Σt gt(i) − Σt gt(at) (using expected payoff for randomized choices): the gap to the best fixed action in hindsight. If RT/T → 0, the algorithm has no regret.
| t | Totals so far A / B | g(A) | g(B) | Follow the leader | Payoff |
|---|---|---|---|---|---|
| 1 | 0.0 / 0.0 | 0.5 | 0.0 | A | 0.5 |
| 2 | |||||
| 3 | |||||
| 4 | |||||
| 5 | |||||
| 6 |
Leader A (totals 0.0 / 0.0) → A is chosen; round payoff 0.5. Algorithm's total: 0.5.
| t | Totals so far A / B | g(A) | g(B) | Follow the leader | Payoff |
|---|---|---|---|---|---|
| 1 | 0.0 / 0.0 | 0.5 | 0.0 | A | 0.5 |
| 2 | 0.5 / 0.0 | 0.0 | 1.0 | A | 0.0 |
| 3 | |||||
| 4 | |||||
| 5 | |||||
| 6 |
Leader A (totals 0.5 / 0.0) → A is chosen; round payoff 0.0. Algorithm's total: 0.5.
| t | Totals so far A / B | g(A) | g(B) | Follow the leader | Payoff |
|---|---|---|---|---|---|
| 1 | 0.0 / 0.0 | 0.5 | 0.0 | A | 0.5 |
| 2 | 0.5 / 0.0 | 0.0 | 1.0 | A | 0.0 |
| 3 | 0.5 / 1.0 | 1.0 | 0.0 | B | 0.0 |
| 4 | |||||
| 5 | |||||
| 6 |
Leader B (totals 0.5 / 1.0) → B is chosen; round payoff 0.0. Algorithm's total: 0.5.
| t | Totals so far A / B | g(A) | g(B) | Follow the leader | Payoff |
|---|---|---|---|---|---|
| 1 | 0.0 / 0.0 | 0.5 | 0.0 | A | 0.5 |
| 2 | 0.5 / 0.0 | 0.0 | 1.0 | A | 0.0 |
| 3 | 0.5 / 1.0 | 1.0 | 0.0 | B | 0.0 |
| 4 | 1.5 / 1.0 | 0.0 | 1.0 | A | 0.0 |
| 5 | |||||
| 6 |
Leader A (totals 1.5 / 1.0) → A is chosen; round payoff 0.0. Algorithm's total: 0.5.
| t | Totals so far A / B | g(A) | g(B) | Follow the leader | Payoff |
|---|---|---|---|---|---|
| 1 | 0.0 / 0.0 | 0.5 | 0.0 | A | 0.5 |
| 2 | 0.5 / 0.0 | 0.0 | 1.0 | A | 0.0 |
| 3 | 0.5 / 1.0 | 1.0 | 0.0 | B | 0.0 |
| 4 | 1.5 / 1.0 | 0.0 | 1.0 | A | 0.0 |
| 5 | 1.5 / 2.0 | 1.0 | 0.0 | B | 0.0 |
| 6 |
Leader B (totals 1.5 / 2.0) → B is chosen; round payoff 0.0. Algorithm's total: 0.5.
| t | Totals so far A / B | g(A) | g(B) | Follow the leader | Payoff |
|---|---|---|---|---|---|
| 1 | 0.0 / 0.0 | 0.5 | 0.0 | A | 0.5 |
| 2 | 0.5 / 0.0 | 0.0 | 1.0 | A | 0.0 |
| 3 | 0.5 / 1.0 | 1.0 | 0.0 | B | 0.0 |
| 4 | 1.5 / 1.0 | 0.0 | 1.0 | A | 0.0 |
| 5 | 1.5 / 2.0 | 1.0 | 0.0 | B | 0.0 |
| 6 | 2.5 / 2.0 | 0.0 | 1.0 | A | 0.0 |
Leader A (totals 2.5 / 2.0) → A is chosen; round payoff 0.0. Algorithm's total: 0.5. Best fixed action B: 3. Regret = 3 − 0.5 = 2.5 ≈ T/2.
Regret and Hedge
Multiplicative Weights / Hedge
- Give each action a weight: wi = 1.
- Each round, play with probabilities pi = wi / Σj wj.
- Observe every action's payoff and update:
wi ← wi · eη·gt(i)An action with payoff 1 has its weight multiplied by eη (exactly doubled if η = ln 2).
Regret and Hedge
Code Trace: Multiplicative Weights
Three experts (actions) A, B, C; η = 0.5. Initial weights are equal: w = [1, 1, 1].
Round 1: W = 3.00, probabilities p = [0.33, 0.33, 0.33].
Payoffs revealed: g = [1, 0, 0.5]. Expected payoff Σ p·g = 0.50 → total = 0.50.
Update w ← w·e0.5·g: w = [1.65, 1.00, 1.28]. Experts that did well this round gain weight.
Round 2: W = 3.93, probabilities p = [0.42, 0.25, 0.33].
Payoffs revealed: g = [1, 0, 0.5]. Expected payoff Σ p·g = 0.58 → total = 1.08.
Update w ← w·e0.5·g: w = [2.72, 1.00, 1.65]. Experts that did well this round gain weight.
Round 3: W = 5.37, probabilities p = [0.51, 0.19, 0.31].
Payoffs revealed: g = [0, 1, 0.5]. Expected payoff Σ p·g = 0.34 → total = 1.42.
Update w ← w·e0.5·g: w = [2.72, 1.65, 2.12]. Experts that did well this round gain weight.
Round 4: W = 6.48, probabilities p = [0.42, 0.25, 0.33].
Payoffs revealed: g = [0, 1, 0.5]. Expected payoff Σ p·g = 0.42 → total = 1.84.
Update w ← w·e0.5·g: w = [2.72, 2.72, 2.72]. Experts that did well this round gain weight.
Round 5: W = 8.15, probabilities p = [0.33, 0.33, 0.33].
Payoffs revealed: g = [0, 1, 0.5]. Expected payoff Σ p·g = 0.50 → total = 2.34.
Update w ← w·e0.5·g: w = [2.72, 4.48, 3.49]. Experts that did well this round gain weight.
return 2.34. Best fixed expert B: 3. Regret = 3 − 2.34 = 0.66 (follow the leader can do worse in 5 rounds).
import math def hedge(G, eta): w = [1.0] * len(G[0]) total = 0.0 for g in G: # each round W = sum(w) p = [x / W for x in w] total += sum(pi * gi for pi, gi in zip(p, g)) w = [x*math.exp(eta*gi) for x, gi in zip(w, g)] return total
Regret and Hedge
Change the Assumption: What If the Opponent Learns Too?
Fixed opponent: Hedge goes to the best response. Expected payoffs: Rock 0.45, Paper 0.65, Scissors 0.40. Hedge's probability of Paper is 0.82 at round 100 and 0.997 at round 300.
Learning opponent: the target moves. Both update at once; player 1 starts at (0.5, 0.3, 0.2), player 2 uniform.
| round | 100 | 200 | 800 | 1000 |
|---|---|---|---|---|
| P(Rock) in that round | 0.19 | 0.46 | 0.14 | 0.19 |
| average P(Rock) | 0.35 | 0.31 | 0.33 | 0.34 |
Fixed opponent: Hedge goes to the best response. Expected payoffs: Rock 0.45, Paper 0.65, Scissors 0.40. Hedge's probability of Paper is 0.82 at round 100 and 0.997 at round 300.
Learning opponent: the target moves. Both update at once; player 1 starts at (0.5, 0.3, 0.2), player 2 uniform.
| round | 100 | 200 | 800 | 1000 |
|---|---|---|---|---|
| P(Rock) in that round | 0.19 | 0.46 | 0.14 | 0.19 |
| average P(Rock) | 0.35 | 0.31 | 0.33 | 0.34 |
Fixed opponent: Hedge goes to the best response. Expected payoffs: Rock 0.45, Paper 0.65, Scissors 0.40. Hedge's probability of Paper is 0.82 at round 100 and 0.997 at round 300.
Learning opponent: the target moves. Both update at once; player 1 starts at (0.5, 0.3, 0.2), player 2 uniform.
| round | 100 | 200 | 800 | 1000 |
|---|---|---|---|---|
| P(Rock) in that round | 0.19 | 0.46 | 0.14 | 0.19 |
| average P(Rock) | 0.35 | 0.31 | 0.33 | 0.34 |
Regret and Hedge
Hedge vs Itself: Matching Pennies
- Both players use Hedge (η = 0.1). Initial weights: matcher (H, T) = (0.8, 0.2), so P(H) = 0.8; mismatcher (0.5, 0.5). A win pays 1, a loss 0; each round both players update simultaneously with the expected payoffs of their actions.
- The probability in each round (the last iterate) never settles at equilibrium; it swings widely around it. The time average approaches the minimax strategy (½, ½) (0.498 after 600 rounds). This is the basis of the learning proof of the minimax theorem.
Regret and Hedge
Which Equilibrium Does No-Regret Learning Reach?
- A mediator tells everyone what to play (like a traffic light).
- Correlated equilibrium (CE): following your recommendation is best even after seeing it. With algorithms that minimize a stronger kind of regret (internal regret), the time average approaches the set of CE.
- Coarse correlated equilibrium (CCE) is weaker: committing to "I'll always play this" before seeing the recommendation does not pay. If everyone plays no-regret, the time average approaches a CCE.
- CE and CCE are easy to compute; mixed Nash equilibrium is PPAD-complete.
Multi-Armed Bandits
The Multi-Armed Bandit
- A K-armed slot machine: arm i pays a random reward with (unknown) mean μi.
- Each round one arm is pulled and only its reward is seen (unlike Hedge: partial feedback).
- Regret: RT = T·μ* − Σt μat, where μ* is the best arm's mean.
UCB1, μ = (0.3, 0.5, 0.7), 40 rounds: a rarely tried arm has a wide interval, so it gets tried again.
Applications: ad and news recommendation, A/B testing, clinical trials, price experimentation.
Multi-Armed Bandits
Code Trace: ε-Greedy (ε = 0.2)
Each round ε-greedy pulls a random arm with probability ε (explore), otherwise the arm with the largest Q (exploit). Call: egreedy([0.3, 0.5, 0.7], 8, 0.2, random.Random(11)). Estimates Q = 0, counts N = 0. The code numbers arms from 0: arms 0, 1, 2 = Arms 1, 2, 3.
t = 1: 0.452 ≥ 0.2 → exploit: the arm with the largest Q is 0 (ties go to the first index).
Reward r = 0; N[0] = 1, Q[0] = 0.00 (incremental mean).
t = 2: 0.924 ≥ 0.2 → exploit: the arm with the largest Q is 0 (ties go to the first index).
Reward r = 0; N[0] = 2, Q[0] = 0.00 (incremental mean).
t = 3: 0.508 ≥ 0.2 → exploit: the arm with the largest Q is 0 (ties go to the first index).
Reward r = 0; N[0] = 3, Q[0] = 0.00 (incremental mean).
t = 4: random number 0.185 < 0.2 → explore, random arm 2.
Reward r = 1; N[2] = 1, Q[2] = 1.00 (incremental mean).
t = 5: 0.614 ≥ 0.2 → exploit: the arm with the largest Q is 2.
Reward r = 1; N[2] = 2, Q[2] = 1.00 (incremental mean).
t = 6: 0.447 ≥ 0.2 → exploit: the arm with the largest Q is 2.
Reward r = 1; N[2] = 3, Q[2] = 1.00 (incremental mean).
t = 7: 0.539 ≥ 0.2 → exploit: the arm with the largest Q is 2.
Reward r = 0; N[2] = 4, Q[2] = 0.75 (incremental mean).
t = 8: 0.634 ≥ 0.2 → exploit: the arm with the largest Q is 2.
Reward r = 1; N[2] = 5, Q[2] = 0.80 (incremental mean).
return N = [3, 0, 5]. The first 3 rounds got stuck on arm 0 (all rewards 0, ties go to the first arm); a single exploration step found the best arm (2).
def egreedy(mu, T, eps, rng): N, Q = [0] * len(mu), [0.0] * len(mu) for t in range(1, T + 1): if rng.random() < eps: # explore a = rng.randrange(len(mu)) else: # exploit a = Q.index(max(Q)) r = 1 if rng.random() < mu[a] else 0 N[a] += 1 Q[a] += (r - Q[a]) / N[a] return N, Q
Multi-Armed Bandits
Code Trace: UCB1 (Upper Confidence Bound)
Call: ucb([0.3, 0.5, 0.7], 8, random.Random(3)). Estimates and counts start at zero. Score Q + √(2 ln t / N): t is the current round, N the arm's pull count. Rarely pulled arms get a large bonus.
t = 1: an untried arm has bonus ∞ → the first untried arm 0 is chosen (each arm once).
Reward r = 1; Q[0] = 1.00, N[0] = 1.
t = 2: an untried arm has bonus ∞ → the first untried arm 1 is chosen (each arm once).
Reward r = 0; Q[1] = 0.00, N[1] = 1.
t = 3: an untried arm has bonus ∞ → the first untried arm 2 is chosen (each arm once).
Reward r = 1; Q[2] = 1.00, N[2] = 1.
t = 4: bonus √(2 ln t / N) = [1.67, 1.67, 1.67]; UCB = Q + b = [2.67, 1.67, 2.67] → largest: arm 0.
Reward r = 0; Q[0] = 0.50, N[0] = 2.
t = 5: bonus √(2 ln t / N) = [1.27, 1.79, 1.79]; UCB = Q + b = [1.77, 1.79, 2.79] → largest: arm 2.
Reward r = 1; Q[2] = 1.00, N[2] = 2.
t = 6: bonus √(2 ln t / N) = [1.34, 1.89, 1.34]; UCB = Q + b = [1.84, 1.89, 2.34] → largest: arm 2.
Reward r = 1; Q[2] = 1.00, N[2] = 3.
t = 7: bonus √(2 ln t / N) = [1.39, 1.97, 1.14]; UCB = Q + b = [1.89, 1.97, 2.14] → largest: arm 2.
Reward r = 1; Q[2] = 1.00, N[2] = 4.
t = 8: bonus √(2 ln t / N) = [1.44, 2.04, 1.02]; UCB = Q + b = [1.94, 2.04, 2.02] → largest: arm 1.
Reward r = 0; Q[1] = 0.00, N[1] = 2.
return N = [2, 2, 4]. A rarely tried arm keeps a large bonus: at t = 8, arm 1 (one try, Q = 0) was tried again — exploration comes for free.
import math def ucb(mu, T, rng): K = len(mu) N, Q = [0] * K, [0.0] * K for t in range(1, T + 1): b = [math.inf if n == 0 else math.sqrt(2*math.log(t)/n) for n in N] a = max(range(K), key=lambda i: Q[i] + b[i]) r = 1 if rng.random() < mu[a] else 0 N[a] += 1 Q[a] += (r - Q[a]) / N[a] return N, Q
Multi-Armed Bandits
Comparison: 5000 Rounds, 100 Runs
- Arms μ = (0.3, 0.5, 0.7), seeds 0–99. Pure exploitation gets stuck on arm 0 every time: regret 0.4·T, linear.
- ε-greedy keeps exploring 10% of the time, so its regret grows linearly. UCB1's regret is above ε-greedy's from roughly round 270 to round 2,370 and stays below afterwards, because UCB1's regret is logarithmic, i.e. it grows very slowly (Auer et al., 2002).
- ε is a tuning parameter: 0.2 in the short trace so that exploration shows up, 0.1 in the long comparison, 0.15 in the bandit game. A smaller ε means less exploration and later correction.
Multi-Armed Bandits
Back to the Game: The Multi-Armed Bandit
Arms μ = (0.3, 0.5, 0.7), 100 rounds:
| Strategy | Reward | Regret |
|---|---|---|
| always random (explore only) | 50 | 20 |
| stick with the 0.5 arm | 50 | 20 |
| ε-greedy, ε = 0.15 (best found) | 67 | 3 |
| always the best arm | 70 | 0 |
ε-greedy explores 15% of rounds forever; UCB1 explores rarely tried arms, with logarithmic regret.
One failed pull of Arm 3 says little. At t = 9 UCB1 picks Arm 3: 0 + √(2 ln 9 / 1) ≈ 2.10 vs. 0.67 + √(2 ln 9 / 3) ≈ 1.88 for Arm 1.
In practice: in a four-armed bandit task, human choices fit softmax on value differences better than a UCB-like uncertainty bonus (Daw et al., 2006).
Reinforcement Learning
From Bandits to Reinforcement Learning
- A bandit is a one-state problem. In reinforcement learning an action determines not only the reward but also the next state (where the agent is, e.g. the position on a board).
- Q-learning (Watkins, 1989): updates the state–action value from an observed transition
α: the learning rate (how much to trust new information); γ: the discount factor for future reward.
Reinforcement Learning
Code Trace: Q-Learning (α = 0.5, γ = 0.9)
Call q_learn(ep, 0.5, 0.9): 3 episodes starting at s0. All Q values are 0; s3 is the goal (+1).
Episode 1: s0 → right → s1, r = 0. best = max Q(s1, ·) = 0.000; target = 0.000; Q(s0,right) = 0.000 + 0.5·(0.000 − 0.000) = 0.000.
Episode 1: s1 → right → s2, r = 0. best = max Q(s2, ·) = 0.000; target = 0.000; Q(s1,right) = 0.000 + 0.5·(0.000 − 0.000) = 0.000.
Episode 1: s2 → right → s3, r = 1. s3 is the goal → best = 0; target = 1.000; Q(s2,right) = 0.000 + 0.5·(1.000 − 0.000) = 0.500.
Episode 2: s0 → right → s1, r = 0. best = max Q(s1, ·) = 0.000; target = 0.000; Q(s0,right) = 0.000 + 0.5·(0.000 − 0.000) = 0.000.
Episode 2: s1 → right → s2, r = 0. best = max Q(s2, ·) = 0.500; target = 0.450; Q(s1,right) = 0.000 + 0.5·(0.450 − 0.000) = 0.225.
Episode 2: s2 → right → s3, r = 1. s3 is the goal → best = 0; target = 1.000; Q(s2,right) = 0.500 + 0.5·(1.000 − 0.500) = 0.750.
Episode 3: s0 → right → s1, r = 0. best = max Q(s1, ·) = 0.225; target = 0.203; Q(s0,right) = 0.000 + 0.5·(0.203 − 0.000) = 0.101.
Episode 3: s1 → left → s0, r = 0. best = max Q(s0, ·) = 0.101; target = 0.091; Q(s1,left) = 0.000 + 0.5·(0.091 − 0.000) = 0.046.
Episode 3: s0 → right → s1, r = 0. best = max Q(s1, ·) = 0.225; target = 0.203; Q(s0,right) = 0.101 + 0.5·(0.203 − 0.101) = 0.152.
Episode 3: s1 → right → s2, r = 0. best = max Q(s2, ·) = 0.750; target = 0.675; Q(s1,right) = 0.225 + 0.5·(0.675 − 0.225) = 0.450.
Episode 3: s2 → right → s3, r = 1. s3 is the goal → best = 0; target = 1.000; Q(s2,right) = 0.750 + 0.5·(1.000 − 0.750) = 0.875.
return Q. Reward propagated back from the goal: Q(s2,right) = 0.875 → Q(s1,right) = 0.450 → Q(s0,right) = 0.152. In every cell Q(right) > Q(left): the greedy policy is "always go right".
def q_learn(episodes, alpha, gamma): Q = [[0.0, 0.0] for s in range(4)] # [left, right] for ep in episodes: for s, a, r, s2 in ep: best = 0 if s2 == 3 else max(Q[s2]) target = r + gamma * best Q[s][a] += alpha * (target - Q[s][a]) return Q
Reinforcement Learning
Beyond Q-Learning: Methods and Milestones
- Thompson sampling (1933): keeps a distribution over each arm's possible mean (the posterior), draws a sample from it each round and pulls the arm with the largest sample.
- Win-Stay, Lose-Shift (WSLS; Nowak–Sigmund, 1993): after a 3 or 5 repeat the move, after a 0 or 1 switch; in the Prisoner's Dilemma it recovers cooperation two rounds after a single error.
- Others: EXP3 (Hedge that sees only the pulled arm's reward; its regret grows slower than T even if an adversary picks the rewards), SARSA and DQN (a neural network instead of a Q table), CFR (a regret-based method for poker).
| Year | System | Method |
|---|---|---|
| 1992 | TD-Gammon (backgammon) | self-play |
| 2017–19 | Libratus, Pluribus (poker) | CFR (regret) |
| 2018 | AlphaZero (chess, Go) | self-play |
Evolutionary Dynamics
Replicator Dynamics
- In a large population, individuals are matched at random; each plays a single strategy and passes it on to its offspring. No rationality, just selection.
- fi: the average payoff of strategy i (its fitness); f̄: the population's average payoff. Strategies that earn above average spread.
- The discrete form is enough for this course: new share = old share × fi / f̄ (a baseline of 3 is added to every payoff to keep fitness positive).
- Evolutionarily stable strategy (ESS): a small group of mutants cannot invade (Maynard Smith–Price, 1973).
Hawk–Dove: resource V = 4, cost of fighting C = 6. Two hawks fight: (V − C)/2 = −1; a hawk against a dove gets V = 4 and the dove 0; two doves share: V/2 = 2.
Evolutionary Dynamics
Replicator Dynamics Step by Step: Hawk–Dove
| generation | x (hawk share) | fH | fD | f̄ | new x = x·fH/f̄ |
|---|---|---|---|---|---|
| 0 | 0.100 | 6.50 | 4.80 | 4.97 | 0.131 |
| 1 | |||||
| 2 | |||||
| 3 | |||||
| 4 | |||||
| 5 | |||||
| 6 | |||||
| 7 |
x = 0.100: fH = 3 + (−1)·x + 4·(1 − x) = 6.50, fD = 3 + 0·x + 2·(1 − x) = 4.80. fH > f̄ → hawks increase.
| generation | x (hawk share) | fH | fD | f̄ | new x = x·fH/f̄ |
|---|---|---|---|---|---|
| 0 | 0.100 | 6.50 | 4.80 | 4.97 | 0.131 |
| 1 | 0.131 | 6.35 | 4.74 | 4.95 | 0.168 |
| 2 | |||||
| 3 | |||||
| 4 | |||||
| 5 | |||||
| 6 | |||||
| 7 |
x = 0.131: fH = 3 + (−1)·x + 4·(1 − x) = 6.35, fD = 3 + 0·x + 2·(1 − x) = 4.74. fH > f̄ → hawks increase.
| generation | x (hawk share) | fH | fD | f̄ | new x = x·fH/f̄ |
|---|---|---|---|---|---|
| 0 | 0.100 | 6.50 | 4.80 | 4.97 | 0.131 |
| 1 | 0.131 | 6.35 | 4.74 | 4.95 | 0.168 |
| 2 | 0.168 | 6.16 | 4.66 | 4.92 | 0.210 |
| 3 | |||||
| 4 | |||||
| 5 | |||||
| 6 | |||||
| 7 |
x = 0.168: fH = 3 + (−1)·x + 4·(1 − x) = 6.16, fD = 3 + 0·x + 2·(1 − x) = 4.66. fH > f̄ → hawks increase.
| generation | x (hawk share) | fH | fD | f̄ | new x = x·fH/f̄ |
|---|---|---|---|---|---|
| 0 | 0.100 | 6.50 | 4.80 | 4.97 | 0.131 |
| 1 | 0.131 | 6.35 | 4.74 | 4.95 | 0.168 |
| 2 | 0.168 | 6.16 | 4.66 | 4.92 | 0.210 |
| 3 | 0.210 | 5.95 | 4.58 | 4.87 | 0.257 |
| 4 | |||||
| 5 | |||||
| 6 | |||||
| 7 |
x = 0.210: fH = 3 + (−1)·x + 4·(1 − x) = 5.95, fD = 3 + 0·x + 2·(1 − x) = 4.58. fH > f̄ → hawks increase.
| generation | x (hawk share) | fH | fD | f̄ | new x = x·fH/f̄ |
|---|---|---|---|---|---|
| 0 | 0.100 | 6.50 | 4.80 | 4.97 | 0.131 |
| 1 | 0.131 | 6.35 | 4.74 | 4.95 | 0.168 |
| 2 | 0.168 | 6.16 | 4.66 | 4.92 | 0.210 |
| 3 | 0.210 | 5.95 | 4.58 | 4.87 | 0.257 |
| 4 | 0.257 | 5.72 | 4.49 | 4.80 | 0.306 |
| 5 | |||||
| 6 | |||||
| 7 |
x = 0.257: fH = 3 + (−1)·x + 4·(1 − x) = 5.72, fD = 3 + 0·x + 2·(1 − x) = 4.49. fH > f̄ → hawks increase.
| generation | x (hawk share) | fH | fD | f̄ | new x = x·fH/f̄ |
|---|---|---|---|---|---|
| 0 | 0.100 | 6.50 | 4.80 | 4.97 | 0.131 |
| 1 | 0.131 | 6.35 | 4.74 | 4.95 | 0.168 |
| 2 | 0.168 | 6.16 | 4.66 | 4.92 | 0.210 |
| 3 | 0.210 | 5.95 | 4.58 | 4.87 | 0.257 |
| 4 | 0.257 | 5.72 | 4.49 | 4.80 | 0.306 |
| 5 | 0.306 | 5.47 | 4.39 | 4.72 | 0.354 |
| 6 | |||||
| 7 |
x = 0.306: fH = 3 + (−1)·x + 4·(1 − x) = 5.47, fD = 3 + 0·x + 2·(1 − x) = 4.39. fH > f̄ → hawks increase.
| generation | x (hawk share) | fH | fD | f̄ | new x = x·fH/f̄ |
|---|---|---|---|---|---|
| 0 | 0.100 | 6.50 | 4.80 | 4.97 | 0.131 |
| 1 | 0.131 | 6.35 | 4.74 | 4.95 | 0.168 |
| 2 | 0.168 | 6.16 | 4.66 | 4.92 | 0.210 |
| 3 | 0.210 | 5.95 | 4.58 | 4.87 | 0.257 |
| 4 | 0.257 | 5.72 | 4.49 | 4.80 | 0.306 |
| 5 | 0.306 | 5.47 | 4.39 | 4.72 | 0.354 |
| 6 | 0.354 | 5.23 | 4.29 | 4.62 | 0.401 |
| 7 |
x = 0.354: fH = 3 + (−1)·x + 4·(1 − x) = 5.23, fD = 3 + 0·x + 2·(1 − x) = 4.29. fH > f̄ → hawks increase.
| generation | x (hawk share) | fH | fD | f̄ | new x = x·fH/f̄ |
|---|---|---|---|---|---|
| 0 | 0.100 | 6.50 | 4.80 | 4.97 | 0.131 |
| 1 | 0.131 | 6.35 | 4.74 | 4.95 | 0.168 |
| 2 | 0.168 | 6.16 | 4.66 | 4.92 | 0.210 |
| 3 | 0.210 | 5.95 | 4.58 | 4.87 | 0.257 |
| 4 | 0.257 | 5.72 | 4.49 | 4.80 | 0.306 |
| 5 | 0.306 | 5.47 | 4.39 | 4.72 | 0.354 |
| 6 | 0.354 | 5.23 | 4.29 | 4.62 | 0.401 |
| 7 | 0.401 | 5.00 | 4.20 | 4.52 | 0.443 |
x = 0.401: fH = 3 + (−1)·x + 4·(1 − x) = 5.00, fD = 3 + 0·x + 2·(1 − x) = 4.20. fH > f̄ → hawks increase.
Evolutionary Dynamics
From Every Interior Start (0 < x₀ < 1) to the ESS: x* = 2/3
- Rest point fH = fD: 7 − 5x = 5 − 2x ⇒ x* = 2/3, the same as the game's symmetric mixed Nash equilibrium.
- In Rock–Paper–Scissors the replicator dynamics cycle around the equilibrium; in the Stag Hunt they reach one of the two pure equilibria, depending on the start.
Repeated Games
The Repeated Prisoner's Dilemma
Points: temptation 5 > mutual cooperation 3 > mutual defection 1 > sucker 0. C: stay silent.
- One-shot, or a known finite number of rounds (backward induction): defect every round.
- If play continues with probability δ (the shadow of the future), grim trigger can sustain cooperation:
Comply: 3 every round ⇒ 3 + 3δ + … = 3/(1 − δ).
Deviate: 5 now, then 1 forever ⇒ 5 + δ/(1 − δ).
Comply ≥ deviate; times (1 − δ): 3 ≥ 5(1 − δ) + δ ⇔ δ ≥ (5 − 3)/(5 − 1) = 1/2.
δ = 0.9: comply 30, deviate 14 ⇒ cooperate. δ = 0.3: comply ≈ 4.29, deviate ≈ 5.43 ⇒ defect.
The punishment is credible: (D, D) is a stage-game equilibrium. Folk theorem: for large δ, any average payoffs above what defecting guarantees can be sustained.
Repeated Games
Grim Trigger: For Which δ Does Cooperation Last?
The more the future is worth (the larger δ), the stronger the deterrent of the grim trigger; as temptation rises, so does the threshold δ* needed for cooperation.
Repeated Games
10-Round Matches: Who Plays What?
TFT answers cooperation with cooperation: 30 points each. (Total: 30 – 30)
TFT is exploited in round 1, then always defects: 9 to 14. TFT can never win a match (at best it ties). (Total: 9 – 14)
Suspicious TFT opens with defection; the two TFTs "echo" each other → alternating C/D, 25 each. (Total: 25 – 25)
Grim never forgives a single defection: from round 2 on, mutual defection. (Total: 13 – 13)
Always Defect fully exploits the unconditional cooperator: 50 to 0. (Total: 50 – 0)
C: staying silent, i.e. cooperating (green); D: defecting (red). TFT: cooperate in round 1, then repeat the opponent's last move.
Repeated Games
Code Trace: match(tft, stft, 6)
Call match(tft, stft, 6): histories empty, scores 0.
t = 0: With empty histories, TFT opens with C and Suspicious TFT with D.
PAY["CD"] = (0, 5) → scores 0 – 5.
t = 1: Both copy the opponent's last move: TFT → D, STFT → C.
PAY["DC"] = (5, 0) → scores 5 – 5.
t = 2: Both copy the opponent's last move: TFT → C, STFT → D.
PAY["CD"] = (0, 5) → scores 5 – 10.
t = 3: Both copy the opponent's last move: TFT → D, STFT → C.
PAY["DC"] = (5, 0) → scores 10 – 10.
t = 4: Both copy the opponent's last move: TFT → C, STFT → D.
PAY["CD"] = (0, 5) → scores 10 – 15.
t = 5: Both copy the opponent's last move: TFT → D, STFT → C.
PAY["DC"] = (5, 0) → scores 15 – 15.
return (15, 15). A single opening defection starts an endless "echo": CD/DC alternating, 2.5 points per round on average (mutual cooperation would give 3).
PAY = {"CC": (3, 3), "CD": (0, 5),
"DC": (5, 0), "DD": (1, 1)}
tft = lambda me, opp: opp[-1] if opp else "C"
stft = lambda me, opp: opp[-1] if opp else "D"
def match(s1, s2, T):
h1, h2, p1, p2 = [], [], 0, 0
for t in range(T):
a, b = s1(h1, h2), s2(h2, h1)
u, v = PAY[a + b]
h1.append(a); h2.append(b)
p1, p2 = p1 + u, p2 + v
return p1, p2Repeated Games
A Mini Axelrod Tournament: Step by Step
| Strategy \ opponent | All-C | All-D | TFT | Grim | STFT | Total | Rank |
|---|---|---|---|---|---|---|---|
| Always Cooperate | 30 | 0 | 30 | 30 | 27 | 117 | |
| Always Defect | |||||||
| Tit-for-Tat | |||||||
| Grim Trigger | |||||||
| Suspicious TFT |
Always Cooperate plays 10 rounds against everyone (including its twin): total 117; it scores most against Always Cooperate (30).
| Strategy \ opponent | All-C | All-D | TFT | Grim | STFT | Total | Rank |
|---|---|---|---|---|---|---|---|
| Always Cooperate | 30 | 0 | 30 | 30 | 27 | 117 | |
| Always Defect | 50 | 10 | 14 | 14 | 10 | 98 | |
| Tit-for-Tat | |||||||
| Grim Trigger | |||||||
| Suspicious TFT |
Always Defect plays 10 rounds against everyone (including its twin): total 98; it scores most against Always Cooperate (50).
| Strategy \ opponent | All-C | All-D | TFT | Grim | STFT | Total | Rank |
|---|---|---|---|---|---|---|---|
| Always Cooperate | 30 | 0 | 30 | 30 | 27 | 117 | |
| Always Defect | 50 | 10 | 14 | 14 | 10 | 98 | |
| Tit-for-Tat | 30 | 9 | 30 | 30 | 25 | 124 | |
| Grim Trigger | |||||||
| Suspicious TFT |
Tit-for-Tat plays 10 rounds against everyone (including its twin): total 124; it scores most against Always Cooperate (30).
| Strategy \ opponent | All-C | All-D | TFT | Grim | STFT | Total | Rank |
|---|---|---|---|---|---|---|---|
| Always Cooperate | 30 | 0 | 30 | 30 | 27 | 117 | |
| Always Defect | 50 | 10 | 14 | 14 | 10 | 98 | |
| Tit-for-Tat | 30 | 9 | 30 | 30 | 25 | 124 | |
| Grim Trigger | 30 | 9 | 30 | 30 | 13 | 112 | |
| Suspicious TFT |
Grim Trigger plays 10 rounds against everyone (including its twin): total 112; it scores most against Always Cooperate (30).
| Strategy \ opponent | All-C | All-D | TFT | Grim | STFT | Total | Rank |
|---|---|---|---|---|---|---|---|
| Always Cooperate | 30 | 0 | 30 | 30 | 27 | 117 | |
| Always Defect | 50 | 10 | 14 | 14 | 10 | 98 | |
| Tit-for-Tat | 30 | 9 | 30 | 30 | 25 | 124 | |
| Grim Trigger | 30 | 9 | 30 | 30 | 13 | 112 | |
| Suspicious TFT | 32 | 10 | 25 | 13 | 10 | 90 |
Suspicious TFT plays 10 rounds against everyone (including its twin): total 90; it scores most against Always Cooperate (32).
| Strategy \ opponent | All-C | All-D | TFT | Grim | STFT | Total | Rank |
|---|---|---|---|---|---|---|---|
| Always Cooperate | 30 | 0 | 30 | 30 | 27 | 117 | 2 |
| Always Defect | 50 | 10 | 14 | 14 | 10 | 98 | 4 |
| Tit-for-Tat | 30 | 9 | 30 | 30 | 25 | 124 | 1 |
| Grim Trigger | 30 | 9 | 30 | 30 | 13 | 112 | 3 |
| Suspicious TFT | 32 | 10 | 25 | 13 | 10 | 90 | 5 |
Ranking: Tit-for-Tat (124), Always Cooperate (117), Grim Trigger (112), Always Defect (98), Suspicious TFT (90). Tit-for-Tat comes first without winning a single match! Always Defect is second to last.
Points scored in 10 rounds by the row strategy against the column opponent. All-C: Always Cooperate, All-D: Always Defect, STFT: Suspicious TFT.
Repeated Games
Axelrod's Tournaments (1980) and Their Lessons
Axelrod's two computer tournaments (14 and 62 programs) were both won by Rapoport's four-line Tit-for-Tat.
- With noise (mistaken moves), two TFTs retaliate endlessly; generous TFT sometimes forgives and does better.
- Success depends on the pool of opponents; in an evolutionary (replicator) tournament, winners multiply.
Two TFTs, one error by A in round 4: an endless C/D echo. 29 – 24 over 10 rounds (30 – 30 without it).
Summary
Discussion: Learning, People and Ethics
Summary
Summary
| Method | Description | Outcome |
|---|---|---|
| Best-response dynamics | Best response to the last action | Equilibrium in potential games; otherwise cycles |
| Fictitious play | Best response to past frequencies | Frequencies reach equilibrium in zero-sum games |
| Hedge | w ← w·eηg | RT = O(√(T ln n)); time average → CCE |
| ε-greedy / UCB | Balancing exploration and exploitation | UCB: logarithmic regret |
| Q-learning | Q ← Q + α(r + γ·max Q′ − Q) | Reward propagates back from the goal |
| Replicator | Successful strategies spread | Hawk–Dove: x* = V/C |
| Repeated PD | The shadow of the future | Cooperation if δ ≥ 1/2; TFT is strong |
Summary
Sample Problems
- Hedge with η = ln 2, two actions, payoffs (1, 0) and (1, 0) in the first two rounds. What is p1 in round 3?
- UCB1 at t = 10: arm 1 has Q = 0.6, N = 5; arm 2 has Q = 0.2, N = 1. Which arm is chosen?
- In Hawk–Dove (V = 4, C = 6, baseline 3) with x = 0.5, does the hawk share increase in the next generation?
- With temptation 5, mutual cooperation 3 and mutual defection 1, what is the smallest δ for which Grim sustains cooperation? What if the temptation were 7?
- How does fictitious play differ from best-response dynamics? Which one "approaches" equilibrium in Matching Pennies?
- In Q-learning, Q(s, a) = 1, r = 0, γ = 0.9, max Q(s′, ·) = 2, α = 0.1. What is Q(s, a) after the update?
Show answers
- w = (e2 ln 2, 1) = (4, 1) ⇒ p1 = 4/5 = 0.8.
- UCB = 0.6 + √(2 ln 10 / 5) ≈ 1.56; 0.2 + √(2 ln 10) ≈ 2.35 ⇒ arm 2 (because it was rarely tried).
- fH = 4.50, fD = 4.00, f̄ = 4.25 ⇒ x′ = 0.529 > 0.5: it increases (toward 2/3).
- δ ≥ (5 − 3)/(5 − 1) = 1/2; with temptation 7, δ ≥ (7 − 3)/(7 − 1) = 2/3.
- Best response looks only at the last action and cycles; fictitious play looks at the frequencies of the whole history, and the frequencies approach (½, ½).
- Q ← 1 + 0.1·(0 + 0.9·2 − 1) = 1.08: the target 1.8 exceeds the old value, so Q increases.
Summary
Try It Yourself
fictitious_play) and replicator dynamics (replicator_dynamics) on any payoff matrix.Summary
References
- D. Fudenberg, D. Levine, The Theory of Learning in Games, MIT Press, 1998.
- N. Cesa-Bianchi, G. Lugosi, Prediction, Learning, and Games, Cambridge, 2006.
- T. Lattimore, C. Szepesvári, Bandit Algorithms, Cambridge, 2020.
- P. Auer, N. Cesa-Bianchi, P. Fischer, “Finite-time analysis of the multiarmed bandit problem”, Machine Learning 47, 2002.
- C. Watkins, P. Dayan, “Q-learning”, Machine Learning 8, 1992; A. Blum, Y. Mansour, “From external to internal regret”, JMLR 8, 2007.
- R. Axelrod, The Evolution of Cooperation, Basic Books, 1984.
- J. Hofbauer, K. Sigmund, Evolutionary Games and Population Dynamics, Cambridge, 1998.
- T. Roughgarden, Twenty Lectures on Algorithmic Game Theory, 2016 (Lectures 16–18); course page: CS364A (Stanford, 2013).
- N. Nisan, T. Roughgarden, É. Tardos, V. Vazirani (eds.), Algorithmic Game Theory, Cambridge, 2007 (Chapter 4: learning, regret minimization and equilibria).
- A. R. Karlin, Y. Peres, Game Theory, Alive, AMS, 2017 (Chapter 7: evolutionary equilibria; Chapter 18: adaptive decision making); D. Easley, J. Kleinberg, Networks, Crowds, and Markets, 2010 (Chapter 7: evolutionary game theory).
- Tool: Nashpy (fictitious play, replicator dynamics).
Summary
References (2/2)
- J. Robinson, “An iterative method of solving a game”, Annals of Mathematics 54(2), 1951; K. Miyasawa, “On the convergence of the learning process in a 2×2 non-zero-sum two-person game”, Princeton Research Memo 33, 1961; L. S. Shapley, “Some topics in two-person games”, Advances in Game Theory, 1964.
- D. Monderer, L. S. Shapley, “Potential games”, Games and Economic Behavior 14(1), 1996.
- D. Foster, R. Vohra, “Calibrated learning and correlated equilibrium”, Games and Economic Behavior 21, 1997; S. Hart, A. Mas-Colell, “A simple adaptive procedure leading to correlated equilibrium”, Econometrica 68(5), 2000.
- J. Maynard Smith, G. R. Price, “The logic of animal conflict”, Nature 246, 1973.
- W. R. Thompson, “On the likelihood that one unknown probability exceeds another in view of the evidence of two samples”, Biometrika 25, 1933.
- P. Auer, N. Cesa-Bianchi, Y. Freund, R. E. Schapire, “The nonstochastic multiarmed bandit problem”, SIAM Journal on Computing 32(1), 2002.
- N. D. Daw, J. P. O’Doherty, P. Dayan, B. Seymour, R. J. Dolan, “Cortical substrates for exploratory decisions in humans”, Nature 441, 2006.
- C. Watkins, Learning from Delayed Rewards, PhD thesis, University of Cambridge, 1989; V. Mnih et al., “Human-level control through deep reinforcement learning”, Nature 518, 2015.
- M. Zinkevich, M. Johanson, M. Bowling, C. Piccione, “Regret minimization in games with incomplete information”, NeurIPS, 2007.
- M. Nowak, K. Sigmund, “A strategy of win-stay, lose-shift that outperforms tit-for-tat in the Prisoner’s Dilemma game”, Nature 364, 1993.