Multi-Armed Bandit
Choose an arm. Reward probabilities are unknown.
How this teaches game theory
Scenario, Mode, Level, and Algorithm Guide
This reference explains every selectable configuration in the laboratory. Use it after an experiment to interpret why the same learner may behave differently under another mode, level, or algorithm.
Experiment scenario
Unknown reward scenario. Each arm represents an action whose true probability of producing reward 1 is hidden. A pull returns either 1 or 0. The learner must estimate each arm from experience and decide when to explore uncertain arms and when to exploit the arm that currently looks best.
At the start of every new experiment, the hidden probabilities are generated again. Consequently, memorizing a previously successful arm is not useful. The central objective is to learn efficiently while keeping cumulative regret low.
Core measures: Reward is the total number of successes; rounds count all pulls; estimated reward is the observed average for an arm; regret compares the learner's accumulated reward with the expected reward of always selecting the currently best true arm.
Modes
Human vs Computer benchmark
You choose every arm manually. The algorithm remains available through the automatic-run button as a benchmark. This mode is best for comparing human intuition with a formal learning rule.
- Primary skill: forming and revising beliefs.
- Observe whether you explore enough before committing.
- Compare your reward and regret with algorithmic runs.
Human-assisted recommendation
You still make the final choice, but the interface marks the arm currently recommended by the selected algorithm. You may follow or reject that advice.
- Primary skill: decision support and human oversight.
- Observe when advice is uncertain or changes.
- Useful for discussing automation bias.
Computer learning
The selected algorithm controls the choices and automatically performs an initial batch of rounds when a new experiment begins. You can then continue the simulation in larger batches.
- Primary skill: observing learning dynamics.
- Compare convergence speed and regret.
- Repeat experiments because outcomes are stochastic.
Levels and scenarios
Level 1 — 3 arms
A stationary environment with three actions. Hidden reward probabilities remain fixed throughout the experiment.
- Lowest search complexity.
- Best introduction to exploration versus exploitation.
- Recommended for learning how estimates stabilize.
Level 2 — 4 arms
A stationary environment with four actions. The extra arm increases uncertainty and spreads exploration across more alternatives.
- More observations are usually needed.
- Premature commitment becomes more costly.
- Useful for comparing sample efficiency.
Level 3 — Changing rewards
A non-stationary environment with four actions. After round 40, the hidden reward probabilities change. An arm that was previously best may no longer be best.
- Tests adaptation rather than only initial learning.
- Old observations can become misleading.
- Watch whether the algorithm resumes exploration.
Algorithms
ε-greedy
With probability ε, the learner explores a random arm; otherwise it exploits the arm with the highest observed mean reward. This implementation uses ε = 0.15.
- Simple and easy to interpret.
- Continues exploring even after finding a strong arm.
- Random exploration may waste pulls on clearly weak arms.
- Often adapts better than a fully greedy rule after change.
UCB1
UCB1 adds an uncertainty bonus to each arm's estimated mean. Arms with few observations receive a larger bonus and are explored until uncertainty decreases.
score = estimated mean + √(2 ln(t) / pulls)
- Explores purposefully rather than uniformly.
- Strong in stationary settings.
- Standard UCB1 can adapt slowly after sudden changes because old evidence remains influential.
Mode × level behavior
| Mode | Level 1 — 3 arms | Level 2 — 4 arms | Level 3 — Changing rewards |
|---|---|---|---|
| Human vs Computer benchmark | Learn the basic explore–exploit trade-off with a small choice set. | Test whether human attention is spread fairly across four uncertain options. | Detect the change after round 40 and decide when earlier beliefs should be discarded. |
| Human-assisted recommendation | Compare personal intuition with a relatively stable recommendation. | Observe recommendation changes while evidence is distributed across more arms. | Evaluate whether following old algorithmic advice delays adaptation after the environment changes. |
| Computer learning | See fast convergence and establish a baseline. | Compare the algorithms under greater exploration demand. | Inspect recovery after change and the effect of stale historical observations. |
Algorithm × level behavior
| Algorithm | Level 1 — 3 arms | Level 2 — 4 arms | Level 3 — Changing rewards |
|---|---|---|---|
| ε-greedy | Usually identifies a promising arm quickly while maintaining occasional random checks. | Random exploration is divided among more arms, so estimates may stabilize more slowly. | Continued exploration gives the algorithm a chance to discover that the best arm changed. |
| UCB1 | Systematically samples all arms and then concentrates on the most promising one. | Uses uncertainty bonuses to allocate exploration across the larger action set. | May react slowly because observations collected before round 40 remain in the averages. |
Recommended learning sequence
- Start with Human vs Computer benchmark + Level 1 + ε-greedy and make at least 20 manual choices.
- Repeat Level 1 with UCB1 and compare how evenly the arms are sampled.
- Move to Level 2 to see how an additional option increases the cost of exploration.
- Use Human-assisted recommendation and deliberately reject some recommendations; evaluate the consequences.
- Finish with Computer learning + Level 3 and compare both algorithms after the round-40 change.