Reinforcement Learning
Introduction
What Question Does It Ask?
Reinforcement learning studies how an agent, interacting with its environment and guided only by a reward signal, learns by trial and error the behavior that maximizes total reward in the long run. The right action is never given in advance, and the effects of actions often arrive with a delay.
At a glance
trial and error with rewards
environment: state, reward
policy: state → action
total expected reward
high: probability, dynamic programming
generated through interaction
Introduction
A Short History
Bellman's Dynamic Programming introduces the Bellman equation and the idea of a value function.
Sutton publishes temporal-difference (TD) learning: correcting a prediction with the next prediction.
Watkins proposes Q-learning in his PhD thesis.
Tesauro's TD-Gammon nears expert level at backgammon by playing against itself.
DeepMind's deep Q-network (DQN) learns Atari games from raw pixels (Nature).
AlphaGo beats Lee Sedol at Go: tree search, deep networks and reinforcement learning together.
The Course
Core Concepts
The agent acts; the environment returns a new state and a reward.
States, actions, transition probabilities and rewards.
The sum of discounted future rewards.
A rule that says which action to choose in each state.
The expected return from a state or an action.
Balancing trying new actions against using what is known.
TD, Q-learning, SARSA: model-free, step-by-step learning.
Improving the policy directly with the gradient of the return.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Multi-armed bandits and exploration
- Markov decision processes and Bellman equations
- Dynamic programming: policy and value iteration
- Monte Carlo methods
- Temporal-difference learning: TD(0) and SARSA
- Q-learning and off-policy learning
- n-step methods and eligibility traces
- Value estimation with function approximation
- Deep Q-networks: experience replay, target network
- Policy gradients and REINFORCE
- Actor-critic methods and PPO
- Model-based RL and Monte Carlo tree search
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
The Vacuum Finds Its Charger
The robot vacuum has no map; each step is −1, the charging dock +10, falling down the stairs −10. Let it learn by trial over episodes: train it, then send the robot from the top-left corner.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Machine Learning | learning from rewards instead of labels |
| Algorithms | dynamic programming and shortest paths in graphs |
| Programming | simulation loops and arrays in Python |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you compute an expected value and a conditional probability? | Probability and Statistics |
| Can you solve a shortest-path problem with dynamic programming? | Dynamic programming from an algorithms course |
| Can you explain a loss function and gradient descent? | Machine Learning |
| Can you write a simulation loop in Python and plot the result? | An intro programming course, then a short Matplotlib tutorial |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| subfield | This course is a subfield of Machine Learning | learning from rewards instead of labels |
| shares methods | Shares methods with Deep Learning | deep Q-networks, deep policy networks |
| shares methods | Shares methods with Multi-Agent Systems | multi-agent reinforcement learning, game theory |
| shares methods | Shares methods with Robotics and Robot Vision | control, learning motor skills |
| shares methods | Shares methods with Optimization | dynamic programming, policy gradient |
Differences
Where It Differs (1/2)
| Criterion | Reinforcement Learning | Machine Learning | Optimization |
|---|---|---|---|
| Core question | How can good decisions be learned by trial and error from a reward signal alone? | How does a program generalize from examples? | How do we find the values that make an objective function best? |
| Approach | trial and error with rewards | statistical learning | mathematical model of objective and constraints |
| Input | environment: state, reward | labeled or unlabeled examples | objective function and constraints |
| Output | policy: state → action | a model that makes predictions | optimal or near-optimal solution |
| Success measure | total expected reward | error on unseen data | objective value, convergence rate |
| Math load | high: probability, dynamic programming | high: linear algebra, probability, optimization | the theory itself |
| Data needs | generated through interaction | moderate to large | none |
- Machine Learning: In supervised learning every example comes with the right answer; in RL there is only a delayed reward, and data is collected by the agent's own actions.
- Optimization: Optimization solves a problem whose model is known; RL learns by interacting, without knowing the environment's model.
Differences
Where It Differs (2/2)
| Criterion | Reinforcement Learning | Multi-Agent Systems | Robotics |
|---|---|---|---|
| Core question | How can good decisions be learned by trial and error from a reward signal alone? | How do several autonomous agents decide together or against each other? | How does a machine sense the physical world, know where it is and move? |
| Approach | trial and error with rewards | interacting autonomous agents | sense-plan-act loop |
| Input | environment: state, reward | agents, goals, protocols | sensor measurements |
| Output | policy: state → action | joint behavior, equilibrium | motor commands, path |
| Success measure | total expected reward | social welfare, stability | task success, safety |
| Math load | high: probability, dynamic programming | medium: game theory | high: transformations, control |
| Data needs | generated through interaction | little; simulation | physical experiments and simulation |
- Multi-Agent Systems: Multi-agent systems study coordination and conflict among agents; this course mostly focuses on a single agent learning.
- Robotics: Robotics covers perception, kinematics and control; RL contributes only learning behavior from experience.
Practice
Applications
Go, chess, Atari and strategy games.
Learning to walk, grasp and move.
Data center cooling, network traffic routing.
Caring about long-term user satisfaction.
Reducing waiting times at intersections.
Reinforcement learning from human feedback (RLHF).
Practice
Risks and Ethics
OpenAI's boat-racing agent circled endlessly to hit the same scoring targets instead of finishing the race.
Trial and error is costly in the real world: a robot or a treatment policy can cause harm while it explores.
The same algorithm gives very different results across random seeds; an advantage reported over a few seeds can mislead.
A recommendation policy that maximizes clicks can push users toward more extreme content and compulsive use.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Classic environments
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| RL is just another name for supervised learning. | The right action is never given; the agent learns only from rewards and the experience it collects. |
| Writing the reward is easy. | An agent exploits a poorly designed reward with unexpected shortcuts (reward hacking). |
| A robot that learns in simulation works in reality. | The gap between simulation and the real world (sim-to-real) must be handled separately. |
| RL is only for games. | Control, resource management and training language models from human feedback also use RL. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
- Szepesvári, Algorithms for Reinforcement Learning, Morgan & Claypool, 2010.
- Bertsekas, Reinforcement Learning and Optimal Control, Athena Scientific, 2019.
- Mnih et al., "Human-level control through deep reinforcement learning", Nature 518, 2015.