Reinforcement Learning

Guide to AI and Data Courses · Part II: Branches of AI
Agent and environmentMarkov decision processValue functionsExploration vs. exploitationDeep RLNeighboring courses
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

What Question Does It Ask?

How can good decisions be learned by trial and error from a reward signal alone?

Reinforcement learning studies how an agent, interacting with its environment and guided only by a reward signal, learns by trial and error the behavior that maximizes total reward in the long run. The right action is never given in advance, and the effects of actions often arrive with a delay.

At a glance

Approach

trial and error with rewards

Input

environment: state, reward

Output

policy: state → action

Success measure

total expected reward

Math load

high: probability, dynamic programming

Data needs

generated through interaction

Introduction

A Short History

1957

Bellman's Dynamic Programming introduces the Bellman equation and the idea of a value function.

1988

Sutton publishes temporal-difference (TD) learning: correcting a prediction with the next prediction.

1989

Watkins proposes Q-learning in his PhD thesis.

1992

Tesauro's TD-Gammon nears expert level at backgammon by playing against itself.

2015

DeepMind's deep Q-network (DQN) learns Atari games from raw pixels (Nature).

2016

AlphaGo beats Lee Sedol at Go: tree search, deep networks and reinforcement learning together.

The Course

Core Concepts

Agent and environment

The agent acts; the environment returns a new state and a reward.

Markov decision process

States, actions, transition probabilities and rewards.

Return

The sum of discounted future rewards.

Policy

A rule that says which action to choose in each state.

Value function

The expected return from a state or an action.

Exploration vs. exploitation

Balancing trying new actions against using what is known.

Temporal difference

TD, Q-learning, SARSA: model-free, step-by-step learning.

Policy gradient

Improving the policy directly with the gradient of the return.

The Course

A Typical Workflow

Observestatefrom theenvironmentChooseactionpolicy,ε-greedyGet rewardnew stateUpdatevalueTD errorImprovepolicythousands ofepisodes
Example: A warehouse robot observes its position on a grid, usually picks the best direction it knows and sometimes a random one, and gets +10 for reaching the shelf and −1 for hitting a wall. Q-learning updates action values at every step; after thousands of trials a short, safe path emerges.

The Course

What Does the Course Cover?

  1. Multi-armed bandits and exploration
  2. Markov decision processes and Bellman equations
  3. Dynamic programming: policy and value iteration
  4. Monte Carlo methods
  5. Temporal-difference learning: TD(0) and SARSA
  6. Q-learning and off-policy learning
  7. n-step methods and eligibility traces
  8. Value estimation with function approximation
  9. Deep Q-networks: experience replay, target network
  10. Policy gradients and REINFORCE
  11. Actor-critic methods and PPO
  12. Model-based RL and Monte Carlo tree search

Skills you will gain

building environments with Gymnasiumwriting Q-learning from scratchtraining agents with Stable-Baselines3designing reward functionsrunning and reporting multi-seed experiments

A typical semester; the order and coverage vary between universities.

Try It

The Vacuum Finds Its Charger

The robot vacuum has no map; each step is −1, the charging dock +10, falling down the stairs −10. Let it learn by trial over episodes: train it, then send the robot from the top-left corner.

Nobody showed the robot the way: after each step Q-learning updates the value of the action in that cell with reward + best future value. With no exploration no new paths are found; with too much the robot falls often.

Context

Prerequisites

Courses worth knowing before this one, and why they are needed.

CourseWhy it is needed
Machine Learninglearning from rewards instead of labels
Algorithmsdynamic programming and shortest paths in graphs
Programmingsimulation loops and arrays in Python

Context

Are You Ready for This Course?

Ask yourself; for every "no", the right-hand column shows where to start.

Ask yourselfIf not, start with
Can you compute an expected value and a conditional probability?Probability and Statistics
Can you solve a shortest-path problem with dynamic programming?Dynamic programming from an algorithms course
Can you explain a loss function and gradient descent?Machine Learning
Can you write a simulation loop in Python and plot the result?An intro programming course, then a short Matplotlib tutorial
If you can say "yes" to most of these, you are ready; one or two gaps can be closed during the semester.

Context

Neighbouring Courses

subfieldbuilds onshares methodsapplication

Overlap

Overlapping Courses

RelationCourseShared ground
subfieldThis course is a subfield of Machine Learninglearning from rewards instead of labels
shares methodsShares methods with Deep Learningdeep Q-networks, deep policy networks
shares methodsShares methods with Multi-Agent Systemsmulti-agent reinforcement learning, game theory
shares methodsShares methods with Robotics and Robot Visioncontrol, learning motor skills
shares methodsShares methods with Optimizationdynamic programming, policy gradient

Differences

Where It Differs (1/2)

CriterionReinforcement LearningMachine LearningOptimization
Core questionHow can good decisions be learned by trial and error from a reward signal alone?How does a program generalize from examples?How do we find the values that make an objective function best?
Approachtrial and error with rewardsstatistical learningmathematical model of objective and constraints
Inputenvironment: state, rewardlabeled or unlabeled examplesobjective function and constraints
Outputpolicy: state → actiona model that makes predictionsoptimal or near-optimal solution
Success measuretotal expected rewarderror on unseen dataobjective value, convergence rate
Math loadhigh: probability, dynamic programminghigh: linear algebra, probability, optimizationthe theory itself
Data needsgenerated through interactionmoderate to largenone
  • Machine Learning: In supervised learning every example comes with the right answer; in RL there is only a delayed reward, and data is collected by the agent's own actions.
  • Optimization: Optimization solves a problem whose model is known; RL learns by interacting, without knowing the environment's model.

Differences

Where It Differs (2/2)

CriterionReinforcement LearningMulti-Agent SystemsRobotics
Core questionHow can good decisions be learned by trial and error from a reward signal alone?How do several autonomous agents decide together or against each other?How does a machine sense the physical world, know where it is and move?
Approachtrial and error with rewardsinteracting autonomous agentssense-plan-act loop
Inputenvironment: state, rewardagents, goals, protocolssensor measurements
Outputpolicy: state → actionjoint behavior, equilibriummotor commands, path
Success measuretotal expected rewardsocial welfare, stabilitytask success, safety
Math loadhigh: probability, dynamic programmingmedium: game theoryhigh: transformations, control
Data needsgenerated through interactionlittle; simulationphysical experiments and simulation
  • Multi-Agent Systems: Multi-agent systems study coordination and conflict among agents; this course mostly focuses on a single agent learning.
  • Robotics: Robotics covers perception, kinematics and control; RL contributes only learning behavior from experience.

Practice

Applications

Games

Go, chess, Atari and strategy games.

Robot control

Learning to walk, grasp and move.

Resource management

Data center cooling, network traffic routing.

Recommender systems

Caring about long-term user satisfaction.

Traffic signals

Reducing waiting times at intersections.

Language model fine-tuning

Reinforcement learning from human feedback (RLHF).

Practice

Risks and Ethics

Reward hacking

OpenAI's boat-racing agent circled endlessly to hit the same scoring targets instead of finishing the race.

Unsafe exploration

Trial and error is costly in the real world: a robot or a treatment policy can cause harm while it explores.

Results that do not reproduce

The same algorithm gives very different results across random seeds; an advantage reported over a few seeds can mislead.

Shaping the user

A recommendation policy that maximizes clicks can push users toward more extreme content and compulsive use.

In depth: Explainable AI and AI Ethics.

Practice

Tools and Datasets

Tools and libraries

GymnasiumStable-Baselines3PyTorchRLlibMuJoCoPettingZoo

Classic environments

GridworldMulti-armed banditCartPoleMountainCarAtari games (ALE)MuJoCo control tasks
RL gives you an environment instead of a ready-made dataset: the agent produces data by its own actions, so simulation is the course's lab.

Practice

Common Misconceptions

Common beliefIn fact
RL is just another name for supervised learning.The right action is never given; the agent learns only from rewards and the experience it collects.
Writing the reward is easy.An agent exploits a poorly designed reward with unexpected shortcuts (reward hacking).
A robot that learns in simulation works in reality.The gap between simulation and the real world (sim-to-real) must be handled separately.
RL is only for games.Control, resource management and training language models from human feedback also use RL.

Next Steps

After This Course

Courses that build on or extend this one

This course is mostly an end point: it builds on the others.

Related careers

Reinforcement learning engineerRobotics software engineerGame AI developerResearcher
After reinforcement learning you can move to multi-agent systems, where many agents learn together, or to robotics, for machines that learn in the physical world. Deep learning helps with deep RL, and optimization makes policy gradient methods easier to understand.

Next Steps

Further Reading

  • Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
  • Szepesvári, Algorithms for Reinforcement Learning, Morgan & Claypool, 2010.
  • Bertsekas, Reinforcement Learning and Optimal Control, Athena Scientific, 2019.
  • Mnih et al., "Human-level control through deep reinforcement learning", Nature 518, 2015.

The End

Chapter 7: Reinforcement Learning
← Previous chapter · Map of the field · Next chapter →
1 / 1 Sercan KÜLCÜ. All rights reserved.