Q-Learning
Q-learning is a model-free reinforcement learning algorithm that learns the value of taking a given action in a given state (the Q-value) through trial and error, enabling an agent to derive an optimal policy without knowing the…
Definition
Q-learning is a model-free reinforcement learning algorithm that learns the value of taking a given action in a given state (the Q-value) through trial and error, enabling an agent to derive an optimal policy without knowing the environment's dynamics in advance.
Overview
Q-learning estimates an action-value function, Q(s, a), representing the expected cumulative discounted reward of taking action a in state s and then following the optimal policy thereafter. It is model-free, meaning it does not require knowledge of the environment's transition probabilities or reward function, and off-policy, meaning it can learn the optimal policy's values while following a different, exploratory behavior policy. The algorithm updates its Q-value estimates using a temporal-difference rule derived from the Bellman optimality equation: after observing a transition (state, action, reward, next state), it nudges Q(s, a) toward the observed reward plus the discounted value of the best action available in the next state. Over many episodes of exploration, typically guided by an epsilon-greedy strategy that balances exploring new actions against exploiting known good ones, the Q-values converge toward their true optimal values under standard conditions, and the greedy policy with respect to those Q-values becomes optimal. Classic Q-learning stores values in a table indexed by state and action, which works well for small, discrete state spaces but becomes infeasible for large or continuous ones. Deep Q-Networks (DQN), introduced by DeepMind in 2015, replaced the table with a neural network that approximates Q-values from raw inputs like pixels, combined with techniques such as experience replay and a separate target network to stabilize training — this combination famously achieved human-level performance on a range of Atari games from raw pixel input alone. Q-learning and its deep variants remain a foundational, widely taught reinforcement learning technique, valued for conceptual simplicity even as policy gradient and actor-critic methods have become dominant for more complex, continuous-control problems.
Key Concepts
- Model-free — learns without needing the environment's transition or reward functions
- Off-policy — learns optimal values while following an exploratory behavior policy
- Temporal-difference updates derived directly from the Bellman optimality equation
- Epsilon-greedy exploration balances trying new actions versus exploiting known ones
- Tabular form suits small discrete state-action spaces
- Deep Q-Networks (DQN) extend it with neural function approximation
- Experience replay and target networks stabilize deep variants
- Provably converges to optimal Q-values under standard conditions