Policy Gradient
Policy gradient methods are a family of reinforcement learning algorithms that directly optimize a parameterized policy by estimating the gradient of expected reward with respect to the policy's parameters, rather than learning value…
Definition
Policy gradient methods are a family of reinforcement learning algorithms that directly optimize a parameterized policy by estimating the gradient of expected reward with respect to the policy's parameters, rather than learning value functions first.
Overview
Unlike value-based methods such as Q-learning, which learn action values and derive a policy indirectly, policy gradient methods parameterize the policy itself — often as a neural network mapping states to a probability distribution over actions — and update its parameters directly to increase the probability of actions that led to higher returns. The core mathematical tool is the policy gradient theorem, which shows that the gradient of expected return can be estimated from sampled trajectories without needing to differentiate through the environment's dynamics. The REINFORCE algorithm is the simplest instance: it increases the log-probability of actions taken in a trajectory in proportion to the return that followed. Because raw returns produce high-variance gradient estimates, practical policy gradient methods subtract a baseline, commonly a learned value function, giving rise to the actor-critic family, where an 'actor' network selects actions and a 'critic' network estimates values to reduce variance. Trust Region Policy Optimization (TRPO) and its simpler successor Proximal Policy Optimization (PPO) further stabilize training by constraining how much the policy can change in a single update, preventing destructively large steps. Policy gradient methods have a key practical advantage over value-based methods: they extend naturally to continuous action spaces (such as robot joint torques) where taking a max over actions, as Q-learning requires, is intractable. They also naturally support stochastic policies, useful in partially observable or adversarial settings. PPO in particular became a standard tool not just in robotics and game-playing agents but in fine-tuning large language models via reinforcement learning from human feedback (RLHF), where a reward model scores generated text and PPO updates the language model's policy to produce higher-scoring outputs.
Key Concepts
- Directly optimizes policy parameters via gradient ascent on expected reward
- Based on the policy gradient theorem, estimable from sampled trajectories
- REINFORCE as the foundational, high-variance baseline algorithm
- Baseline subtraction (e.g. a learned value function) reduces gradient variance
- Actor-critic architectures combine a policy (actor) and value estimator (critic)
- Trust-region methods (TRPO, PPO) constrain update size for stable training
- Naturally handles continuous and high-dimensional action spaces
- Supports stochastic policies useful under uncertainty or partial observability