Actor-Critic Method
Actor-critic methods are a family of reinforcement learning algorithms that combine a policy network (the actor), which selects actions, with a value-function network (the critic), which evaluates those actions to reduce the variance of…
Definition
Actor-critic methods are a family of reinforcement learning algorithms that combine a policy network (the actor), which selects actions, with a value-function network (the critic), which evaluates those actions to reduce the variance of the actor's learning signal.
Overview
Actor-critic methods sit between pure policy-gradient methods and pure value-based methods, taking the direct policy optimization of the former and combining it with the variance reduction offered by learning a value function, as in the latter. The actor is a parameterized policy that chooses actions given a state; the critic estimates a value function — typically the state value V(s) or the advantage function A(s, a) — that tells the actor whether an action turned out better or worse than expected, providing a lower-variance training signal than raw episodic returns. Training proceeds by alternating updates: the critic is updated using temporal-difference learning, similar in spirit to Q-learning, to make its value estimates more accurate; the actor is updated using the policy gradient theorem, weighted by the critic's advantage estimate rather than by the full noisy return. This decomposition allows actor-critic methods to learn online, from single steps or short rollouts, rather than waiting for full episodes to complete as pure Monte Carlo policy-gradient methods like REINFORCE must. Several influential algorithms extend this basic template. Advantage Actor-Critic (A2C) and its asynchronous predecessor A3C parallelize rollouts across multiple environment instances for more stable, decorrelated updates. Proximal Policy Optimization (PPO) is technically an actor-critic method that additionally constrains policy updates to a trust region for stability, and it has become one of the most widely used reinforcement learning algorithms in practice, including for RLHF-based fine-tuning of large language models. Soft Actor-Critic (SAC) adds an entropy-maximization term to encourage exploration and has become a standard choice for continuous-control robotics tasks. Because they combine the strengths of both value-based and policy-based approaches, actor-critic methods are now the dominant paradigm in deep reinforcement learning research and applications.
Key Concepts
- Combines a policy network (actor) with a value-estimating network (critic)
- Critic reduces variance of policy gradient updates via TD-learned value/advantage estimates
- Supports online, step-by-step updates rather than requiring full episode rollouts
- Advantage function highlights whether an action outperformed the state's average value
- A2C/A3C parallelize environment rollouts for more stable training
- PPO adds trust-region-style clipping for stable large-scale training
- Soft Actor-Critic (SAC) adds entropy regularization for better exploration
- Dominant paradigm in modern deep reinforcement learning