MuZero
By Google DeepMind
MuZero is a reinforcement learning algorithm from Google DeepMind, introduced in 2019, that learns to master games and planning tasks — including chess, shogi, Go, and Atari games — without ever being given the rules or dynamics of the…
Definition
MuZero is a reinforcement learning algorithm from Google DeepMind, introduced in 2019, that learns to master games and planning tasks — including chess, shogi, Go, and Atari games — without ever being given the rules or dynamics of the environment, instead learning an internal model of only the aspects relevant to predicting rewards and outcomes.
Overview
MuZero's predecessor, AlphaZero, could master chess, shogi, and Go through self-play, but it required a known, hand-coded simulator of the game's exact rules to plan ahead. MuZero removed that requirement: rather than being given the rules, it learns its own internal, abstract model of the environment purely from the outcomes it observes — predicting only what matters for choosing good actions and estimating future rewards, not necessarily a faithful pixel-level or rule-level reconstruction of the environment itself. This distinction let MuZero extend beyond board games, where the rules are known and fixed, into more complex domains like Atari video games, where the underlying game logic is not provided to the agent in advance and must be inferred from observation. In DeepMind's evaluation, MuZero matched AlphaZero's superhuman performance in chess, shogi, and Go — building on the same lineage of research as AlphaGo — while also achieving state-of-the-art results on the Atari benchmark suite at the time of release, using the same general algorithm across all of these very different environments. Architecturally, MuZero learns three connected neural network functions: a representation function that encodes an observation into an internal state, a dynamics function that predicts how that internal state changes given an action, and a prediction function that estimates the value and best next moves from that state — all trained jointly so that the model's internal predictions stay accurate enough to support effective Monte Carlo tree search planning. MuZero's approach — learning a model of "what matters" for planning rather than a complete simulation of the environment — influenced subsequent research into model-based reinforcement learning more broadly, including applications explored by DeepMind beyond games, such as using MuZero-derived techniques for practical optimization problems like video compression.
Key Concepts
- Learns to plan without being given the explicit rules of the environment
- Builds an internal, abstract model predicting only reward-relevant dynamics
- Matches AlphaZero's superhuman performance in chess, shogi, and Go
- Achieved state-of-the-art results on the Atari benchmark suite at release
- Combines representation, dynamics, and prediction neural network functions
- Uses Monte Carlo tree search over its learned internal model to plan actions