Mixture of Experts
Mixture of Experts (MoE) is a neural network architecture that replaces a single dense feed-forward layer with many smaller "expert" sub-networks and a learned router that activates only a small subset of them per input, increasing total…
Definition
Mixture of Experts (MoE) is a neural network architecture that replaces a single dense feed-forward layer with many smaller "expert" sub-networks and a learned router that activates only a small subset of them per input, increasing total model capacity without proportionally increasing inference compute.
Overview
In a standard dense Transformer, every token passes through the same feed-forward layer using all of that layer's parameters. Mixture of Experts changes this by replacing that dense layer with a bank of many expert sub-networks — each a smaller feed-forward network — plus a lightweight router (or "gate") that, for each token, selects a small number of experts (commonly one or two out of dozens) to actually process it. Because only a fraction of the total parameters are active for any given token, an MoE model can have a very large total parameter count while keeping the actual compute cost per token much closer to that of a smaller dense model. This sparsity is what makes MoE attractive at scale: it decouples total model capacity (which correlates with how much a model can "know" or represent) from per-token inference cost (which determines latency and serving expense). Notable production models using MoE include Mistral's Mixtral line, Google's Gemini and Switch Transformer research, and reportedly some of the largest frontier models from major labs. The tradeoff is added engineering complexity: routing decisions must be load-balanced across experts to avoid some experts being overused and others undertrained, and MoE models require more GPU memory to hold all experts even though only a few are active per token, which complicates deployment compared to dense models of equivalent active-parameter count. Training MoE models also introduces unique challenges, such as router collapse (where the router learns to send most tokens to only a few experts, wasting capacity) and the need for auxiliary load-balancing losses during training to keep expert utilization roughly even. Despite this complexity, MoE has become one of the standard techniques for scaling frontier models efficiently, alongside advances in attention mechanisms and hardware-aware training.
Key Concepts
- Replaces a dense feed-forward layer with many smaller expert sub-networks
- A learned router activates only a small subset of experts per token
- Decouples total model capacity from per-token inference compute cost
- Enables much larger total parameter counts at manageable serving cost
- Requires load-balancing techniques to prevent router collapse onto few experts
- Needs more GPU memory to hold all experts despite sparse activation
- Used in production models such as Mixtral and other frontier-scale systems
- Adds architectural and training complexity relative to dense models