Gradient Accumulation
Gradient accumulation is a training technique that simulates a larger batch size than fits in available memory by summing gradients over several smaller mini-batches before performing a single optimizer update.
Definition
Gradient accumulation is a training technique that simulates a larger batch size than fits in available memory by summing gradients over several smaller mini-batches before performing a single optimizer update.
Overview
Larger batch sizes generally produce more stable gradient estimates and can improve training throughput, but they require proportionally more memory to hold activations for the forward and backward pass. GPU or TPU memory is often the binding constraint, especially when training large models like transformers, which forces practitioners to use smaller batch sizes than they would prefer. Gradient accumulation offers a workaround: instead of updating the model's weights after every mini-batch, the gradients from several consecutive mini-batches are accumulated (summed or averaged) and the optimizer step is only applied after a chosen number of accumulation steps. Mechanically, this means the forward and backward passes run normally for each small mini-batch, but the resulting gradients are added to a running total rather than immediately used to update weights, and the model's parameters are held fixed until enough mini-batches have been processed. Once the target number of accumulation steps is reached, the accumulated gradients are used for a single optimizer update, and the accumulator is reset to zero. The effective batch size becomes the mini-batch size multiplied by the number of accumulation steps, letting a team train with an effective batch size of, say, 512 while only ever holding activations for a mini-batch of 16 in memory at once. This is particularly valuable for large language model training and fine-tuning, where memory is the primary bottleneck and the desired batch size for stable, efficient training may be far larger than what a single GPU can hold. Gradient accumulation trades wall-clock training time for memory savings, since it requires multiple sequential forward/backward passes to achieve the update frequency of one large batch. It is commonly combined with mixed-precision training and memory-efficient optimizers to further reduce memory pressure, and most training frameworks (including Hugging Face Transformers and PyTorch Lightning) support it as a simple configuration flag.
Key Concepts
- Simulates a large effective batch size without needing the memory for it
- Sums or averages gradients over multiple mini-batches before an optimizer step
- Effective batch size equals mini-batch size times number of accumulation steps
- Trades additional wall-clock time for reduced memory usage
- Commonly paired with mixed-precision training to further save memory
- Widely supported via a simple configuration option in major training frameworks
- Especially useful for fine-tuning large language models on limited hardware