Gradient Clipping
Gradient clipping is a training technique that caps the magnitude of gradients during backpropagation, preventing the exploding-gradient problem that can destabilize neural network training, particularly in recurrent networks and deep…
Definition
Gradient clipping is a training technique that caps the magnitude of gradients during backpropagation, preventing the exploding-gradient problem that can destabilize neural network training, particularly in recurrent networks and deep transformers.
Overview
During backpropagation, gradients can occasionally grow extremely large, especially in deep networks or recurrent architectures processing long sequences. When this happens, a single optimizer step can push weights far outside a reasonable range, causing the loss to spike, diverge, or produce NaN values that permanently corrupt training. Gradient clipping addresses this by rescaling or truncating gradients before they are applied to the weights. There are two common variants. Clipping by value simply truncates each individual gradient component to lie within a fixed range, such as [-1, 1]. Clipping by norm, the more widely used approach, computes the overall norm of the gradient vector (or of all parameters combined) and, if that norm exceeds a threshold, rescales the entire vector so its norm equals the threshold, preserving the gradient's direction while limiting its size. Norm-based clipping is generally preferred because it does not distort the relative proportions between gradient components. Gradient clipping is especially important for recurrent neural networks and LSTMs, where gradients are backpropagated through many time steps and can compound multiplicatively, and it remains standard practice in training large transformer-based language models, where a clip threshold (commonly around 1.0) is applied globally across all parameters at every optimizer step. It is typically combined with other stabilization techniques such as learning rate warmup, weight decay, and careful weight initialization. While clipping does not fix an underlying instability in model architecture or learning rate choice, it acts as a safety valve that lets training tolerate occasional large gradients — for example from an unusual batch or an early, poorly calibrated model state — without derailing the whole run. Most deep learning frameworks expose a simple utility function to apply gradient clipping in a single line before the optimizer step.
Key Concepts
- Prevents exploding gradients from destabilizing training
- Two main forms: clip-by-value and clip-by-norm (global norm clipping)
- Norm clipping preserves gradient direction while limiting magnitude
- Standard practice in training RNNs, LSTMs, and large transformers
- Applied just before the optimizer step, after backpropagation
- Threshold is a tunable hyperparameter, commonly around 1.0 for norm clipping
- Complements learning rate warmup and weight initialization for training stability
- Built into most deep learning frameworks as a single utility call