Batch Normalization
Batch normalization is a neural network training technique that normalizes the inputs to each layer using the mean and variance computed over a mini-batch, then rescales and shifts the result with learnable parameters.
Definition
Batch normalization is a neural network training technique that normalizes the inputs to each layer using the mean and variance computed over a mini-batch, then rescales and shifts the result with learnable parameters.
Overview
Introduced by Sergey Ioffe and Christian Szegedy in 2015, batch normalization addresses a problem the authors called "internal covariate shift" — the tendency for the distribution of each layer's inputs to change during training as the parameters of preceding layers update. By normalizing each mini-batch to zero mean and unit variance before applying two learnable parameters (a scale, gamma, and a shift, beta), the technique keeps activations in a well-behaved range throughout training. In practice, batch normalization layers are inserted between a linear or convolutional layer and its activation function. During training, statistics are computed per mini-batch; at inference time, the layer instead uses running averages of the mean and variance accumulated during training, so predictions do not depend on batch composition. This distinction between train-mode and eval-mode behavior is a common source of bugs when models are deployed incorrectly. The main practical benefits are faster convergence, tolerance for higher learning rates, and a mild regularizing effect from the noise introduced by mini-batch statistics, which can reduce (though not eliminate) the need for other regularization such as dropout. Its main drawback is a dependency on batch size — performance degrades with very small batches, which is why alternatives like layer normalization, group normalization, and instance normalization were developed for settings such as recurrent networks, transformers, and small-batch or per-example inference. Batch normalization remains a default building block in convolutional architectures such as ResNet, while transformer-based models overwhelmingly favor layer normalization instead.
Key Concepts
- Normalizes layer inputs using per-mini-batch mean and variance
- Learnable scale (gamma) and shift (beta) parameters restore representational flexibility after normalization
- Uses running statistics accumulated during training for inference-time normalization
- Allows higher learning rates and generally speeds up convergence
- Provides a mild regularization effect from mini-batch noise
- Placed between a linear/convolutional layer and its activation function
- Performance depends on batch size; degrades with very small batches
- Standard component in CNN architectures like ResNet and VGG variants