L2 Regularization
L2 regularization is a technique that adds a penalty proportional to the sum of the squared values of a model's weights to its loss function, discouraging large weights and reducing overfitting.
Definition
L2 regularization is a technique that adds a penalty proportional to the sum of the squared values of a model's weights to its loss function, discouraging large weights and reducing overfitting.
Overview
L2 regularization, known as Ridge regression in the linear regression context and commonly implemented as "weight decay" in neural network training, adds a term equal to lambda times the sum of squared weights to the training objective. Because the penalty grows quadratically with weight magnitude, it strongly discourages very large individual weights while barely penalizing small ones, producing smooth, evenly distributed shrinkage across all weights rather than driving any of them to exactly zero, which is the key difference from L1 regularization. The intuition behind L2 regularization is that models with very large weights tend to be more sensitive to small changes in input, which is a hallmark of overfitting to training data noise. By constraining weight magnitudes, L2 regularization encourages simpler, smoother decision boundaries or functions that generalize better to unseen data. In gradient descent, L2 regularization has an elegant interpretation as "weight decay": at each update step, weights are multiplied by a factor slightly less than one before the gradient step is applied, continuously pulling them toward zero. L2 regularization is ubiquitous across machine learning — in linear and logistic regression, support vector machines (where it appears as the margin-maximization term), and deep neural networks, where it is typically applied via an optimizer's weight_decay parameter (as in PyTorch or the AdamW variant of the Adam optimizer). It is frequently combined with other regularizers such as dropout and data augmentation, and unlike L1 regularization, it does not perform feature selection since it rarely produces exactly-zero weights.
Key Concepts
- Adds a penalty proportional to the sum of squared weight values to the loss function
- Shrinks weights smoothly and uniformly rather than driving them to zero
- Equivalent to weight decay in gradient-based optimization
- Known as Ridge regression in the linear regression context
- Discourages overly large weights that make models sensitive to input noise
- Commonly implemented via an optimizer's weight_decay hyperparameter
- Widely used in deep learning, often via AdamW
- Does not perform feature selection, unlike L1 regularization