L1 Regularization
L1 regularization is a technique that adds a penalty proportional to the sum of the absolute values of a model's weights to its loss function, encouraging sparse solutions where many weights become exactly zero.
Definition
L1 regularization is a technique that adds a penalty proportional to the sum of the absolute values of a model's weights to its loss function, encouraging sparse solutions where many weights become exactly zero.
Overview
L1 regularization, also called Lasso (Least Absolute Shrinkage and Selection Operator) in linear regression contexts, adds a term equal to lambda times the sum of the absolute values of a model's weights to the training objective. The hyperparameter lambda controls the strength of the penalty: a larger lambda pushes more weights toward zero, while lambda equal to zero recovers the unregularized model. The defining property of L1 regularization is that its penalty has a constant gradient magnitude regardless of a weight's size, which — geometrically, because the L1 penalty's contours have sharp corners along the coordinate axes — tends to drive many weights to exactly zero rather than just shrinking them toward zero. This makes L1 regularization useful as a built-in feature selection mechanism: features whose weights are driven to zero are effectively excluded from the model, producing a sparse, more interpretable solution. In practice, L1 regularization is common in linear and logistic regression (as in scikit-learn's Lasso and LogisticRegression with an L1 penalty), and in settings with high-dimensional data where many features are expected to be irrelevant, such as genomics or text classification with bag-of-words features. It is used less often in deep neural networks than L2 regularization or weight decay, because the sparsity it induces on individual weights doesn't map as cleanly onto useful structure in deep architectures — though it does appear in techniques for pruning or sparsifying networks. Elastic Net regularization combines L1 and L2 penalties to get both sparsity and the smoother shrinkage behavior of L2.
Key Concepts
- Adds a penalty proportional to the sum of absolute weight values to the loss function
- Drives many weights to exactly zero, producing sparse models
- Acts as an implicit feature selection mechanism
- Controlled by a regularization strength hyperparameter, lambda
- Known as Lasso regression in the linear regression context
- Improves model interpretability by eliminating irrelevant features
- Less common than L2 regularization in deep neural networks
- Combinable with L2 regularization as Elastic Net