Regularization: Ridge and Lasso
Ordinary least squares regression minimizes only the sum of squared residuals, which means it will happily assign large coefficients to features if doing so reduces training error even slightly — a behavior that becomes dangerous when features are numerous, correlated, or the sample size is small. Regularization adds a penalty term to the loss function that discourages large coefficients, trading a small increase in bias for a often much larger reduction in variance. Ridge regression (L2 penalty) and lasso regression (L1 penalty) are the two canonical forms, and they behave differently enough that the choice between them is a meaningful modeling decision rather than a formality.
Cricket analogy: A batting average model that lets one small sample size (say, three innings against a weak attack) inflate a player's projected rating is behaving like unregularized OLS; regularization is like a selector penalizing wildly optimistic projections, trading a bit of caution for much more stable rankings.
Ridge Regression (L2 Penalty)
Ridge regression minimizes the residual sum of squares plus alpha times the sum of squared coefficients: RSS + alpha * sum(bi^2). As alpha grows, coefficients shrink smoothly toward zero but essentially never reach exactly zero. This makes ridge well suited to situations with many correlated predictors, since it spreads weight across correlated features rather than arbitrarily favoring one over another, stabilizing the solution and reducing variance. Because it does not zero out coefficients, ridge is a shrinkage method, not a feature-selection method.
Cricket analogy: Ridge regression is like a selection committee that gradually discounts every player's inflated stats from flat pitches without ever ruling anyone out entirely, spreading skepticism evenly across correlated stats like average and strike rate rather than picking one to trust.
Lasso Regression (L1 Penalty)
Lasso regression instead penalizes the sum of the absolute values of the coefficients: RSS + alpha * sum(|bi|). Geometrically, the L1 penalty's constraint region has corners on the coordinate axes, and the least-squares solution tends to land exactly on those corners as alpha increases — driving some coefficients to exactly zero. This gives lasso a built-in feature-selection effect: it produces sparse models where irrelevant or redundant features are dropped entirely, which aids interpretability but can behave erratically when predictors are highly correlated, arbitrarily keeping one and zeroing another.
Cricket analogy: Lasso is like a selection committee that, as scrutiny increases, drops entirely unconvincing players from contention (zeroing their chances) while keeping only the clearly deserving ones, though it can arbitrarily favor one of two similar all-rounders over the other.
import numpy as np
from sklearn.linear_model import Ridge, Lasso, RidgeCV, LassoCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(1)
n_samples, n_features = 100, 20
X = rng.normal(size=(n_samples, n_features))
true_coef = np.zeros(n_features)
true_coef[:5] = [3, -2, 1.5, 0, 4] # only a few features actually matter
y = X @ true_coef + rng.normal(0, 1, n_samples)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=1)
# Ridge: shrinks all coefficients, none exactly zero
ridge = make_pipeline(StandardScaler(), RidgeCV(alphas=np.logspace(-3, 3, 20)))
ridge.fit(X_train, y_train)
# Lasso: drives many coefficients to exactly zero
lasso = make_pipeline(StandardScaler(), LassoCV(alphas=np.logspace(-3, 1, 20), random_state=1))
lasso.fit(X_train, y_train)
ridge_coef = ridge.named_steps['ridgecv'].coef_
lasso_coef = lasso.named_steps['lassocv'].coef_
print("ridge nonzero coefs:", np.sum(np.abs(ridge_coef) > 1e-6))
print("lasso nonzero coefs:", np.sum(np.abs(lasso_coef) > 1e-6))
print("ridge test R^2:", ridge.score(X_test, y_test))
print("lasso test R^2:", lasso.score(X_test, y_test))A helpful mental picture: ridge is like turning down the volume on every feature a little, while lasso is like muting some channels entirely and leaving the rest at normal volume. Elastic Net (a mix of L1 and L2 penalties) is a common middle ground that keeps some grouping behavior of ridge while still allowing sparsity from lasso.
Regularization penalizes the raw magnitude of coefficients, so features must be on comparable scales (typically standardized to zero mean, unit variance) before fitting ridge or lasso — otherwise a feature measured in large units will be penalized more heavily than an equally important feature measured in small units, purely due to scale.
Choosing the Regularization Strength
Alpha is a hyperparameter, not something learned from the training loss directly, since larger alpha always increases training error. It is selected via cross-validation, typically using scikit-learn's RidgeCV or LassoCV, which sweep a grid of alpha values and pick the one minimizing cross-validated error. As alpha increases from zero, models move along the bias-variance spectrum: at alpha=0 the model is ordinary least squares (low bias, possibly high variance); as alpha grows, coefficients shrink further, bias increases, and variance decreases, until at very large alpha nearly all coefficients collapse toward zero and the model underfits.
Cricket analogy: How harshly a selection panel discounts small-sample stats (alpha) isn't decided by gut feel, it's tuned by checking which discount level best predicts performance in held-out series, similar to how RidgeCV or LassoCV sweep values to find the best cross-validated setting.
- Ridge (L2) shrinks coefficients smoothly toward zero without eliminating them; good for correlated features.
- Lasso (L1) can drive coefficients to exactly zero, performing implicit feature selection and producing sparse models.
- Both penalties trade increased bias for reduced variance, controlled by the hyperparameter alpha (lambda).
- Feature scaling is required before regularization, since penalties are sensitive to coefficient magnitude, which depends on feature scale.
- Alpha should be tuned via cross-validation (e.g., RidgeCV, LassoCV), not chosen by minimizing training error.
- Elastic Net combines L1 and L2 penalties to balance sparsity and stability under correlated features.
Practice what you learned
1. What is the key structural difference between the penalty terms in ridge and lasso regression?
2. Why can lasso regression produce sparse models while ridge regression typically cannot?
3. Why must features generally be scaled before applying ridge or lasso regression?
4. As the regularization parameter alpha increases from zero toward a very large value, what generally happens?
5. In a dataset with several highly correlated features, how do ridge and lasso typically differ in behavior?
Was this page helpful?
You May Also Like
Linear Regression Explained
Understand how linear regression fits a straight-line relationship between features and a continuous target, and how it is trained and evaluated.
Polynomial and Multiple Regression
Extends simple linear regression to multiple predictors and curved relationships, showing how feature expansion lets a linear model fit non-linear patterns.
Overfitting and Underfitting
The two failure modes of model fitting — memorizing noise in training data versus failing to capture real patterns — and the diagnostic and mitigation techniques for each.
Bias-Variance Tradeoff
A foundational concept explaining how model error decomposes into bias, variance, and irreducible noise, and how balancing them guides model complexity choices.