Ensemble Learning
Ensemble learning is a machine learning approach that combines the predictions of multiple individual models to produce a single, typically more accurate and robust prediction than any one model alone.
Definition
Ensemble learning is a machine learning approach that combines the predictions of multiple individual models to produce a single, typically more accurate and robust prediction than any one model alone.
Overview
The core intuition behind ensemble learning is that different models make different errors, and if those errors are sufficiently uncorrelated, combining the models' predictions — by averaging, voting, or weighting — cancels out some of that error, reducing variance, bias, or both compared to any single constituent model. This idea underlies some of the most reliable and widely deployed techniques in applied machine learning, particularly for structured/tabular data. Bagging (bootstrap aggregating) trains many models independently and in parallel on different random subsamples of the training data, then averages or votes on their predictions; Random Forest is the canonical bagging algorithm, building many decorrelated decision trees. Boosting instead trains models sequentially, with each new model focused on correcting the errors of the ensemble built so far; Gradient Boosting and its highly optimized implementation XGBoost are dominant examples, consistently among the strongest performers on tabular data competitions. Stacking takes a different approach, training a meta-model to learn how to best combine the outputs of several diverse base models, which may themselves be of very different types. Ensemble methods generally trade some interpretability and inference speed for improved predictive accuracy and robustness to overfitting, since averaging across many models smooths out idiosyncratic mistakes any single model might make on noisy data. They remain a default choice for structured data problems even in the deep learning era, where competitions on tabular data are still frequently won by gradient-boosted tree ensembles rather than neural networks. Ensembling is also used at inference time with deep learning models themselves — for example, averaging predictions from multiple neural network checkpoints or architectures — to squeeze out additional accuracy and calibration improvements.
Key Concepts
- Combines multiple models to reduce variance, bias, or both versus any single model
- Bagging trains models in parallel on bootstrap-resampled data (e.g. Random Forest)
- Boosting trains models sequentially, each correcting prior errors (e.g. Gradient Boosting, XGBoost)
- Stacking uses a meta-model to combine diverse base model outputs
- Reduces overfitting risk by averaging out individual models' idiosyncratic errors
- Dominant approach for structured/tabular data prediction tasks
- Trades some interpretability and inference speed for accuracy and robustness
- Applicable to both classical ML models and deep neural network checkpoints