What Is Ensemble Learning?

Ensemble learning, also called ensemble methods, is a family of machine learning techniques in which several models are trained on the same predictive task and their outputs are combined into a single decision. The combination is usually a vote for classification problems or an average for regression problems, sometimes weighted by each model's estimated reliability. The premise is statistical: individual models make errors on different examples, and if those errors are sufficiently uncorrelated, aggregation cancels part of them and leaves a predictor more accurate than any of its members.

The approach draws on both statistics and computational learning theory. Work in the late 1980s and early 1990s established that a collection of weak learners, each performing only slightly better than chance, can be combined into an arbitrarily accurate strong learner, a result that gave boosting its theoretical footing. The bias-variance decomposition supplies the complementary intuition: averaging many high-variance, low-bias models reduces variance without adding much bias, while sequentially correcting a high-bias model reduces bias. Most practical ensemble algorithms are variations on one of those two strategies.

Bagging and Randomized Ensembles

Bootstrap aggregating, or bagging, trains each member on a bootstrap resample of the training data and averages the results. Because the resamples differ, unstable learners such as decision trees produce noticeably different fits, and averaging them suppresses the variance that makes a single deep tree unreliable. Random forests extend the idea by also sampling a random subset of features at each split, which decorrelates the trees further and improves the ensemble beyond what resampling alone achieves. Leo Breiman's 2001 paper introducing random forests established the generalization error bound that explains why adding trees does not cause overfitting and why the out-of-bag samples give a free estimate of test error.

Boosting and Stacking

Boosting builds its members sequentially, with each new learner concentrating on the examples its predecessors got wrong. AdaBoost does this by reweighting misclassified training points; gradient boosting recasts the procedure as functional gradient descent on a differentiable loss, which allows arbitrary objectives and underlies widely used implementations such as XGBoost, LightGBM, and CatBoost. Stacking takes a different route: it trains a set of heterogeneous base models, then fits a second-level meta-learner on their cross-validated predictions to learn how best to blend them. Voting classifiers are the simplest case, combining independently trained models with a fixed hard or soft vote. The scikit-learn documentation on ensembles sets out the algorithmic differences among bagging, boosting, voting, and stacking in implementation terms.

Diversity, Cost, and Interpretability

The performance of an ensemble depends on member diversity as much as on member accuracy, and a good deal of research addresses how to generate and measure that diversity through data resampling, feature subspaces, varied hyperparameters, or different model families altogether. Ensembles are also a standard response to class imbalance, where resampling schemes are combined with bagging or boosting so that rare classes are not swamped, an area surveyed in a review of ensemble and data augmentation methods for imbalanced problems. The costs are real: training and serving many models multiplies compute and memory requirements, and an ensemble of hundreds of trees is far harder to explain to a regulator or clinician than a single model. Model compression and distillation techniques exist partly to recover some of that lost efficiency and transparency.

Applications

Ensemble learning has applications in a wide range of fields, including:

• Credit scoring and fraud detection, where gradient boosted trees are a common production standard
• Medical diagnosis and clinical risk prediction from tabular patient records
• Remote sensing land cover classification using random forest classifiers
• Anomaly and intrusion detection in network security
• Bioinformatics tasks such as gene expression classification and protein function prediction
• Competitive predictive modeling, where stacked ensembles routinely top benchmark leaderboards