Ensemble learning

Also known as: ensemble method

On this page 7

Ensemble learning is a family of methods in machine learning that trains several base models and combines their predictions into one output, so that the combined system is more accurate, more stable or better calibrated than any individual member. An ensemble method works because the errors of different models partly cancel out when the models are diverse and each is better than random guessing.

Within artificial intelligence practice, ensembles dominate many tabular prediction tasks and competitions. Random forest, gradient boosting, AdaBoost and stacking are all ensemble techniques, and each has its own page; this entry explains the shared principles behind them.

Why combining models reduces error

Combining models reduces error because independent mistakes average out while the shared correct signal remains. If several classifiers each have an error rate below 50% and their errors are uncorrelated, a majority vote is wrong only when most of them fail on the same example, which becomes increasingly unlikely as the ensemble grows.

The decisive requirement is diversity among members. Ten copies of the same model produce the same mistakes, so averaging them changes nothing. In practice errors are always partly correlated, which is why gains flatten after a moderate number of members rather than growing without limit.

The bias-variance tradeoff gives the formal view. Averaging many high-variance, low-bias models such as deep decision trees mainly reduces variance, while sequentially fitting weak, high-bias learners to the remaining errors mainly reduces bias. The table below summarizes which error component each strategy targets.

StrategyMain effectTypical base learnerTypical risk
Averaging or votingLower varianceDeep, unstable modelsLittle gain if members are similar
Sequential correctionLower biasShallow trees or stumpsOverfitting noisy labels
Learned combinationLower bias and varianceHeterogeneous modelsLeakage into the meta-model
Error components addressed by ensemble strategies

Main families of ensemble methods

Ensemble methods fall into three main families, distinguished by how members are trained and how their outputs are merged. Each family creates diversity in a different way.

  • Bagging (bootstrap aggregating) trains each model on a bootstrap sample of the training data and averages or votes; random forest adds random feature subsets at each split.
  • Boosting trains models one after another, each focusing on the examples or residuals that earlier models handled poorly; AdaBoost reweights examples, while gradient boosting fits the negative gradient of a loss function.
  • Stacking feeds the predictions of diverse base models into a meta-learner that learns how much to trust each one, usually using out-of-fold predictions.

Simpler schemes also exist. Hard voting takes the most frequent class, soft voting averages predicted probabilities, and blending fits the combiner on a single held-out split instead of cross-validated predictions. The following table compares the families on practical properties.

FamilyTraining orderSource of diversityParallelizableTypical examples
BaggingIndependentResampled data and featuresYesRandom forest, extra trees
BoostingSequentialReweighted errors or residualsOnly within each stepAdaBoost, XGBoost, LightGBM
StackingTwo levelsDifferent algorithmsBase level onlyLinear meta-model over trees and SVMs
VotingIndependentDifferent algorithmsYesSoft vote of three classifiers
Comparison of the main ensemble families

Key design parameters of an ensemble

An ensemble is shaped by a small set of design parameters that control the balance between accuracy, cost and overfitting. Most of them are tuned with cross-validation as part of ordinary hyperparameter tuning.

ParameterTypical rangeEffect when increased
Number of members50-1,000 trees in bagging; 100-5,000 rounds in boostingBetter accuracy until a plateau; longer inference
Learning rate0.01-0.3 in boostingFaster fitting; higher overfitting risk
Base learner depth1-8 in boosting; unrestricted in baggingMore capacity per member; less diversity
Row subsampling50-100% of examplesLess correlation; more noise per member
Feature subsamplingAbout the square root of features to 100%More diversity; weaker individual members
Ensemble parameters and what they affect

In bagging, adding members rarely causes overfitting; it only costs computation. In boosting, the number of rounds and the learning rate interact strongly, so practitioners use a small learning rate with early stopping on a validation set rather than a fixed round count.

Choosing an ensemble for a problem

The right ensemble depends on data type, noise level and deployment constraints. For structured tabular data, gradient-boosted trees are the usual first choice; for unstructured data such as images or text, ensembles of neural networks are effective but expensive.

SituationRecommended approachReason
Tabular data, need strong baseline fastRandom forestRobust defaults, little tuning
Tabular data, maximum accuracyGradient boosting with early stoppingLowest bias on structured features
Several strong but different modelsStacking or soft votingExploits complementary errors
Many noisy or mislabeled examplesBagging over boostingBoosting chases noisy points
Strict latency or memory limitSmall ensemble or distillationLarge ensembles are slow to serve
Decision guide for selecting an ensemble approach

Ensembles also provide uncertainty estimates. Disagreement among members, for example the variance of predictions across a deep ensemble, is a practical signal that an input lies outside the training distribution or near a decision boundary.

Limitations and common mistakes

Ensembles trade interpretability and efficiency for accuracy. A forest of 500 trees cannot be read like a single decision tree, so explanation relies on feature importance, SHAP values or partial dependence plots, each with its own caveats.

  • Data leakage in stacking: fitting the meta-learner on in-sample base predictions makes it trust overfitted members; use out-of-fold predictions.
  • Correlated members: combining five variants of the same gradient boosting model adds cost but little accuracy.
  • Ignoring calibration: averaged probabilities from boosted trees are often poorly calibrated and may need Platt scaling or isotonic regression.
  • Serving cost: inference time and memory grow roughly linearly with the number of members, which matters in real-time systems.
  • Overconfidence in leaderboard gains: tiny improvements from huge ensembles often do not survive concept drift in production.

A common technique to keep the benefit while cutting cost is knowledge distillation, in which a single compact model is trained to imitate the ensemble outputs and usually retains most of its accuracy.

A plain-language summary

Ensemble learning means asking many models instead of one and combining their answers. It works much like asking several experienced doctors for an opinion: each may miss something, but they rarely all miss the same thing, so the combined judgment is more reliable.

The trick only works when the models think differently. Some ensembles train many models on slightly different samples of the data and average them. Others train models one after another, each fixing the mistakes of the last. The price is that the result is slower and harder to explain than a single model.

An example from practice

A regional bank builds a model to predict which credit card transactions are fraudulent, using about 2 million labeled transactions with 60 features such as amount, merchant category and time since the previous purchase. A single decision tree reaches about 0.86 area under the ROC curve on held-out data, but its predictions change noticeably whenever the training sample changes.

The data science team first trains a random forest of 400 trees, which raises the score to about 0.93 and stabilizes the results. They then train a gradient boosting model with a learning rate of 0.05 and early stopping, and combine both with a logistic regression meta-learner fitted on out-of-fold predictions. The stacked ensemble reaches about 0.95 and catches noticeably more fraud at the same false-alarm rate.

Because the stacked model is too slow for the card authorization path, the team distills it into one compact boosted model. The production model keeps most of the gain while answering within the required few milliseconds.