Ensemble learning is a family of methods in machine learning that trains several base models and combines their predictions into one output, so that the combined system is more accurate, more stable or better calibrated than any individual member. An ensemble method works because the errors of different models partly cancel out when the models are diverse and each is better than random guessing.
Within artificial intelligence practice, ensembles dominate many tabular prediction tasks and competitions. Random forest, gradient boosting, AdaBoost and stacking are all ensemble techniques, and each has its own page; this entry explains the shared principles behind them.
Why combining models reduces error
Combining models reduces error because independent mistakes average out while the shared correct signal remains. If several classifiers each have an error rate below 50% and their errors are uncorrelated, a majority vote is wrong only when most of them fail on the same example, which becomes increasingly unlikely as the ensemble grows.
The decisive requirement is diversity among members. Ten copies of the same model produce the same mistakes, so averaging them changes nothing. In practice errors are always partly correlated, which is why gains flatten after a moderate number of members rather than growing without limit.
The bias-variance tradeoff gives the formal view. Averaging many high-variance, low-bias models such as deep decision trees mainly reduces variance, while sequentially fitting weak, high-bias learners to the remaining errors mainly reduces bias. The table below summarizes which error component each strategy targets.
| Strategy | Main effect | Typical base learner | Typical risk |
|---|---|---|---|
| Averaging or voting | Lower variance | Deep, unstable models | Little gain if members are similar |
| Sequential correction | Lower bias | Shallow trees or stumps | Overfitting noisy labels |
| Learned combination | Lower bias and variance | Heterogeneous models | Leakage into the meta-model |
Main families of ensemble methods
Ensemble methods fall into three main families, distinguished by how members are trained and how their outputs are merged. Each family creates diversity in a different way.
- Bagging (bootstrap aggregating) trains each model on a bootstrap sample of the training data and averages or votes; random forest adds random feature subsets at each split.
- Boosting trains models one after another, each focusing on the examples or residuals that earlier models handled poorly; AdaBoost reweights examples, while gradient boosting fits the negative gradient of a loss function.
- Stacking feeds the predictions of diverse base models into a meta-learner that learns how much to trust each one, usually using out-of-fold predictions.
Simpler schemes also exist. Hard voting takes the most frequent class, soft voting averages predicted probabilities, and blending fits the combiner on a single held-out split instead of cross-validated predictions. The following table compares the families on practical properties.
| Family | Training order | Source of diversity | Parallelizable | Typical examples |
|---|---|---|---|---|
| Bagging | Independent | Resampled data and features | Yes | Random forest, extra trees |
| Boosting | Sequential | Reweighted errors or residuals | Only within each step | AdaBoost, XGBoost, LightGBM |
| Stacking | Two levels | Different algorithms | Base level only | Linear meta-model over trees and SVMs |
| Voting | Independent | Different algorithms | Yes | Soft vote of three classifiers |
Key design parameters of an ensemble
An ensemble is shaped by a small set of design parameters that control the balance between accuracy, cost and overfitting. Most of them are tuned with cross-validation as part of ordinary hyperparameter tuning.
| Parameter | Typical range | Effect when increased |
|---|---|---|
| Number of members | 50-1,000 trees in bagging; 100-5,000 rounds in boosting | Better accuracy until a plateau; longer inference |
| Learning rate | 0.01-0.3 in boosting | Faster fitting; higher overfitting risk |
| Base learner depth | 1-8 in boosting; unrestricted in bagging | More capacity per member; less diversity |
| Row subsampling | 50-100% of examples | Less correlation; more noise per member |
| Feature subsampling | About the square root of features to 100% | More diversity; weaker individual members |
In bagging, adding members rarely causes overfitting; it only costs computation. In boosting, the number of rounds and the learning rate interact strongly, so practitioners use a small learning rate with early stopping on a validation set rather than a fixed round count.
Choosing an ensemble for a problem
The right ensemble depends on data type, noise level and deployment constraints. For structured tabular data, gradient-boosted trees are the usual first choice; for unstructured data such as images or text, ensembles of neural networks are effective but expensive.
| Situation | Recommended approach | Reason |
|---|---|---|
| Tabular data, need strong baseline fast | Random forest | Robust defaults, little tuning |
| Tabular data, maximum accuracy | Gradient boosting with early stopping | Lowest bias on structured features |
| Several strong but different models | Stacking or soft voting | Exploits complementary errors |
| Many noisy or mislabeled examples | Bagging over boosting | Boosting chases noisy points |
| Strict latency or memory limit | Small ensemble or distillation | Large ensembles are slow to serve |
Ensembles also provide uncertainty estimates. Disagreement among members, for example the variance of predictions across a deep ensemble, is a practical signal that an input lies outside the training distribution or near a decision boundary.
Limitations and common mistakes
Ensembles trade interpretability and efficiency for accuracy. A forest of 500 trees cannot be read like a single decision tree, so explanation relies on feature importance, SHAP values or partial dependence plots, each with its own caveats.
- Data leakage in stacking: fitting the meta-learner on in-sample base predictions makes it trust overfitted members; use out-of-fold predictions.
- Correlated members: combining five variants of the same gradient boosting model adds cost but little accuracy.
- Ignoring calibration: averaged probabilities from boosted trees are often poorly calibrated and may need Platt scaling or isotonic regression.
- Serving cost: inference time and memory grow roughly linearly with the number of members, which matters in real-time systems.
- Overconfidence in leaderboard gains: tiny improvements from huge ensembles often do not survive concept drift in production.
A common technique to keep the benefit while cutting cost is knowledge distillation, in which a single compact model is trained to imitate the ensemble outputs and usually retains most of its accuracy.
A plain-language summary
Ensemble learning means asking many models instead of one and combining their answers. It works much like asking several experienced doctors for an opinion: each may miss something, but they rarely all miss the same thing, so the combined judgment is more reliable.
The trick only works when the models think differently. Some ensembles train many models on slightly different samples of the data and average them. Others train models one after another, each fixing the mistakes of the last. The price is that the result is slower and harder to explain than a single model.
An example from practice
A regional bank builds a model to predict which credit card transactions are fraudulent, using about 2 million labeled transactions with 60 features such as amount, merchant category and time since the previous purchase. A single decision tree reaches about 0.86 area under the ROC curve on held-out data, but its predictions change noticeably whenever the training sample changes.
The data science team first trains a random forest of 400 trees, which raises the score to about 0.93 and stabilizes the results. They then train a gradient boosting model with a learning rate of 0.05 and early stopping, and combine both with a logistic regression meta-learner fitted on out-of-fold predictions. The stacked ensemble reaches about 0.95 and catches noticeably more fraud at the same false-alarm rate.
Because the stacked model is too slow for the card authorization path, the team distills it into one compact boosted model. The production model keeps most of the gain while answering within the required few milliseconds.