Stacking, formally called stacked generalization, is an ensemble method in machine learning that trains a second-level model, the meta-learner, to combine the predictions of several diverse base models into one final prediction. Instead of averaging votes with fixed weights, stacking learns how much to trust each model, and in which situations, from data the base models did not see during training.
Within classic machine learning, stacking sits beside bagging and boosting as the third major family of ensemble learning. It is most valuable when the base models make different kinds of errors, because the meta-learner can exploit those differences to reduce overall error.
How stacked generalization works
Stacking works in two levels: level-0 base models produce predictions, and a level-1 meta-learner uses those predictions as input features. The critical detail is that the meta-learner must be trained on predictions the base models made for examples they were not trained on; otherwise it learns to trust overfitted models.
The standard procedure uses k-fold cross-validation to create these honest predictions. Each base model is trained k times, each time on k-1 folds, and predicts the held-out fold. Concatenating the held-out predictions yields a full set of out-of-fold predictions for every training row, which becomes the training matrix for the meta-learner.
- Split the training data into k folds, typically 5 or 10 folds.
- For each base model, collect out-of-fold predictions across all folds.
- Train the meta-learner on the out-of-fold predictions and the true labels.
- Refit each base model on the full training set, or average the k fold models.
- At inference time, feed base model predictions into the meta-learner.
The table below summarizes the role of each component in a typical stacking pipeline.
| Component | Role | Typical choice |
|---|---|---|
| Base models | Produce diverse first-level predictions | Gradient boosting, random forest, SVM, k-NN |
| Meta-features | Inputs to the second level | Class probabilities or regression outputs |
| Meta-learner | Learns the combination rule | Regularized logistic or linear regression |
| Validation scheme | Prevents information leakage | Stratified k-fold cross-validation |
Variants of stacking and blending
Several variants of stacking differ in how they generate meta-features and how many levels they use. The choice mainly trades statistical efficiency against implementation simplicity and compute cost.
Blending is the simplest variant: base models are trained on one part of the data and predict a separate holdout set, often 10 to 20 percent of the rows, on which the meta-learner is trained. It is easy to implement and leak-free, but it wastes data, because the meta-learner sees only the holdout rows.
| Variant | How meta-features are made | Main trade-off |
|---|---|---|
| Classic stacking | Out-of-fold predictions from k-fold CV | Uses all data, costs k fits per model |
| Blending | Predictions on a single holdout split | Simple but uses less data |
| Multi-level stacking | Level-1 outputs feed a level-2 learner | Small gains, high overfitting risk |
| Feature-weighted stacking | Meta-features plus original features | More flexible, easier to overfit |
| Restacking | Original features passed through to meta-learner | Captures context where models differ |
For classification, base models should usually pass predicted probabilities rather than hard class labels, because probabilities carry information about confidence. For multi-class problems, each base model contributes one column per class, so the meta-feature matrix grows as models times classes.
How stacking compares with bagging and boosting
Stacking differs from bagging and boosting because it combines heterogeneous models through a learned rule, whereas the other two usually repeat one algorithm. Bagging reduces variance by averaging models trained on bootstrap samples, as in random forest. Boosting reduces bias by training models sequentially on the errors of earlier ones, as in AdaBoost and gradient boosting.
The following comparison shows where each ensemble family fits in the bias-variance tradeoff.
| Method | Base models | Combination rule | Main effect |
|---|---|---|---|
| Bagging | Same algorithm, bootstrap samples | Simple average or vote | Lower variance |
| Boosting | Same algorithm, sequential fits | Weighted sum of weak learners | Lower bias |
| Stacking | Different algorithms in parallel | Learned by a meta-learner | Exploits complementary errors |
| Voting | Different algorithms in parallel | Fixed or hand-set weights | Modest, stable gains |
Stacking often uses bagged or boosted models as its base layer, so the methods are complementary rather than competing. A simple weighted average is a special case of stacking in which the meta-learner is a linear model without an intercept.
Designing an effective stacking ensemble
An effective stack depends more on base model diversity than on the number of models. Five strong models with low correlation of errors usually outperform twenty variants of the same gradient boosting configuration, whose predictions are nearly identical.
The meta-learner should be simple and regularized. Ridge regression or logistic regression with an L2 penalty, sometimes with non-negative weights, is the standard choice, because the meta-feature matrix has few columns that are highly correlated. A flexible meta-learner such as a deep tree tends to overfit the meta-features.
The decision table below lists typical design situations and recommended choices.
| Situation | Recommended choice | Reason |
|---|---|---|
| Small dataset | 10-fold CV, linear meta-learner | Maximizes data for both levels |
| Large dataset, tight compute | Blending with a holdout set | Avoids k refits per model |
| Base models highly correlated | Replace some with different algorithms | Correlated errors add little |
| Time-ordered data | Forward-chaining validation folds | Prevents leakage from the future |
| Need for interpretability | Non-negative linear meta-learner | Weights read as model contributions |
Hyperparameter tuning should happen inside the cross-validation folds for base models, or with a separate nested scheme, so that the selection process itself does not leak information into the meta-features.
Limitations and common mistakes in stacking
The most serious mistake in stacking is information leakage: training the meta-learner on in-sample predictions from base models that already saw those rows. The meta-learner then rewards the most overfitted model, and validation scores look excellent while test performance drops.
- Leaky meta-features: in-sample predictions instead of out-of-fold predictions.
- Inconsistent folds: different fold splits for different base models distort the comparison.
- Excessive levels: three or more levels rarely help outside competitions.
- Operational cost: every base model must be trained, versioned, monitored and served.
- Reduced interpretability: explaining a prediction requires tracing two levels of models.
Gains from stacking are typically modest, often a fraction of a percent to a few percent on standard metrics, compared with the best single model. Those gains decide machine learning competitions, but in production the added latency and maintenance may outweigh them, so teams should measure the improvement on a true holdout set before adopting a stack.
Put simply
Stacking is like asking several experts for an opinion and then hiring a manager who learns whom to trust. One expert may be excellent with certain cases, another with different ones. The manager studies their past track record and learns how to weigh their answers.
The key rule is that the manager must judge the experts on questions they had not already seen the answers to. Otherwise the manager would trust whoever memorized the most, not whoever actually understands the problem. When the experts think differently, the combined answer is usually better than any single expert.
In practice
A credit risk team at a mid-sized lender predicts the probability that a loan applicant will default within twelve months. It has about 200,000 historical applications with 80 features. The team trains three base models: a gradient boosting model, a random forest and a regularized logistic regression on scaled features.
Using 5-fold stratified cross-validation, the team collects out-of-fold default probabilities from each model and trains a logistic regression meta-learner on those three columns. The meta-learner assigns the largest weight to gradient boosting but also uses the logistic regression, which is better calibrated for low-risk applicants.
On a held-out test set from a later quarter, the stack raises the area under the ROC curve from 0.781 for the best single model to 0.789. The team judges the small but consistent gain worth the extra pipeline complexity, because the improved ranking reduces expected losses across a large loan portfolio.