Stacking · machine learning

Also known as: Stacked generalization

On this page 7

Stacking, formally called stacked generalization, is an ensemble method in machine learning that trains a second-level model, the meta-learner, to combine the predictions of several diverse base models into one final prediction. Instead of averaging votes with fixed weights, stacking learns how much to trust each model, and in which situations, from data the base models did not see during training.

Within classic machine learning, stacking sits beside bagging and boosting as the third major family of ensemble learning. It is most valuable when the base models make different kinds of errors, because the meta-learner can exploit those differences to reduce overall error.

How stacked generalization works

Stacking works in two levels: level-0 base models produce predictions, and a level-1 meta-learner uses those predictions as input features. The critical detail is that the meta-learner must be trained on predictions the base models made for examples they were not trained on; otherwise it learns to trust overfitted models.

The standard procedure uses k-fold cross-validation to create these honest predictions. Each base model is trained k times, each time on k-1 folds, and predicts the held-out fold. Concatenating the held-out predictions yields a full set of out-of-fold predictions for every training row, which becomes the training matrix for the meta-learner.

  1. Split the training data into k folds, typically 5 or 10 folds.
  2. For each base model, collect out-of-fold predictions across all folds.
  3. Train the meta-learner on the out-of-fold predictions and the true labels.
  4. Refit each base model on the full training set, or average the k fold models.
  5. At inference time, feed base model predictions into the meta-learner.

The table below summarizes the role of each component in a typical stacking pipeline.

ComponentRoleTypical choice
Base modelsProduce diverse first-level predictionsGradient boosting, random forest, SVM, k-NN
Meta-featuresInputs to the second levelClass probabilities or regression outputs
Meta-learnerLearns the combination ruleRegularized logistic or linear regression
Validation schemePrevents information leakageStratified k-fold cross-validation
Components of a stacking ensemble

Variants of stacking and blending

Several variants of stacking differ in how they generate meta-features and how many levels they use. The choice mainly trades statistical efficiency against implementation simplicity and compute cost.

Blending is the simplest variant: base models are trained on one part of the data and predict a separate holdout set, often 10 to 20 percent of the rows, on which the meta-learner is trained. It is easy to implement and leak-free, but it wastes data, because the meta-learner sees only the holdout rows.

VariantHow meta-features are madeMain trade-off
Classic stackingOut-of-fold predictions from k-fold CVUses all data, costs k fits per model
BlendingPredictions on a single holdout splitSimple but uses less data
Multi-level stackingLevel-1 outputs feed a level-2 learnerSmall gains, high overfitting risk
Feature-weighted stackingMeta-features plus original featuresMore flexible, easier to overfit
RestackingOriginal features passed through to meta-learnerCaptures context where models differ
Common stacking variants

For classification, base models should usually pass predicted probabilities rather than hard class labels, because probabilities carry information about confidence. For multi-class problems, each base model contributes one column per class, so the meta-feature matrix grows as models times classes.

How stacking compares with bagging and boosting

Stacking differs from bagging and boosting because it combines heterogeneous models through a learned rule, whereas the other two usually repeat one algorithm. Bagging reduces variance by averaging models trained on bootstrap samples, as in random forest. Boosting reduces bias by training models sequentially on the errors of earlier ones, as in AdaBoost and gradient boosting.

The following comparison shows where each ensemble family fits in the bias-variance tradeoff.

MethodBase modelsCombination ruleMain effect
BaggingSame algorithm, bootstrap samplesSimple average or voteLower variance
BoostingSame algorithm, sequential fitsWeighted sum of weak learnersLower bias
StackingDifferent algorithms in parallelLearned by a meta-learnerExploits complementary errors
VotingDifferent algorithms in parallelFixed or hand-set weightsModest, stable gains
Stacking versus other ensemble methods

Stacking often uses bagged or boosted models as its base layer, so the methods are complementary rather than competing. A simple weighted average is a special case of stacking in which the meta-learner is a linear model without an intercept.

Designing an effective stacking ensemble

An effective stack depends more on base model diversity than on the number of models. Five strong models with low correlation of errors usually outperform twenty variants of the same gradient boosting configuration, whose predictions are nearly identical.

The meta-learner should be simple and regularized. Ridge regression or logistic regression with an L2 penalty, sometimes with non-negative weights, is the standard choice, because the meta-feature matrix has few columns that are highly correlated. A flexible meta-learner such as a deep tree tends to overfit the meta-features.

The decision table below lists typical design situations and recommended choices.

SituationRecommended choiceReason
Small dataset10-fold CV, linear meta-learnerMaximizes data for both levels
Large dataset, tight computeBlending with a holdout setAvoids k refits per model
Base models highly correlatedReplace some with different algorithmsCorrelated errors add little
Time-ordered dataForward-chaining validation foldsPrevents leakage from the future
Need for interpretabilityNon-negative linear meta-learnerWeights read as model contributions
Design decisions for stacking

Hyperparameter tuning should happen inside the cross-validation folds for base models, or with a separate nested scheme, so that the selection process itself does not leak information into the meta-features.

Limitations and common mistakes in stacking

The most serious mistake in stacking is information leakage: training the meta-learner on in-sample predictions from base models that already saw those rows. The meta-learner then rewards the most overfitted model, and validation scores look excellent while test performance drops.

  • Leaky meta-features: in-sample predictions instead of out-of-fold predictions.
  • Inconsistent folds: different fold splits for different base models distort the comparison.
  • Excessive levels: three or more levels rarely help outside competitions.
  • Operational cost: every base model must be trained, versioned, monitored and served.
  • Reduced interpretability: explaining a prediction requires tracing two levels of models.

Gains from stacking are typically modest, often a fraction of a percent to a few percent on standard metrics, compared with the best single model. Those gains decide machine learning competitions, but in production the added latency and maintenance may outweigh them, so teams should measure the improvement on a true holdout set before adopting a stack.

Put simply

Stacking is like asking several experts for an opinion and then hiring a manager who learns whom to trust. One expert may be excellent with certain cases, another with different ones. The manager studies their past track record and learns how to weigh their answers.

The key rule is that the manager must judge the experts on questions they had not already seen the answers to. Otherwise the manager would trust whoever memorized the most, not whoever actually understands the problem. When the experts think differently, the combined answer is usually better than any single expert.

In practice

A credit risk team at a mid-sized lender predicts the probability that a loan applicant will default within twelve months. It has about 200,000 historical applications with 80 features. The team trains three base models: a gradient boosting model, a random forest and a regularized logistic regression on scaled features.

Using 5-fold stratified cross-validation, the team collects out-of-fold default probabilities from each model and trains a logistic regression meta-learner on those three columns. The meta-learner assigns the largest weight to gradient boosting but also uses the logistic regression, which is better calibrated for low-risk applicants.

On a held-out test set from a later quarter, the stack raises the area under the ROC curve from 0.781 for the best single model to 0.789. The team judges the small but consistent gain worth the extra pipeline complexity, because the improved ranking reduces expected losses across a large loan portfolio.