Bagging

Also known as: Bootstrap aggregating

On this page 7

Bagging, short for bootstrap aggregating, is an ensemble method in machine learning that trains many copies of the same base model on random bootstrap samples of the training data and combines their predictions by averaging or majority voting. Its purpose is to reduce variance: individual models that react strongly to small changes in the data become far more stable once their errors are averaged out.

Leo Breiman introduced the method in 1996, and it remains one of the most reliable tools in artificial intelligence practice for tabular data. Bagging is the foundation of the random forest, it is the natural counterpart to boosting within ensemble learning, and it offers a free performance estimate through out-of-bag error.

How bootstrap aggregating works

Bagging works in two stages: bootstrap resampling creates many slightly different training sets, and aggregation merges the models trained on them. Each bootstrap sample is drawn with replacement and usually has the same size as the original dataset, so some rows appear several times while others are missing entirely.

Because every row has a chance of 1 – 1/n of being skipped on each draw, the probability that it is absent from a sample of size n approaches about 36.8 percent. Each base model therefore sees roughly 63.2 percent of the unique rows, and the remaining rows form its out-of-bag set.

StepWhat happensTypical setting
ResampleDraw B bootstrap samples with replacementB from 50 to 500
TrainFit one base learner per sample independentlyUnpruned decision trees
PredictEach learner scores a new inputRuns in parallel
AggregateAverage outputs or take a majority voteMean or soft voting
ValidateScore each row only with models that never saw itOut-of-bag estimate
The bagging procedure step by step

For regression, the ensemble prediction is the arithmetic mean of the base predictions. For classification, practitioners use either hard voting, where the most frequent class wins, or soft voting, where predicted probabilities are averaged; soft voting usually yields better calibrated probabilities and slightly higher accuracy.

Why bagging reduces variance

Bagging reduces variance because averaging several noisy estimates cancels part of their independent errors. If B models each have variance sigma squared and a pairwise correlation rho, the variance of their average equals rho times sigma squared plus (1 – rho) times sigma squared divided by B. As B grows, the second term vanishes, but the first term remains: correlation between models sets the floor.

This formula explains the core of the bias-variance tradeoff for bagging. The method leaves bias almost unchanged, since every model has the same structure, while it can cut variance sharply when the base learners are unstable and weakly correlated. Random forests push correlation lower by also sampling a random subset of features at every split.

Base learnerStabilityBenefit from bagging
Deep decision treeVery unstableLarge accuracy gain
Neural networkUnstable across seedsModerate gain, high cost
k-nearest neighborsStable for large kSmall or none
Linear regressionStableNegligible
Naive Bayes classifierVery stableNegligible
Which base learners benefit from bagging

A practical rule follows: bagging helps most when the base model has low bias and high variance. Applying it to a stable, high-bias model such as logistic regression adds computation without fixing the real problem, which is underfitting rather than instability.

Bagging compared with boosting and stacking

Bagging differs from the other major ensemble strategies in how models are trained and what kind of error they attack. Boosting trains models sequentially, each one focusing on the mistakes of its predecessors, and mainly reduces bias. Stacking trains a meta-model that learns how to weigh the outputs of diverse base models.

PropertyBaggingBoostingStacking
Training orderParallelSequentialTwo levels
Main error reducedVarianceBias and varianceModel selection error
Typical base learnerDeep treesShallow treesDiverse model families
Sensitivity to label noiseLowHigherModerate
Overfitting risk with more modelsVery lowGrows without early stoppingDepends on meta-model
Representative algorithmsRandom forest, extra treesAdaBoost, gradient boostingBlending, super learner
Bagging, boosting and stacking side by side

A distinctive property of bagging is that adding more models does not cause overfitting: the error curve flattens as B increases instead of turning upward. Boosting, by contrast, needs a learning rate and early stopping to control its capacity. On noisy data with mislabeled examples, bagging is usually the more robust choice.

Variants and configuration choices

Several variants change what is resampled, and each trades diversity against individual model strength. Resampling rows, columns, or both alters how decorrelated the ensemble becomes and how much data each model sees.

VariantWhat is sampledWhen it fits
Classic baggingRows with replacementGeneral purpose variance reduction
PastingRows without replacementVery large datasets
Random subspacesFeatures onlyMany features, few rows
Random patchesRows and featuresWide, large datasets
Random forestRows plus features per splitDefault for tabular data
Common variants of bagging

The most important hyperparameters are the number of estimators, the sample fraction and the feature fraction. In hyperparameter tuning, the number of estimators is rarely worth searching: it should simply be large enough for the out-of-bag error to plateau, typically between 100 and 500. Sample fractions below 1.0 speed up training and increase diversity, which can help on large datasets.

  • Set the out-of-bag score option to obtain validation accuracy without a separate holdout split.
  • Use stratified or balanced bootstrap samples when class imbalance is severe, so minority cases appear in every sample.
  • Fix the random seed for reproducibility, because results otherwise change slightly between runs.

Limitations and common mistakes

Bagging is not a universal accuracy booster, and its weaknesses matter in production. The ensemble cannot reduce bias, it multiplies memory and inference cost by the number of models, and it replaces one interpretable tree with hundreds, which weakens interpretability.

SymptomLikely causeRemedy
No gain over a single modelStable or high-bias base learnerUse deeper trees or switch to boosting
Gain stalls earlyHighly correlated modelsAdd feature subsampling
Slow predictionsToo many large treesLimit depth or prune the ensemble
Optimistic validation scoreDuplicated rows leak across samplesDeduplicate or group rows before resampling
Poor minority class recallBootstrap samples miss rare classesUse balanced bagging
Typical problems and how to address them

A frequent misconception is that bagging creates new information. It only reuses the existing sample, so it cannot compensate for biased data collection or missing features. Another pitfall concerns time series: ordinary bootstrap sampling breaks temporal order, so practitioners use block bootstrap methods instead.

A plain-language summary

Imagine asking one person to guess how many beans are in a jar. The guess may be far off. Now ask two hundred people, each of whom has looked at the jar from a slightly different angle, and take the average. The individual mistakes partly cancel out, and the average is usually much closer to the truth.

Bagging does the same with computer models. It shows each model a slightly different, randomly shuffled version of the same data and then lets them vote. The crowd of models is steadier than any single model, especially when each one on its own tends to overreact to small details.

An example from practice

A credit risk team at a mid-sized lender predicts whether small business loans will default, using about 40,000 historical applications and 60 features. A single deep decision tree reaches an area under the ROC curve of 0.74 on a holdout set, but its scores swing noticeably whenever the training data is refreshed each month.

The team replaces it with a bagged ensemble of 300 trees, using balanced bootstrap samples because only 6 percent of loans default. The out-of-bag estimate stabilizes after about 200 trees, and holdout performance rises to an area under the curve of 0.81. More importantly for the risk committee, month-to-month score changes for the same applicant fall by more than half, which makes approval thresholds far easier to defend.