Bagging, short for bootstrap aggregating, is an ensemble method in machine learning that trains many copies of the same base model on random bootstrap samples of the training data and combines their predictions by averaging or majority voting. Its purpose is to reduce variance: individual models that react strongly to small changes in the data become far more stable once their errors are averaged out.
Leo Breiman introduced the method in 1996, and it remains one of the most reliable tools in artificial intelligence practice for tabular data. Bagging is the foundation of the random forest, it is the natural counterpart to boosting within ensemble learning, and it offers a free performance estimate through out-of-bag error.
How bootstrap aggregating works
Bagging works in two stages: bootstrap resampling creates many slightly different training sets, and aggregation merges the models trained on them. Each bootstrap sample is drawn with replacement and usually has the same size as the original dataset, so some rows appear several times while others are missing entirely.
Because every row has a chance of 1 – 1/n of being skipped on each draw, the probability that it is absent from a sample of size n approaches about 36.8 percent. Each base model therefore sees roughly 63.2 percent of the unique rows, and the remaining rows form its out-of-bag set.
| Step | What happens | Typical setting |
|---|---|---|
| Resample | Draw B bootstrap samples with replacement | B from 50 to 500 |
| Train | Fit one base learner per sample independently | Unpruned decision trees |
| Predict | Each learner scores a new input | Runs in parallel |
| Aggregate | Average outputs or take a majority vote | Mean or soft voting |
| Validate | Score each row only with models that never saw it | Out-of-bag estimate |
For regression, the ensemble prediction is the arithmetic mean of the base predictions. For classification, practitioners use either hard voting, where the most frequent class wins, or soft voting, where predicted probabilities are averaged; soft voting usually yields better calibrated probabilities and slightly higher accuracy.
Why bagging reduces variance
Bagging reduces variance because averaging several noisy estimates cancels part of their independent errors. If B models each have variance sigma squared and a pairwise correlation rho, the variance of their average equals rho times sigma squared plus (1 – rho) times sigma squared divided by B. As B grows, the second term vanishes, but the first term remains: correlation between models sets the floor.
This formula explains the core of the bias-variance tradeoff for bagging. The method leaves bias almost unchanged, since every model has the same structure, while it can cut variance sharply when the base learners are unstable and weakly correlated. Random forests push correlation lower by also sampling a random subset of features at every split.
| Base learner | Stability | Benefit from bagging |
|---|---|---|
| Deep decision tree | Very unstable | Large accuracy gain |
| Neural network | Unstable across seeds | Moderate gain, high cost |
| k-nearest neighbors | Stable for large k | Small or none |
| Linear regression | Stable | Negligible |
| Naive Bayes classifier | Very stable | Negligible |
A practical rule follows: bagging helps most when the base model has low bias and high variance. Applying it to a stable, high-bias model such as logistic regression adds computation without fixing the real problem, which is underfitting rather than instability.
Bagging compared with boosting and stacking
Bagging differs from the other major ensemble strategies in how models are trained and what kind of error they attack. Boosting trains models sequentially, each one focusing on the mistakes of its predecessors, and mainly reduces bias. Stacking trains a meta-model that learns how to weigh the outputs of diverse base models.
| Property | Bagging | Boosting | Stacking |
|---|---|---|---|
| Training order | Parallel | Sequential | Two levels |
| Main error reduced | Variance | Bias and variance | Model selection error |
| Typical base learner | Deep trees | Shallow trees | Diverse model families |
| Sensitivity to label noise | Low | Higher | Moderate |
| Overfitting risk with more models | Very low | Grows without early stopping | Depends on meta-model |
| Representative algorithms | Random forest, extra trees | AdaBoost, gradient boosting | Blending, super learner |
A distinctive property of bagging is that adding more models does not cause overfitting: the error curve flattens as B increases instead of turning upward. Boosting, by contrast, needs a learning rate and early stopping to control its capacity. On noisy data with mislabeled examples, bagging is usually the more robust choice.
Variants and configuration choices
Several variants change what is resampled, and each trades diversity against individual model strength. Resampling rows, columns, or both alters how decorrelated the ensemble becomes and how much data each model sees.
| Variant | What is sampled | When it fits |
|---|---|---|
| Classic bagging | Rows with replacement | General purpose variance reduction |
| Pasting | Rows without replacement | Very large datasets |
| Random subspaces | Features only | Many features, few rows |
| Random patches | Rows and features | Wide, large datasets |
| Random forest | Rows plus features per split | Default for tabular data |
The most important hyperparameters are the number of estimators, the sample fraction and the feature fraction. In hyperparameter tuning, the number of estimators is rarely worth searching: it should simply be large enough for the out-of-bag error to plateau, typically between 100 and 500. Sample fractions below 1.0 speed up training and increase diversity, which can help on large datasets.
- Set the out-of-bag score option to obtain validation accuracy without a separate holdout split.
- Use stratified or balanced bootstrap samples when class imbalance is severe, so minority cases appear in every sample.
- Fix the random seed for reproducibility, because results otherwise change slightly between runs.
Limitations and common mistakes
Bagging is not a universal accuracy booster, and its weaknesses matter in production. The ensemble cannot reduce bias, it multiplies memory and inference cost by the number of models, and it replaces one interpretable tree with hundreds, which weakens interpretability.
| Symptom | Likely cause | Remedy |
|---|---|---|
| No gain over a single model | Stable or high-bias base learner | Use deeper trees or switch to boosting |
| Gain stalls early | Highly correlated models | Add feature subsampling |
| Slow predictions | Too many large trees | Limit depth or prune the ensemble |
| Optimistic validation score | Duplicated rows leak across samples | Deduplicate or group rows before resampling |
| Poor minority class recall | Bootstrap samples miss rare classes | Use balanced bagging |
A frequent misconception is that bagging creates new information. It only reuses the existing sample, so it cannot compensate for biased data collection or missing features. Another pitfall concerns time series: ordinary bootstrap sampling breaks temporal order, so practitioners use block bootstrap methods instead.
A plain-language summary
Imagine asking one person to guess how many beans are in a jar. The guess may be far off. Now ask two hundred people, each of whom has looked at the jar from a slightly different angle, and take the average. The individual mistakes partly cancel out, and the average is usually much closer to the truth.
Bagging does the same with computer models. It shows each model a slightly different, randomly shuffled version of the same data and then lets them vote. The crowd of models is steadier than any single model, especially when each one on its own tends to overreact to small details.
An example from practice
A credit risk team at a mid-sized lender predicts whether small business loans will default, using about 40,000 historical applications and 60 features. A single deep decision tree reaches an area under the ROC curve of 0.74 on a holdout set, but its scores swing noticeably whenever the training data is refreshed each month.
The team replaces it with a bagged ensemble of 300 trees, using balanced bootstrap samples because only 6 percent of loans default. The out-of-bag estimate stabilizes after about 200 trees, and holdout performance rises to an area under the curve of 0.81. More importantly for the risk committee, month-to-month score changes for the same applicant fall by more than half, which makes approval thresholds far easier to defend.