Random forest

Also known as: random decision forest

On this page 8

A random forest is a supervised machine learning method that builds many decision trees on random variations of the training data and combines their outputs, by majority vote for classification and by averaging for regression. Also called a random decision forest, it is one of the most widely used classic algorithms in artificial intelligence because it delivers strong accuracy on tabular data with little tuning and little preprocessing.

The method rests on two sources of randomness: bootstrap sampling of rows and random selection of candidate features at every split. Together they produce trees that make different errors, and averaging those errors away is what gives the forest its low variance compared with a single tree.

How a random forest builds and combines trees

A random forest trains each tree independently on a bootstrap sample, a dataset of the same size drawn from the original training set with replacement. On average, each bootstrap sample contains about 63.2% of the unique rows; the remaining roughly 36.8% are the out-of-bag (OOB) rows for that tree.

While growing a tree, the algorithm does not search all features at each node. It draws a random subset of features, controlled by the parameter usually called max_features or mtry, and chooses the best split only among them, using a criterion such as Gini impurity, entropy (information gain) or the reduction in squared error for regression. Trees are typically grown deep and unpruned, which keeps their bias low.

Prediction then aggregates the trees. The following table summarizes the steps of training and inference and the purpose of each step.

StageWhat happensWhy it matters
Bootstrap samplingEach tree gets rows drawn with replacementTrees see different data
Feature subsamplingEach split considers a random feature subsetDecorrelates the trees
Tree growthSplits continue until leaves are small or pureKeeps individual bias low
AggregationVotes or averages across all treesCancels independent errors
OOB evaluationEach row is scored by trees that did not see itBuilt-in validation estimate
Stages of a random forest and their role

Why averaging decorrelated trees reduces error

A random forest improves on a single tree by reducing variance without substantially raising bias. For an average of B trees, each with variance sigma squared and pairwise correlation rho, the variance of the average equals rho times sigma squared plus (1 minus rho) times sigma squared divided by B. Adding trees shrinks only the second term, so the correlation between trees sets the floor on achievable error.

This is why feature subsampling matters. Plain bagging of trees uses every feature at every split, so a few dominant predictors appear near the top of most trees and the trees remain highly correlated. Restricting the candidate features forces trees to explore other predictors, which lowers rho at the cost of slightly weaker individual trees. The result illustrates the bias-variance tradeoff directly: a small increase in per-tree bias buys a large reduction in ensemble variance.

An important consequence is that adding more trees does not cause overfitting in the usual sense. The test error converges as the number of trees grows; beyond that point extra trees only cost computation and memory.

Key hyperparameters and typical values

Random forests are known for sensible defaults, yet a handful of hyperparameters control most of the behavior. The table lists the common ones, their usual starting values and their effect on the model.

HyperparameterCommon defaultEffect of increasing it
Number of trees100 to 500More stable predictions, higher cost
Features per splitSquare root of p (classification), p/3 (regression)Stronger but more correlated trees
Maximum depthUnlimitedLower bias, higher variance per tree
Minimum samples per leaf1 (classification), 5 (regression)Smoother predictions, less overfitting
Bootstrap sample sizeEqual to training set sizeLess diversity between trees
Class weightsNoneMore attention to minority classes
Main random forest hyperparameters

In practice, the number of features per split and the minimum leaf size have the largest influence on accuracy. Systematic hyperparameter tuning with random search or Bayesian optimization usually yields modest gains of a few percentage points at most, which makes the forest a strong baseline model before more demanding methods are tried.

A random forest belongs to the family of ensemble learning methods, and its closest neighbors differ mainly in how trees are built and combined. The comparison below highlights the practical differences.

MethodHow trees relateMain strengthMain weakness
Single decision treeOne tree onlyFully interpretableHigh variance, unstable
BaggingIndependent trees, all featuresReduces varianceCorrelated trees
Random forestIndependent trees, random featuresRobust with little tuningLarge model, slower inference
Gradient boostingSequential trees fix residualsOften highest tabular accuracySensitive to tuning
Isolation forestRandom splits isolate pointsUnsupervised anomaly detectionNot a predictive model
Random forest and neighboring tree methods

Gradient boosting libraries frequently beat random forests on well-tuned benchmarks, but the forest is harder to misconfigure and trains in parallel because its trees do not depend on one another. A related variant, extremely randomized trees, also draws split thresholds at random, which further lowers correlation and speeds up training.

Feature importance and interpretation

Random forests provide built-in feature importance scores, but the two common variants behave differently. Impurity-based importance, also called mean decrease in impurity, sums the impurity reduction contributed by each feature across all splits. It is fast but biased toward high-cardinality and continuous features, which offer more candidate split points.

Permutation importance measures how much accuracy drops when the values of one feature are shuffled, ideally on out-of-bag or held-out data. It is more reliable, yet it underestimates correlated features, because the model can fall back on a correlated substitute. For individual predictions, practitioners often add SHAP values, which have an efficient exact algorithm for tree ensembles.

Limitations and common mistakes

A random forest is robust, but it has structural limits that experienced practitioners plan around. The table pairs typical symptoms with their causes and remedies.

SymptomCauseRemedy
Flat predictions outside training rangeTrees cannot extrapolateUse linear or boosted models with trends
Minority class ignoredClass imbalance dominates votesClass weights, balanced sampling, SMOTE
Misleading importance rankingImpurity bias or correlated featuresUse permutation importance or SHAP
Large memory footprintHundreds of deep treesLimit depth or prune the number of trees
Poor calibrated probabilitiesVote shares are not true probabilitiesApply isotonic or Platt calibration
Typical problems with random forests

Two further mistakes are common. Treating the forest as fully interpretable is misleading, since hundreds of trees cannot be read like one tree. Applying it to raw high-dimensional data such as images or text usually underperforms neural networks, because the forest relies on informative, well-engineered features. On the other hand, random forests need no feature scaling and tolerate outliers and mixed variable types well.

The simple explanation

Imagine asking one expert to guess the price of a house. The expert may be quite good, but any single person has blind spots. Now ask several hundred people, each of whom studied a slightly different set of past sales and paid attention to different details. Their individual guesses vary, yet the average of all guesses is usually closer to the truth than almost any single guess.

A random forest works the same way with decision trees, which are simple flowcharts of yes or no questions. Each tree learns from a random slice of the data and looks at a random selection of facts, so the trees make different mistakes that cancel out when their answers are combined. The key idea: many varied, imperfect opinions beat one confident opinion.

How it is used in practice

A credit risk team at a mid-sized consumer lender needs to predict which loan applicants will default within twelve months. Its data set holds about 200,000 past loans with 60 variables, including income, debt-to-income ratio, employment length and payment history, and only about 4% of the loans defaulted.

The team trains a random forest with 500 trees, the square root of the feature count at each split, a minimum leaf size of 20 and balanced class weights. The out-of-bag score gives an early estimate of an area under the ROC curve of about 0.82, which a separate test set confirms. Permutation importance shows that debt-to-income ratio and recent missed payments drive most predictions, which matches the judgment of the underwriters.

Because vote shares are not true probabilities, the team calibrates the outputs before setting approval thresholds. The calibrated forest becomes the production baseline, and a later gradient boosting model must outperform it before it can replace it.