A random forest is a supervised machine learning method that builds many decision trees on random variations of the training data and combines their outputs, by majority vote for classification and by averaging for regression. Also called a random decision forest, it is one of the most widely used classic algorithms in artificial intelligence because it delivers strong accuracy on tabular data with little tuning and little preprocessing.
The method rests on two sources of randomness: bootstrap sampling of rows and random selection of candidate features at every split. Together they produce trees that make different errors, and averaging those errors away is what gives the forest its low variance compared with a single tree.
How a random forest builds and combines trees
A random forest trains each tree independently on a bootstrap sample, a dataset of the same size drawn from the original training set with replacement. On average, each bootstrap sample contains about 63.2% of the unique rows; the remaining roughly 36.8% are the out-of-bag (OOB) rows for that tree.
While growing a tree, the algorithm does not search all features at each node. It draws a random subset of features, controlled by the parameter usually called max_features or mtry, and chooses the best split only among them, using a criterion such as Gini impurity, entropy (information gain) or the reduction in squared error for regression. Trees are typically grown deep and unpruned, which keeps their bias low.
Prediction then aggregates the trees. The following table summarizes the steps of training and inference and the purpose of each step.
| Stage | What happens | Why it matters |
|---|---|---|
| Bootstrap sampling | Each tree gets rows drawn with replacement | Trees see different data |
| Feature subsampling | Each split considers a random feature subset | Decorrelates the trees |
| Tree growth | Splits continue until leaves are small or pure | Keeps individual bias low |
| Aggregation | Votes or averages across all trees | Cancels independent errors |
| OOB evaluation | Each row is scored by trees that did not see it | Built-in validation estimate |
Why averaging decorrelated trees reduces error
A random forest improves on a single tree by reducing variance without substantially raising bias. For an average of B trees, each with variance sigma squared and pairwise correlation rho, the variance of the average equals rho times sigma squared plus (1 minus rho) times sigma squared divided by B. Adding trees shrinks only the second term, so the correlation between trees sets the floor on achievable error.
This is why feature subsampling matters. Plain bagging of trees uses every feature at every split, so a few dominant predictors appear near the top of most trees and the trees remain highly correlated. Restricting the candidate features forces trees to explore other predictors, which lowers rho at the cost of slightly weaker individual trees. The result illustrates the bias-variance tradeoff directly: a small increase in per-tree bias buys a large reduction in ensemble variance.
An important consequence is that adding more trees does not cause overfitting in the usual sense. The test error converges as the number of trees grows; beyond that point extra trees only cost computation and memory.
Key hyperparameters and typical values
Random forests are known for sensible defaults, yet a handful of hyperparameters control most of the behavior. The table lists the common ones, their usual starting values and their effect on the model.
| Hyperparameter | Common default | Effect of increasing it |
|---|---|---|
| Number of trees | 100 to 500 | More stable predictions, higher cost |
| Features per split | Square root of p (classification), p/3 (regression) | Stronger but more correlated trees |
| Maximum depth | Unlimited | Lower bias, higher variance per tree |
| Minimum samples per leaf | 1 (classification), 5 (regression) | Smoother predictions, less overfitting |
| Bootstrap sample size | Equal to training set size | Less diversity between trees |
| Class weights | None | More attention to minority classes |
In practice, the number of features per split and the minimum leaf size have the largest influence on accuracy. Systematic hyperparameter tuning with random search or Bayesian optimization usually yields modest gains of a few percentage points at most, which makes the forest a strong baseline model before more demanding methods are tried.
Random forest compared with related ensemble methods
A random forest belongs to the family of ensemble learning methods, and its closest neighbors differ mainly in how trees are built and combined. The comparison below highlights the practical differences.
| Method | How trees relate | Main strength | Main weakness |
|---|---|---|---|
| Single decision tree | One tree only | Fully interpretable | High variance, unstable |
| Bagging | Independent trees, all features | Reduces variance | Correlated trees |
| Random forest | Independent trees, random features | Robust with little tuning | Large model, slower inference |
| Gradient boosting | Sequential trees fix residuals | Often highest tabular accuracy | Sensitive to tuning |
| Isolation forest | Random splits isolate points | Unsupervised anomaly detection | Not a predictive model |
Gradient boosting libraries frequently beat random forests on well-tuned benchmarks, but the forest is harder to misconfigure and trains in parallel because its trees do not depend on one another. A related variant, extremely randomized trees, also draws split thresholds at random, which further lowers correlation and speeds up training.
Feature importance and interpretation
Random forests provide built-in feature importance scores, but the two common variants behave differently. Impurity-based importance, also called mean decrease in impurity, sums the impurity reduction contributed by each feature across all splits. It is fast but biased toward high-cardinality and continuous features, which offer more candidate split points.
Permutation importance measures how much accuracy drops when the values of one feature are shuffled, ideally on out-of-bag or held-out data. It is more reliable, yet it underestimates correlated features, because the model can fall back on a correlated substitute. For individual predictions, practitioners often add SHAP values, which have an efficient exact algorithm for tree ensembles.
Limitations and common mistakes
A random forest is robust, but it has structural limits that experienced practitioners plan around. The table pairs typical symptoms with their causes and remedies.
| Symptom | Cause | Remedy |
|---|---|---|
| Flat predictions outside training range | Trees cannot extrapolate | Use linear or boosted models with trends |
| Minority class ignored | Class imbalance dominates votes | Class weights, balanced sampling, SMOTE |
| Misleading importance ranking | Impurity bias or correlated features | Use permutation importance or SHAP |
| Large memory footprint | Hundreds of deep trees | Limit depth or prune the number of trees |
| Poor calibrated probabilities | Vote shares are not true probabilities | Apply isotonic or Platt calibration |
Two further mistakes are common. Treating the forest as fully interpretable is misleading, since hundreds of trees cannot be read like one tree. Applying it to raw high-dimensional data such as images or text usually underperforms neural networks, because the forest relies on informative, well-engineered features. On the other hand, random forests need no feature scaling and tolerate outliers and mixed variable types well.
The simple explanation
Imagine asking one expert to guess the price of a house. The expert may be quite good, but any single person has blind spots. Now ask several hundred people, each of whom studied a slightly different set of past sales and paid attention to different details. Their individual guesses vary, yet the average of all guesses is usually closer to the truth than almost any single guess.
A random forest works the same way with decision trees, which are simple flowcharts of yes or no questions. Each tree learns from a random slice of the data and looks at a random selection of facts, so the trees make different mistakes that cancel out when their answers are combined. The key idea: many varied, imperfect opinions beat one confident opinion.
How it is used in practice
A credit risk team at a mid-sized consumer lender needs to predict which loan applicants will default within twelve months. Its data set holds about 200,000 past loans with 60 variables, including income, debt-to-income ratio, employment length and payment history, and only about 4% of the loans defaulted.
The team trains a random forest with 500 trees, the square root of the feature count at each split, a minimum leaf size of 20 and balanced class weights. The out-of-bag score gives an early estimate of an area under the ROC curve of about 0.82, which a separate test set confirms. Permutation importance shows that debt-to-income ratio and recent missed payments drive most predictions, which matches the judgment of the underwriters.
Because vote shares are not true probabilities, the team calibrates the outputs before setting approval thresholds. The calibrated forest becomes the production baseline, and a later gradient boosting model must outperform it before it can replace it.