What Are Random Forests?

Random Forests are an ensemble machine learning method that constructs a large collection of decision trees during training and outputs the aggregated prediction of the entire collection at inference time. For classification tasks, the forest returns the class that receives the most votes across all trees; for regression tasks, it returns the mean of the individual tree outputs. The approach addresses the central weakness of single decision trees, which tend to overfit training data and produce predictions that vary widely when the training set changes. By averaging many diverse trees, a random forest achieves lower variance than any individual tree with little increase in bias.

The method was introduced by Leo Breiman in a 2001 paper in Machine Learning, building on earlier ideas about random subspace methods by Tin Kam Ho and on Breiman's own bootstrap aggregation (bagging) technique from 1996. The foundational Random Forests paper by Breiman established that forests do not overfit as the number of trees grows, converging instead to a generalization error bound determined by tree strength and pairwise correlation.

Ensemble Construction and Bagging

Each tree in a random forest is grown on a bootstrap sample, a dataset of the same size as the original training set drawn with replacement. Because sampling with replacement omits some training examples from each tree, roughly one-third of the data is left out of any given tree on average. These out-of-bag samples serve as a built-in validation set: predictions on out-of-bag examples aggregate across all trees that excluded that example, producing an unbiased estimate of generalization error without a separate held-out set. A regression analysis framing applies here as well: the variance of the ensemble prediction equals the average pairwise correlation between trees times the average variance of a single tree, which reveals the two complementary strategies for improving performance: reduce tree correlation or reduce individual tree error.

Feature Randomness and Variance Reduction

What distinguishes random forests from ordinary bagging is the injection of additional randomness at each tree node. When splitting a node, only a random subset of features is considered as split candidates, typically the square root of the total number of features for classification and one-third for regression. This constraint ensures that strongly predictive features do not dominate every tree, allowing weaker features to contribute splits in some trees and thereby decorrelating the ensemble. The scikit-learn ensemble documentation provides a detailed treatment of how the number of candidate features at each split trades off ensemble diversity against individual tree quality. Feature importance scores, derived by measuring the average reduction in node impurity attributable to each feature across all trees, offer a useful but correlation-sensitive ranking of predictor variables.

Applications

Random Forests are applied across a wide range of predictive modeling and data analysis tasks, including:

• Image classification and object detection in computer vision pipelines
• Remote sensing and land cover mapping from multispectral satellite imagery
• Bioinformatics, where forests handle the high-dimensional, small-sample regime common in genomic data
• Financial services applications including credit scoring and anomaly-based fraud detection
• Medical diagnosis from tabular clinical features such as laboratory values and patient history
• Ecological modeling, predicting species distribution from environmental covariates