What Is Regression Tree Analysis?

Regression tree analysis is a nonparametric machine learning method that predicts a continuous numerical outcome by recursively partitioning the input space into regions and assigning a constant predicted value to each region. It belongs to the broader family of decision tree methods and shares its algorithmic foundation with classification trees, differing only in that the target variable is quantitative rather than categorical. The technique was formalized by Leo Breiman, Jerome Friedman, Richard Olshen, and Charles Stone in their 1984 book introducing the CART (Classification and Regression Trees) framework.

Unlike linear regression, regression tree analysis makes no assumption about the functional form of the relationship between predictors and response. A tree partitions the predictor space into a set of rectangular regions, fitting a simple model (typically the mean of the training responses) inside each region. This piecewise constant structure allows the method to capture interactions and discontinuities that would require complex polynomial or interaction terms in a parametric model.

Tree Construction and Splitting

Building a regression tree begins with the full training dataset and proceeds by finding the predictor and split threshold that most reduce prediction error. The standard criterion is minimizing the residual sum of squares: at each candidate split, the algorithm calculates the sum of squared deviations from the mean within each resulting child node and selects the split that achieves the greatest total reduction. The process repeats recursively within each child node until a stopping criterion is reached, such as a minimum node size or a maximum tree depth.

The result is a binary tree whose internal nodes contain split rules of the form "if predictor j is less than or equal to threshold t, go left; otherwise go right." As scikit-learn's decision tree documentation describes, a tree can be understood as a piecewise constant approximation: the deeper the tree, the more complex the decision rules and the tighter the fit to training data.

Pruning and Overfitting Control

A fully grown regression tree tends to overfit by modeling noise in the training data. Pruning addresses this by removing branches that add little predictive value. Cost-complexity pruning, introduced as part of the CART framework, adds a penalty proportional to the number of leaf nodes and then selects the subtree that minimizes penalized prediction error on held-out data. Cross-validation is the standard method for choosing the penalty parameter, trading off between tree complexity and generalization.

The bias-variance tradeoff is pronounced in regression trees. Shallow trees have high bias but low variance and may miss genuine patterns. Deep, unpruned trees have low bias but high variance and are sensitive to small changes in the training data.

Ensemble Extensions

A single regression tree is rarely the most accurate predictor available, but trees serve as the foundation for powerful ensemble methods. Random forests, introduced by Breiman in 2001, average the predictions of many trees trained on bootstrap samples with randomly selected subsets of predictors at each split, substantially reducing variance without increasing bias. Gradient boosted trees, developed by Friedman and described in detail in his 2001 paper in the Annals of Statistics, fit trees sequentially to the residuals of the current model, producing a highly accurate additive predictor.

Applications

Regression tree analysis has applications in a wide range of disciplines, including:

• Medical prognosis, predicting continuous outcomes such as blood pressure or length of hospital stay
• Real estate valuation, modeling sale prices as a function of property characteristics
• Environmental monitoring, predicting pollutant concentrations from meteorological and geographic features
• Financial risk modeling, estimating credit loss amounts given borrower and loan attributes