What Is Overfitting?

Overfitting is a failure mode of statistical and machine learning models in which a fitted model reproduces the training data closely while performing poorly on new data drawn from the same distribution. The model has absorbed sampling noise, measurement error, and accidental structure specific to the training sample and treated all of it as signal. Because the quantity a modeler actually cares about is expected error on unseen inputs, and the quantity that is directly observable is error on the training sample, overfitting is the gap between the two.

The concept predates machine learning. It appears in regression analysis, curve fitting, time series modeling, and experimental design, wherever a model with adjustable parameters is estimated from a finite sample. The mirror-image failure is underfitting, in which the model is too constrained to express the structure that is genuinely present. Model selection is the practice of positioning a model between those two failures.

Bias, Variance, and Model Capacity

The classical account frames the problem through the bias-variance decomposition. Expected prediction error separates into a bias term, reflecting systematic error from an overly restrictive hypothesis class, and a variance term, reflecting sensitivity of the fitted parameters to the particular training sample drawn. Increasing capacity, whether through polynomial degree, tree depth, or network width, lowers bias and raises variance, producing the U-shaped test error curve found in textbooks. A tutorial treatment of overfitting and generalization error works through this decomposition alongside the resampling methods used to estimate it. Capacity itself can be measured in several ways, including the number of free parameters, the Vapnik-Chervonenkis dimension, and Rademacher complexity, each yielding different generalization bounds.

Detection and Estimation of Generalization Error

Overfitting cannot be diagnosed from training error, so it is detected by holding data back. A held-out validation set gives a single estimate; k-fold cross-validation averages over k partitions and uses the data more efficiently at higher computational cost; leave-one-out cross-validation takes k to the sample size. Learning curves that plot training and validation error against training set size or training epoch make the divergence visible directly. Care is needed with dependent data: time series require forward-chaining splits, and grouped observations require splitting by group rather than by row, or the validation estimate becomes optimistic. Repeated model selection against the same validation set eventually overfits that set too, which is why a separate test set is held in reserve.

Regularization and Capacity Control

Mitigation works by constraining the effective capacity of the model. Ridge and lasso regression add L2 and L1 penalties on coefficient magnitude, with L1 additionally driving coefficients to exactly zero and performing variable selection. Neural network training adds weight decay, dropout, early stopping, batch normalization, and data augmentation. Ensemble methods such as bagging and random forests reduce variance by averaging over resampled fits. Deep networks complicate the classical picture: heavily overparameterized models trained to interpolate the training data often generalize well, a phenomenon described by the double descent curve reported by Belkin and colleagues. Overfitting also arises in settings without a fixed dataset, including deep reinforcement learning, where agents memorize particular environment instances rather than learning transferable policies.

Applications

Overfitting is a central concern in a range of fields, including:

• Predictive modeling in medicine and epidemiology
• Financial forecasting and quantitative trading strategy backtests
• Computer vision and speech recognition system development
• Genome-wide association studies and other high-dimensional biology
• Industrial process control and reliability modeling
• Benchmark design and evaluation in machine learning research