What Is Linear Regression?

Linear regression is a statistical modeling method that estimates the relationship between one or more input variables (predictors) and a continuous output variable (response) by fitting a linear function to observed data. The model assumes that the expected value of the response is a linear combination of the predictors, with the goal of determining the coefficients that best explain the observed variation. Linear regression is among the oldest and most widely used quantitative methods in science, engineering, and economics, originating in work by Gauss and Legendre in the early nineteenth century on celestial orbit fitting and formalized through the development of the method of least squares.

The appeal of linear regression lies in its interpretability and its strong theoretical grounding. Each coefficient has a direct meaning: it quantifies the expected change in the response for a unit change in the corresponding predictor, holding all other predictors constant. This property makes the method not just a predictive tool but an instrument for understanding causal structure, though attribution of causal meaning requires careful experimental design or additional assumptions beyond the statistical model.

Model Formulation and Ordinary Least Squares

The standard linear regression model expresses the response y as the sum of a linear predictor and a random error term: y = beta_0 + beta_1x1 + ... + beta_pxp + epsilon, where the epsilon are independent errors with zero mean and constant variance (homoscedasticity). In matrix notation, this becomes y = Xβ + ε, where X is the design matrix. The ordinary least squares (OLS) estimator minimizes the sum of squared residuals between observed and fitted values, yielding the closed-form solution β̂ = (X^T X)^{-1} X^T y. The Gauss-Markov theorem guarantees that, under the standard assumptions, OLS produces the best linear unbiased estimator, providing a strong optimality guarantee. The scikit-learn LinearRegression implementation uses this formulation directly, solving the normal equations via singular value decomposition for numerical stability.

Inference and Model Diagnostics

Under the additional assumption that the errors are normally distributed, the OLS estimator follows a multivariate normal distribution, enabling hypothesis tests and confidence intervals for each coefficient. The F-test for overall model significance and the t-test for individual coefficients are the standard inferential tools. Residual analysis is the primary diagnostic instrument: plots of residuals against fitted values reveal departures from linearity and heteroscedasticity, while Q-Q plots test the normality assumption. The coefficient of determination R^2 measures the fraction of total variance explained by the model, ranging from zero for a null model to one for a perfect fit. Variance inflation factors (VIF) detect multicollinearity, the condition where predictors are nearly linearly dependent, which inflates coefficient standard errors. The Cornell Optimization Wiki overview of regression and classical textbooks by Draper and Smith give full treatments of these diagnostic procedures.

Extensions and Regularized Variants

Multiple linear regression extends simple (one-predictor) regression to any number of predictors. Polynomial regression adds powers of the predictors as additional columns in the design matrix, allowing nonlinear response surfaces while remaining linear in the coefficients. Ridge regression adds an L2 penalty term lambda times the sum of squared coefficients to the least squares objective, shrinking the estimates toward zero and reducing variance at the cost of small bias, a useful approach when predictors are correlated or the design matrix is near-singular. Lasso regression uses an L1 penalty that drives some coefficients exactly to zero, performing simultaneous estimation and variable selection. The elastic net combines both penalties. These regularized variants are analyzed in detail in the ACM literature on matrix quantization and regression-based signal modeling.

Applications

Linear regression has applications across a wide range of fields, including:

• Prediction and forecasting in economics, finance, and engineering
• Calibration of sensors and measurement instruments
• Epidemiology and clinical trials for estimating treatment effects
• Machine learning pipelines as a baseline and interpretable model
• Signal processing for system identification and autoregressive modeling