What Are Least Squares Methods?

Least squares methods are a family of mathematical and computational techniques that estimate unknown quantities by minimizing the sum of squared differences between observed measurements and model predictions. The family encompasses linear and nonlinear regression, overdetermined system solution, curve and surface fitting, and adaptive parameter estimation, all unified by the common optimality criterion of minimizing the sum of squared residuals. Least squares methods occupy a foundational role in statistics, numerical analysis, and engineering, providing the basis for parameter estimation in fields ranging from geodesy and astronomy to signal processing and machine learning.

The method originated in the work of Carl Friedrich Gauss, who applied it to astronomical data around 1795, and Adrien-Marie Legendre, who published the first formal description in 1805. Both recognized that for measurement data corrupted by independent, identically distributed Gaussian noise, minimizing the sum of squared residuals yields the maximum likelihood estimate of the unknown parameters. This statistical justification, combined with the method's computational tractability, led to its adoption across quantitative science during the nineteenth century and its subsequent expansion into the computational methods documented by NIST in their least-squares fitting algorithm testing work.

Linear Least Squares

Linear least squares methods apply when the unknown parameters enter the model linearly. The fundamental problem is to find the parameter vector that minimizes the Euclidean norm of the residual for an overdetermined linear system. The solution satisfies the normal equations, and when the data matrix has full column rank, the solution is unique and can be computed by QR factorization or singular value decomposition with well-controlled numerical accuracy. Weighted least squares generalizes the criterion by allowing different observations to carry different importance, accommodating datasets where measurement precision varies across observations. These numerical methods are catalogued in the NIST Digital Library of Mathematical Functions section on approximation techniques and least squares, which provides reference implementations and error bounds for scientific computing applications.

Nonlinear Least Squares

Nonlinear least squares methods handle models in which the parameters appear nonlinearly, requiring iterative solution strategies. The Gauss-Newton method linearizes the model about the current parameter estimate at each iteration, solving a local linear least squares subproblem to compute the update step. The Levenberg-Marquardt algorithm interpolates between Gauss-Newton and steepest descent, using a damping parameter to ensure progress even when the Gauss-Newton step would lead away from the minimum. Convergence depends on the quality of the initial parameter estimate and the degree of nonlinearity in the model. Nonlinear least squares is the standard technique for fitting exponential decay curves, nonlinear physical models, and geometric primitives such as circles and ellipses to point cloud data.

Constrained and Regularized Variants

Constrained least squares methods incorporate equality or inequality constraints on the solution, arising in problems where physical feasibility limits the allowable parameter range. The equality-constrained solution is obtained by eliminating constraint equations or by the method of Lagrange multipliers, which augments the cost function with penalty terms for constraint violations. Regularized least squares adds a penalty on the solution norm to combat ill-conditioning: Tikhonov regularization produces a smoothed solution by penalizing large parameter values, while truncated singular value decomposition discards components corresponding to small singular values. These variants are widely used in tomographic image reconstruction, inverse problems in geophysics, and statistical model selection. The SIAM numerical methods book, cited as a standard reference for least squares problems, covers constrained and regularized formulations with rigorous theoretical treatment.

Applications

Least squares methods have applications in a wide range of disciplines, including:

• Orbit determination and navigation in aerospace and geodesy
• Statistical regression and experimental design in scientific research
• System identification and process modeling in control engineering
• Tomographic reconstruction in medical and geophysical imaging
• Sensor calibration and coordinate metrology
• Machine learning model fitting and feature regression