What Is Least Squares Approximation?

Least squares approximation is a mathematical technique for finding the best-fitting function to a set of data points by minimizing the sum of the squared differences between the observed values and the values predicted by the function. When a system of equations has more equations than unknowns, an exact solution generally does not exist, and least squares approximation produces the solution that makes the total squared residual as small as possible. This criterion, first formulated by Carl Friedrich Gauss and Adrien-Marie Legendre independently at the beginning of the nineteenth century, remains the standard measure of fit in regression analysis, signal reconstruction, numerical computation, and engineering system identification.

The method applies to both linear and nonlinear problems. In the linear case, the unknown parameters enter the model as a linear combination, and the solution is obtained by solving the normal equations, a square system derived from the overdetermined original. Nonlinear least squares approximation requires iterative optimization methods because the normal equations become nonlinear functions of the parameters. The NIST publication on least-squares fitting algorithms in the NIST Algorithm Testing System documents reference implementations for fitting curves and surfaces to three-dimensional measurement data using orthogonal distance regression.

Linear Least Squares and the Normal Equations

For a linear model with a data matrix A and observation vector b, the least squares approximation seeks the parameter vector x that minimizes the Euclidean norm of the residual Ax minus b. Setting the gradient of this norm to zero produces the normal equations, A-transpose A times x equals A-transpose b. When A has full column rank, the normal equations have a unique solution given by x equals the pseudoinverse of A applied to b. Direct solution via the normal equations is computationally efficient but can suffer from numerical ill-conditioning when the columns of A are nearly linearly dependent. Orthogonal decomposition methods, including QR factorization and singular value decomposition, solve the same problem with greater numerical stability, as described in the Georgia Tech linear algebra resource on the method of least squares.

Discrete and Continuous Least Squares Approximation

Least squares approximation takes two principal forms depending on how the data are represented. Discrete least squares approximation fits a function to a finite set of data points, as in polynomial regression or trigonometric approximation of sampled signals. Continuous least squares approximation approximates a known function by a simpler function over an interval, minimizing an integral of the squared difference rather than a finite sum. Orthogonal polynomial families, including Legendre, Chebyshev, and Hermite polynomials, arise naturally in continuous least squares because they are mutually orthogonal under the relevant inner product, allowing the approximation coefficients to be computed independently for each polynomial term.

Numerical Stability and Regularization

Ill-conditioned least squares problems, where small changes in data lead to large changes in the solution, arise when the data matrix is nearly rank-deficient. Regularization addresses this by adding a penalty on the solution magnitude to the minimization criterion. Tikhonov regularization, also called ridge regression in statistical contexts, adds a term proportional to the squared norm of the solution, biasing it toward smaller values while preserving the linearity of the problem. The NIST Digital Library of Mathematical Functions provides approximation techniques including least squares methods with precise definitions and numerical guidance.

Applications

Least squares approximation has applications in a wide range of disciplines, including:

• Curve fitting and polynomial regression in experimental data analysis
• Geodetic coordinate measurement and coordinate metrology
• Image reconstruction and signal denoising
• System identification from input-output measurements
• Spectroscopic data fitting in analytical chemistry
• Orbit determination and navigation in aerospace systems