What Are Least Squares Approximations?

Least squares approximations are solutions to overdetermined fitting problems that minimize the sum of squared differences between observed data and values predicted by a parameterized model. When the number of observations exceeds the number of free parameters, an exact fit is typically impossible, and the least squares criterion selects the parameter values that bring the model as close as possible to all observations simultaneously. The approach extends across a wide range of model classes: linear regression, polynomial fitting, spline interpolation, Fourier series truncation, and nonlinear curve fitting all employ least squares as the governing optimality criterion. The technique draws on linear algebra, numerical analysis, and statistical estimation theory, and it remains one of the most frequently applied mathematical methods in engineering and the physical sciences.

The classical least squares formulation traces to the work of Carl Friedrich Gauss on planetary orbit determination in the early 1800s. Gauss showed that, under the assumption of normally distributed measurement errors, the least squares estimate coincides with the maximum likelihood estimate of the unknown parameters. This probabilistic interpretation underpins much of modern statistical regression theory. The NIST reference on least squares fitting algorithms documents computational methods for fitting geometric primitives and surfaces in three-dimensional measurement applications, illustrating the range of practical approximation problems the criterion addresses.

Curve Fitting

Curve fitting with least squares approximations seeks a functional form, often a polynomial, rational function, or trigonometric series, whose evaluated values are as close as possible to a set of measured data points. Polynomial least squares fitting of degree d requires solving the normal equations for d plus one coefficients, and the solution is unique when d is less than the number of data points. Spline fitting subdivides the domain into intervals and fits piecewise polynomials with continuity constraints at the junction points, providing flexibility to represent complex shapes without the oscillatory artifacts that accompany high-degree global polynomials. The choice between polynomial and spline approximations depends on the smoothness of the underlying function and the distribution of the data points. The Georgia Tech resource on the method of least squares provides accessible derivations of the normal equation approach applied to polynomial regression.

Recursive Estimation

Recursive least squares (RLS) approximations update the parameter estimate and the inverse correlation matrix incrementally as each new observation arrives, without re-solving the full normal equations from scratch. The update equations take constant time per step regardless of how many observations have accumulated, making RLS suitable for online estimation in adaptive control, system identification, and communications channel tracking. The Kalman filter is a closely related recursive estimation framework that incorporates a state-space model for parameter evolution and a model of observation noise, yielding statistically optimal estimates under Gaussian assumptions. Recursive estimation methods are preferred over batch least squares in nonstationary problems where the parameters of interest drift over time.

Approximation Methods and Regularization

Least squares approximations extend beyond direct fitting through a family of approximation methods that impose additional structure on the solution. Orthogonal polynomial expansions, including Chebyshev and Legendre series, yield approximation coefficients through inner products and avoid solving a linear system entirely. Regularized least squares adds a penalty term on solution complexity to prevent overfitting when the data are noisy or the model is over-parameterized. Total least squares accommodates errors in both the observations and the predictor variables, relevant when measurement noise affects all quantities in the model. Numerical methods for these variants are catalogued in the NIST Digital Library of Mathematical Functions under approximation techniques for numerical methods.

Applications

Least squares approximations have applications in a wide range of disciplines, including:

• Parameter estimation in dynamic system identification
• Data smoothing and signal reconstruction in instrumentation
• Geodetic surface fitting and geoid determination
• Spectral analysis and frequency estimation
• Image registration and geometric transformation fitting
• Statistical regression in experimental design and analysis