What Are Approximation Methods?

Approximation methods are mathematical and computational techniques for producing tractable representations of functions, solutions, or data that cannot be expressed exactly in a desired form, whether due to computational cost, measurement noise, or the inherent complexity of the underlying system. Rather than seeking exact solutions, these methods seek solutions that are close in a well-defined sense, quantified by a norm, a residual, or a worst-case bound. Approximation methods underpin numerical analysis, signal processing, control engineering, statistics, and machine learning, making them among the most broadly applied tools in applied mathematics and engineering.

The field draws its foundations from functional analysis, linear algebra, optimization theory, and the classical theory of polynomial and trigonometric approximation developed by Chebyshev, Weierstrass, and Fourier in the nineteenth and early twentieth centuries. Contemporary engineering applications have extended these foundations into high-dimensional settings, sparse representations, and online adaptive schemes.

Least Squares and Optimization-Based Approximation

Least squares is the central workhorse of approximation in engineering. Given a set of data points or a system of overdetermined equations, the least squares method finds the parameter vector that minimizes the sum of squared residuals between the approximation and the target. It provides a single, well-defined solution even when the system is inconsistent, and it connects directly to maximum-likelihood estimation under Gaussian noise assumptions.

Applications of least squares extend from curve fitting and system identification to filter design and image reconstruction. NYU lecture notes on least squares with examples in signal processing demonstrate how linear prediction, smoothing, deconvolution, and missing data estimation all reduce to least squares problems with different structure matrices. The SIAM volume on numerical methods for least squares problems provides rigorous treatment of generalized least squares, rank-deficient problems, and iterative solvers for large sparse systems.

Signal Representation and Basis Approximation

A second major class of approximation methods concerns the representation of signals or functions as weighted superpositions of basis elements. Fourier series decompose periodic signals into sinusoidal components; wavelet decompositions provide time-frequency localization suited to signals with transient features; polynomial bases such as Chebyshev and Legendre polynomials are used in spectral methods for differential equations. In each case, the approximation retains a finite number of coefficients chosen to minimize a representation error.

Sparse approximation, developed extensively since the 1990s, pursues the fewest basis elements needed to represent a signal within a specified tolerance. Matching pursuit, basis pursuit, and the LASSO algorithm are canonical methods in this class. They have found direct application in compressed sensing, where signals are recovered from far fewer measurements than classical Nyquist sampling would require, reducing both data acquisition cost and storage requirements in sensor networks and medical imaging systems.

Minimization Methods and Iterative Solvers

Many approximation problems reduce to minimizing an objective function over a parameter space. Gradient descent, Newton's method, conjugate gradient iteration, and their stochastic variants are the primary tools. Each involves a trade-off among convergence rate, per-iteration computational cost, and sensitivity to the problem's conditioning. Optimal sampling strategies for least-squares approximation, as described in Springer's Foundations of Computational Mathematics on optimal sampling, address the question of which data locations minimize the approximation error for a given sample budget, with implications for sensor placement and active experimental design.

Applications

Approximation methods are applied across a wide range of fields, including:

• Robotics, for real-time approximate solutions to inverse kinematics and trajectory optimization
• Control system design, where plant models are identified from data using least squares
• Signal processing, including adaptive filtering, spectral estimation, and audio coding
• Finite element analysis, where complex geometries are approximated by piecewise polynomial meshes
• Machine learning model training, where stochastic gradient descent approximates the full-batch minimizer