What Is Function Approximation?

Function approximation is the practice of representing an unknown or computationally intractable function by a simpler, well-defined mathematical object that closely matches the target function's outputs over a domain of interest. It sits at the intersection of numerical analysis, approximation theory, and machine learning, and provides the theoretical foundation for methods ranging from polynomial interpolation to neural network regression. The goal is to select a family of candidate functions parameterized by a finite set of coefficients, then tune those coefficients so that the approximant minimizes some measure of error relative to the true function or a set of observed samples.

Classical approximation theory draws on the Weierstrass approximation theorem, which guarantees that any continuous function on a closed interval can be approximated to arbitrary accuracy by a polynomial. Fourier analysis extends this principle to periodic functions using trigonometric bases, while wavelet bases enable approximation of functions with localized features. These theoretical results bound the achievable accuracy and guide the choice of basis for practical computations.

Polynomial and Basis Function Methods

The most direct approach to function approximation is to express the target function as a linear combination of basis functions selected from a known family. Polynomial bases, including monomials, Chebyshev polynomials, and Legendre polynomials, are standard choices for smooth functions on compact intervals. Chebyshev polynomials are preferred in numerical practice because they minimize the maximum interpolation error (the Runge phenomenon) when interpolation nodes are placed at Chebyshev points rather than uniformly. Spline approximation pieces together low-degree polynomials over sub-intervals, providing control over smoothness at the joints and superior performance for piecewise smooth functions.

Radial basis functions (RBFs) extend these ideas to higher-dimensional domains by centering basis functions at scattered data points and measuring arguments by their distance from each center. Gaussian and multiquadric RBFs are common choices for scattered-data interpolation in two and three dimensions, applications that arise in geographic information systems, computational fluid dynamics mesh-free methods, and machine learning kernel methods.

Neural Network Approximation

Feedforward neural networks are general-purpose function approximators. The universal approximation theorem, formalized by Hornik, Stinchcombe, and White in 1989 and extended in subsequent work, states that a network with a single hidden layer of sufficient width and a nonlinear activation function can approximate any continuous function on a compact subset of Rn to arbitrary accuracy. A survey on universal approximation theorems covers extensions to deep networks, showing that depth provides exponential expressiveness advantages for certain function classes that shallow networks approximate only at the cost of exponential width.

Deep networks learn hierarchical representations in which each layer constructs a feature from outputs of the previous layer, enabling efficient approximation of compositionally structured functions. Deep neural network approximation theory analyzes how the number of parameters needed to approximate smooth functions scales with dimension and smoothness class, quantifying when deep networks overcome the curse of dimensionality that afflicts classical linear methods.

Error Analysis and Convergence

The quality of an approximation is characterized by error norms: the maximum (L-infinity) norm measures the worst-case deviation, while the mean-squared (L2) norm averages squared deviations over the domain. Convergence theory describes how quickly the error decreases as more basis functions or parameters are added. For smooth functions, polynomial approximation converges exponentially in the number of degrees, while neural network approximation rates depend on the smoothness of the target and the architecture of the network. Numerical function approximation notes from LSE cover practical error control strategies in the context of economic modeling, where high-dimensional policy function approximation is a common computational task.

Applications

Function approximation has applications in a range of fields, including:

• Reinforcement learning, where value functions and policy functions are approximated from sampled trajectories
• Scientific computing, including surrogate models for expensive simulations
• Signal processing, using Fourier and wavelet approximations for compression and denoising
• Control systems, where model-free controllers approximate unknown plant dynamics
• Computer graphics and CAD, using spline and NURBS representations of surfaces
• Robotics, for learning inverse kinematics and dynamics from sensor data