What Are Gaussian Processes?

Gaussian processes are probabilistic models that define a distribution over functions, where any finite collection of function values follows a multivariate Gaussian distribution. Rather than specifying a parametric form for a function and estimating its parameters, a Gaussian process treats the function itself as the unknown, expressing prior beliefs about its smoothness and structure through a covariance function called the kernel. This approach provides both predictions at unobserved inputs and calibrated uncertainty estimates at every predicted point, which distinguishes Gaussian processes from most deterministic regression and classification methods.

The theoretical foundations of Gaussian processes draw from stochastic process theory and Bayesian inference. Their application to machine learning was systematized in the 2006 textbook by Carl Rasmussen and Christopher Williams, Gaussian Processes for Machine Learning, which remains the standard reference and is freely available online. Gaussian processes connect naturally to kernel methods such as support vector machines and spline interpolation, and they occupy a central position in Bayesian nonparametric statistics.

Inference and Regression

In Gaussian process regression, a prior distribution over functions is specified by a mean function, usually taken as zero, and a covariance kernel that encodes assumptions about the function's behavior. Common kernels include the squared exponential (also called the radial basis function kernel), which enforces infinitely smooth functions, and the Matern family, which allows control over differentiability. Given observed training data, the prior is updated using Bayes' rule to produce a posterior distribution over functions. The posterior mean provides point predictions, while the posterior variance quantifies prediction uncertainty.

Inference under a Gaussian likelihood is analytically tractable and reduces to solving a linear system involving the kernel matrix evaluated on the training inputs. The computational cost scales as the cube of the number of training points, which limits direct application to datasets of more than a few thousand points. Sparse approximations, such as inducing point methods, reduce this cost substantially and are described in the scikit-learn Gaussian process documentation alongside practical implementation guidance.

Learning and Kernel Selection

The kernel determines the qualitative properties of the functions a Gaussian process can represent, including smoothness, periodicity, and length scale. Kernel hyperparameters, such as the length scale and output variance, are typically learned from data by maximizing the marginal likelihood, also called the evidence. This procedure automatically balances model complexity against data fit, analogously to regularization in frequentist regression.

Combining kernels through addition or multiplication allows practitioners to encode structured assumptions. For example, a product of a periodic kernel with a squared exponential produces a kernel suited to quasi-periodic data where the period slowly drifts. The connection between Gaussian processes and kernel methods, including their equivalence to ridge regression and support vector regression under certain formulations, is reviewed in an arxiv survey on Gaussian processes and kernel methods.

Prediction and Uncertainty Quantification

One of the main reasons Gaussian processes are used in scientific and engineering applications is the principled uncertainty quantification they provide. Every prediction comes with a credible interval, and the width of that interval reflects both the distance from training data and the noise level in the observations. This property is valuable in experimental design and active learning, where the next measurement location can be chosen to maximize information gain or minimize prediction variance.

Gaussian processes underlie Bayesian optimization, a method for optimizing expensive black-box functions by building a surrogate model and selecting evaluation points using an acquisition function. Bayesian optimization has been applied to hyperparameter tuning in machine learning, materials discovery, and the optimization of engineering simulations.

Applications

Gaussian processes have applications in a wide range of disciplines, including:

• Bayesian optimization for hyperparameter tuning and black-box function optimization
• Geostatistical modeling and kriging for spatial interpolation
• Time series prediction with calibrated uncertainty in finance and climate modeling
• Reinforcement learning, where GP models represent reward or value functions
• Scientific emulation of computationally expensive simulations in aerospace and materials science