What Are Relevance Vector Machines?

Relevance vector machines (RVMs) are probabilistic machine learning models for regression and classification that share the functional form of support vector machines but derive their predictions from a Bayesian framework rather than from convex optimization. Introduced by Michael E. Tipping in a 2001 paper published in the Journal of Machine Learning Research, the RVM places a hierarchical prior over the model weights that automatically drives the majority of them to zero, producing a sparse predictive model. Because sparsity emerges from the Bayesian inference process rather than from a margin maximization criterion, the RVM does not require a valid Mercer kernel, can produce calibrated probabilistic output directly, and avoids the need to specify a regularization hyperparameter such as the SVM's C. The RVM is situated within the broader field of sparse Bayesian learning, which applies automatic relevance determination (ARD) to models that are linear in their parameters.

The term "relevance vector" refers to the small subset of training examples whose corresponding weights survive the sparsification: these are the data points the model deems relevant for prediction, analogous to the support vectors of an SVM but typically far fewer in number.

Sparse Bayesian Learning and Automatic Relevance Determination

The Bayesian treatment of the RVM begins with a model in which each weight has its own independent Gaussian prior controlled by a hyperparameter. Variational or type-II maximum likelihood (evidence approximation) inference is then applied to optimize these hyperparameters, and the procedure drives the hyperparameter for each irrelevant weight to infinity, effectively setting that weight to zero. This automatic relevance determination mechanism was developed by David MacKay and Radford Neal for neural networks in the early 1990s and adapted by Tipping to kernel-based models. The result is a model that selects its own relevant basis functions from the training set without cross-validation over a sparsity parameter. As the original Tipping (2001) paper in JMLR demonstrates, this sparse Bayesian learning framework yields models that use dramatically fewer basis functions than a comparable SVM trained on the same data.

Comparison with Support Vector Machines

The RVM and the SVM occupy complementary positions in the space of kernel learning methods. SVMs find a maximum-margin hyperplane by solving a quadratic program, with sparsity arising from the Karush-Kuhn-Tucker conditions: only the training examples lying on or inside the margin boundary become support vectors. SVMs provide excellent generalization bounds through statistical learning theory but produce deterministic point predictions, require a valid Mercer kernel, and use the hinge loss, which does not directly yield class probabilities. The RVM produces a full posterior distribution over its predictions, allowing confidence intervals and risk-sensitive decision making, but training via Bayesian inference is iterative and computationally more expensive than SVM quadratic programming for large datasets. The original Tipping JMLR paper on sparse Bayesian learning discusses this comparison in detail, noting that on benchmark classification tasks the RVM typically uses an order of magnitude fewer basis functions than the SVM with comparable test accuracy.

Model Training and Implementation

RVM training iterates between updating the posterior distribution over weights (a Gaussian inference step) and updating the hyperparameters by maximizing the marginal likelihood. The original sequential sparse Bayesian learning algorithm adds basis functions one at a time, selecting at each step the candidate that most increases the marginal likelihood, which allows the algorithm to scale to larger datasets than a batch approach. Tipping's SparseBayes MATLAB implementation is the canonical reference implementation and has been used in numerous research comparisons. Extensions of the RVM include multi-class formulations, online and incremental training algorithms, and adaptations for structured prediction tasks in natural language processing and image analysis.

Applications

Relevance vector machines have applications in a range of fields, including:

• Medical diagnosis and prognosis, where probabilistic outputs support clinical risk scoring
• Remote sensing and hyperspectral image classification requiring sparse feature selection
• Structural health monitoring and prognostics, predicting remaining useful life of components
• Financial risk modeling where calibrated probability estimates are required
• Bioinformatics tasks such as gene expression classification with high-dimensional sparse features