Support vector machines

What Are Support Vector Machines?

Support vector machines (SVMs) are a family of supervised learning algorithms that find a decision boundary by maximizing the geometric margin between classes in a feature space. The central idea is to identify the subset of training points, called support vectors, that are closest to the boundary, and to position the boundary as far from those points as possible. This maximum-margin principle, grounded in Vapnik's statistical learning theory and the concept of structural risk minimization, provides a theoretical basis for preferring wide-margin solutions: a larger margin reduces the upper bound on generalization error, making the model less likely to overfit when the number of training examples is limited relative to the dimensionality of the feature space.

SVMs were introduced in their modern form by Cortes and Vapnik in 1995 and quickly became one of the dominant methods in machine learning through the late 1990s and 2000s. They are closely related to feedforward neural networks, which also learn discriminative boundaries from labeled data, but SVMs differ in that their training reduces to a convex optimization problem with a unique global solution, avoiding the local minima that complicate neural network training. The arXiv survey on support vector machines and applications provides a thorough treatment of the mathematical foundations and the breadth of the SVM family.

Pattern Classification

Pattern classification is the primary application for which SVMs were developed and remain widely used. In binary classification, the SVM finds the hyperplane that maximally separates two classes in the input feature space or in a kernel-induced space. The scikit-learn SVM documentation describes how the soft-margin formulation, which admits bounded misclassifications at the cost of margin width, extends the method to the overlapping class distributions typical of real data. For multi-class problems, binary SVMs are combined through one-versus-one or one-versus-rest decomposition. SVMs achieved strong results on benchmark classification tasks in handwritten digit recognition, text categorization, and bioinformatics throughout the early 2000s, and they remain competitive on tasks with high-dimensional, low-sample-count data where deep learning methods require more examples to generalize well.

Support Vector Regression

SVMs extend naturally to regression tasks through an epsilon-insensitive loss formulation. In support vector regression (SVR), the goal is to find a function that deviates from the training targets by at most a tolerance epsilon while maintaining the flatness, or small weight norm, that corresponds to the margin concept in classification. Training points that fall within the epsilon tube around the regression function are not penalized; only points outside the tube become support vectors and influence the model. SVR is particularly useful when the noise in the target variable is bounded or when a smooth regression function is preferred over one that fits every training point closely. Applications include time series prediction, load forecasting in power systems, and financial price modeling.

Relevance Vector Machines

The relevance vector machine (RVM), introduced by Tipping in 2001, is a probabilistic reformulation of the SVM that produces sparse solutions with calibrated probability estimates rather than point predictions. Where an SVM selects support vectors through a margin constraint, the RVM uses a Bayesian automatic relevance determination framework to prune most training points from the model, retaining only those that are most informative. The result is typically sparser than an SVM and provides posterior probability distributions over predictions rather than bare class labels or regression values. RVMs are particularly valued in applications where uncertainty quantification matters and where the overhead of calibrating SVM outputs through Platt scaling or isotonic regression is undesirable.

Applications

Support vector machines are applied in fields where labeled data is limited relative to feature dimensionality or where well-calibrated decision boundaries are required, including: