Support vector machine classification

What Is Support Vector Machine Classification?

Support vector machine (SVM) classification is a supervised learning method that assigns data points to discrete categories by finding the hyperplane in a feature space that maximally separates the classes. Unlike classifiers that simply seek any boundary that correctly separates training examples, an SVM explicitly maximizes the margin, the perpendicular distance between the decision boundary and the nearest training points from each class. Those nearest points are the support vectors, and the resulting maximum-margin boundary generalizes better to unseen data than boundaries that fit the training set more tightly with a narrower margin.

The method originated in work by Vladimir Vapnik in the 1960s on linear classifiers and was extended into its modern form in 1992 and 1995 when Boser, Guyon, and Vapnik introduced the kernel trick and Cortes and Vapnik published the soft-margin formulation that tolerates misclassifications in exchange for a wider margin. SVM classification is grounded in Vapnik's statistical learning theory and the principle of structural risk minimization, which bounds generalization error by controlling model complexity. A broad mathematical treatment of SVM theory and applications is available in the arXiv survey on support vector machines.

Decision Boundaries and Margin Maximization

The core optimization problem in SVM classification is to find the weight vector and bias that define a separating hyperplane while maximizing the margin. For linearly separable data, this is a convex quadratic programming problem whose solution involves only the support vectors, the subset of training points that lie on the margin boundaries. The practical importance of this sparsity is that the classifier's complexity depends on the number of support vectors, not the total number of training examples, which allows SVMs to generalize well even in high-dimensional feature spaces with relatively few training examples. When classes are not perfectly separable, the soft-margin SVM introduces slack variables that permit violations of the margin constraint at a cost controlled by a regularization parameter, trading off margin width against training set accuracy.

Kernel Methods

The kernel trick extends SVM classification to nonlinearly separable data without explicitly computing a transformation into a higher-dimensional feature space. A kernel function computes the inner product between two data points in a transformed space from the original representation, allowing the SVM to fit nonlinear decision boundaries in the original feature space while solving only a linear problem in the transformed space. Common kernels include the polynomial kernel, the radial basis function (RBF) kernel, and the sigmoid kernel. The choice of kernel and its hyperparameters has a large effect on classification performance, and cross-validation is typically used to select them. The scikit-learn documentation on support vector machines provides a practical reference on kernel selection guidelines and their computational trade-offs, covering both the mathematical foundations and implementation considerations.

Multi-class Classification

The standard SVM formulation is binary: it assigns inputs to one of two classes. Extending SVMs to problems with three or more classes is done through decomposition strategies. The one-versus-rest (OVR) approach trains one binary SVM for each class against all others and assigns a test point to the class whose classifier reports the highest confidence. The one-versus-one (OVO) approach trains a binary classifier for every pair of classes and resolves conflicts by majority vote. For k classes, OVO requires k(k-1)/2 classifiers, which becomes expensive for large k, while OVR requires only k classifiers but each must separate one class from all others. Both strategies are used in practice, with OVO often preferred for SVMs because each binary subproblem is balanced and typically easier to solve.

Applications

Support vector machine classification has been applied across a wide range of fields where high-dimensional feature spaces and limited labeled data make other classifiers less effective, including: