A support vector machine (SVM) is a supervised learning model that finds the decision boundary separating two classes with the largest possible margin, measured to the nearest training points, which are called support vectors. In machine learning, the SVM is one of the classic algorithms for classification and regression, and it remains a reliable baseline across artificial intelligence applications where data sets are small to medium and features are well defined.
The method rests on a clear geometric idea and a convex optimization problem, so training always reaches the global optimum rather than a local one. Combined with the kernel trick, a support vector classifier can draw highly nonlinear boundaries while still solving a problem that is mathematically linear in a transformed feature space.
How a support vector machine finds the maximum margin
An SVM separates classes by placing a hyperplane so that the distance to the closest points of each class is as large as possible. For a linear model, the hyperplane is defined by a weight vector w and a bias b, and a new point is classified by the sign of w·x + b. The margin equals 2 divided by the norm of w, so maximizing the margin is equivalent to minimizing the squared norm of the weights.
Only the points that lie on or inside the margin influence the final solution. These support vectors usually form a small fraction of the training set, which is why the trained model is compact and why removing any other point leaves the boundary unchanged. The optimization is a convex quadratic program, typically solved in its dual form, where the data appear only as dot products between pairs of points.
Real data rarely separate perfectly, so practical SVMs use a soft margin. Slack variables allow some points to fall inside the margin or on the wrong side, and the hinge loss penalizes each violation in proportion to its distance from the correct side. The regularization parameter C controls how strongly those violations are punished relative to the width of the margin.
The following table summarizes the main components of the model and the role each one plays.
| Component | What it is | Role in the model |
|---|---|---|
| Hyperplane | Linear boundary w·x + b = 0 | Separates the two classes |
| Margin | Gap between the boundary and nearest points | Wider margin means better generalization |
| Support vectors | Points on or inside the margin | Alone determine the boundary |
| Slack variables | Allowed margin violations | Make noisy data trainable |
| Hinge loss | max(0, 1 minus y times the score) | Penalizes misclassified or close points |
Kernels and nonlinear decision boundaries
A kernel lets an SVM learn a nonlinear decision boundary by computing dot products in a higher-dimensional feature space without ever building that space explicitly. Because the dual problem depends only on dot products, replacing them with a kernel function K(x, z) is enough; this substitution is the kernel trick, and it gets its own entry in the same Topic.
The choice of kernel defines the shape of boundaries the model can express. The radial basis function (RBF) kernel is the usual default, but each kernel brings its own parameters and assumptions, as the table below shows.
| Kernel | Formula idea | Key parameters | Typical use |
|---|---|---|---|
| Linear | Plain dot product | C only | Text, sparse or very wide data |
| Polynomial | (γ x·z + r) to the power d | Degree d, γ, r | Feature interactions of known order |
| RBF (Gaussian) | exp(minus γ times squared distance) | γ and C | General nonlinear default |
| Sigmoid | tanh(γ x·z + r) | γ, r | Rarely used; not always valid |
With the RBF kernel, gamma (γ) sets how far the influence of a single training point reaches. A small gamma produces smooth, almost linear boundaries; a large gamma lets each support vector shape a small local region, which quickly leads to overfitting. C and gamma interact strongly, so they are tuned together on a logarithmic grid, for example C from 0.01 to 1,000 and gamma from 0.0001 to 10.
SVM compared with related classifiers
An SVM differs from neighboring methods mainly in its loss function and in how it handles nonlinearity. Logistic regression minimizes log loss and produces calibrated probabilities, while the SVM minimizes hinge loss and produces a distance-based score that is not a probability by default.
| Method | Loss or rule | Nonlinearity | Probability output |
|---|---|---|---|
| Support vector machine | Hinge loss with margin | Through kernels | Only with Platt scaling |
| Logistic regression | Log loss | Manual features | Native and calibrated |
| k-nearest neighbors | Majority vote of neighbors | Implicit and local | Vote proportions |
| Random forest | Averaged decision trees | Built in | Tree vote fractions |
| Gradient boosting | Sequential residual fitting | Built in | Native after link function |
For multi-class classification, an SVM is extended with decomposition schemes: one-vs-rest trains one classifier per class, and one-vs-one trains a classifier for every pair of classes and lets them vote. A regression variant, support vector regression (SVR), fits a tube of width epsilon around the data and ignores errors smaller than that tolerance.
Training and tuning an SVM in practice
A support vector machine performs well only when its inputs and hyperparameters are prepared carefully. Feature scaling is mandatory, because both the margin and the RBF distance are computed in raw feature units; an unscaled feature measured in thousands will dominate one measured in fractions.
The decision table below gives practical starting points for common situations.
| Situation | Recommended setup | Reason |
|---|---|---|
| Many features, few samples | Linear kernel | Data are often already separable |
| Under 100,000 samples, dense features | RBF kernel with grid search | Flexible and well understood |
| Millions of samples | Linear SVM with stochastic solver | Kernel training scales poorly |
| Imbalanced classes | Class weights on C | Prevents the majority class from dominating |
| Probabilities needed | Platt scaling on held-out data | Raw scores are not calibrated |
Hyperparameters should be selected with cross-validation on training data only, ideally inside a pipeline that fits the scaler on each training fold. Fitting the scaler on the full data set first is a common source of data leakage that inflates validation scores.
Limitations and common mistakes with SVMs
The main limitation of kernel SVMs is computational cost. Training time grows roughly between the square and the cube of the number of samples, and the kernel matrix must fit in memory, so data sets beyond a few hundred thousand rows usually call for a linear SVM, an approximation such as random Fourier features, or a different algorithm.
- Skipping feature scaling distorts distances and produces poor or unstable boundaries.
- Tuning C and gamma separately misses the region where the two parameters work well together.
- Treating decision scores as probabilities gives misleading confidence values without calibration.
- Expecting interpretability from an RBF model fails, because its boundary is defined by many support vectors rather than readable weights.
- Using kernels on raw images or audio usually underperforms deep networks that learn their own features.
A high share of support vectors, for example more than half of the training set, is a useful warning sign. It often indicates heavy class overlap, poorly scaled inputs or a value of C that is too low for the problem.
The simple explanation
Imagine two groups of colored marbles scattered on a table and the task of laying a ruler between them. Many positions would separate the groups, but the safest one leaves the widest empty corridor on both sides, so a new marble that lands slightly off is still sorted correctly.
A support vector machine does exactly that with data. It looks only at the marbles closest to the dividing line, the support vectors, and ignores the rest. When the groups cannot be split by a straight line, the method bends the line by viewing the data from a different angle. The key idea is simple: a wide safety margin makes predictions more reliable.
How it is used in practice
A quality engineer at an electronics plant needs to flag faulty solder joints from 24 measured features, such as joint height, reflectance and resistance. The team has about 8,000 labeled joints, of which only 4 percent are defective, which is a typical small, structured and imbalanced data set.
The engineer standardizes every feature, trains an RBF support vector machine with balanced class weights, and searches C and gamma with five-fold cross-validation. The best model uses C of 10 and gamma of 0.05 and relies on about 900 support vectors. Platt scaling is then fitted on a held-out set so that operators see a defect probability rather than a raw score.
On fresh production data, the model catches about 92 percent of defective joints while keeping false alarms low enough for a single inspector to review flagged boards. Because the model is small and fast, it runs on the line computer without dedicated hardware.