The Question SVM Asks
k-Nearest Neighbors trusted your local crowd. A Support Vector Machine does the opposite: it hunts for the single best dividing line.
But many lines can separate two classes. Which is best? A line that barely squeezes past the points is fragile, one new point just over the edge gets misclassified. SVM’s answer is bold and simple:
Pick the line with the widest possible gap between the two classes.
The Margin Is a Street
Don’t picture the boundary as a thin line. Picture it as a street: the widest street you could pave between the two classes without a single point sitting on the road. The dividing line is just the centre of that street, and the street’s width is the margin.
SVM makes that street as wide as it can. Why? More clearance on both sides means more room for error on new points. A wide margin generalizes best.
Support Vectors
Here is the elegant part. Only the points touching the edges of the street actually matter. They are the support vectors, they hold the boundary up.
Every other point, resting comfortably deep in its own territory, is irrelevant. Delete them all and the boundary would not move a hair. Only the borderline points count, which is exactly why an SVM is defined by just a handful of them, and where the name comes from.
Soft Margin: When Classes Overlap
Real data is messy, and classes overlap. A perfectly clean street often does not exist. So SVM permits a few violations, points inside the street or even on the wrong side, for a penalty. A knob called C sets the trade:
| Effect | |
|---|---|
| High C | tolerate almost no violations → narrow street, fits training tightly (risks overfitting) |
| Low C | allow more violations → wider, sturdier street (more robust) |
Yet another dial on the bias-variance tradeoff.
The Kernel Trick
Now the beautiful idea. What if no straight line can separate the classes at all? The classic case: one class is a blob in the centre, the other a ring wrapped around it. No line in 2D can split those.
The trick is to lift the data into a higher dimension where it does become separable by a flat plane. Give every point a new coordinate equal to its distance from the centre. Now the inner blob sits low and the outer ring sits high, and a flat plane slides right between them. Project that plane back down to 2D, and it becomes a circle.
And here’s the “trick”: an SVM never actually computes those higher-dimensional coordinates. A kernel function computes the needed similarities directly, as if the data had been lifted, but far more cheaply. So it can draw circles, wiggles, almost any shape, while still just “finding a straight line” in some hidden space.
Different kernels give different boundary shapes (polynomial, or the popular Gaussian “RBF” kernel). This is what made SVMs so powerful.
Where SVMs Sit
Before deep learning, kernel SVMs were often the best classifiers around, especially on medium-sized, high-dimensional problems like text and bioinformatics. They are precise, principled, and defined by only their support vectors.
They lost the crown on huge datasets (they are slow to train there, and deep networks pulled ahead), but the ideas, margins, support vectors, and kernels, echo throughout machine learning.
We’ve now drawn boundaries with lines, curves, probabilities, and neighbours. Next comes a completely different style: a classifier that asks a chain of plain yes/no questions, the decision tree, and the surprisingly powerful ensembles built by combining many of them.