Margins and support vectors
Logistic regression picks a separating line and stops. SVMs ask: which separating line is most *robust*?
Two classes separable by a straight line are separable by many straight lines — infinite families of them. Logistic regression stops at whichever one its training run happens to reach. Support vector machines (SVMs) ask a sharper question: among all the lines that classify the training set perfectly, which one leaves the widest cushion of empty space on both sides?
Why width is the point
The test point is likely near the training points. A boundary that threads close to points on both sides classifies them but with zero cushion — a whisker and the label flips. Maximum margin is built-in insurance: you're maximizing the least-confident prediction.
Tilt & shift the boundary — watch the cushion
margin 0.52 · misclassified 4
Big circles = current support vectors, the only points this boundary cares about.
Shaded band = the margin corridor the boundary bisects. Any separating position scores 0 errors, but the cushion differs: SVM’s rule is “push both sides to equal, maximal width.” Drive margin below 0.32 and a single noisy re-measurement could flip a label — that’s what the width is insurance against.
The machinery, at intuition resolution
The margin is measured by the distance from the boundary line to the nearest points on either side — the "support vectors" (they alone support the line; moving any other point slightly changes nothing, a property logistic regression decisively lacks). Training maximizes that distance under the constraint that everything classifies correctly.
The connection worth keeping: SVMs run margin maximization; logistic regression runs confidence shaping — both are convex, linear-in-weights models that differ mostly in which part of the same geometric picture they optimize. One honest difference matters in practice: logistic regression outputs probabilities you can threshold however you like; SVM outputs a side, full stop.
The one clever trick worth naming
Real data is rarely separable by anything you can draw. SVMs handle that with the kernel trick: rather than mapping data into some higher-dimensional space where a separating plane exists — a mapping which could be enormously expensive — kernels arrange for only the pairwise geometric relations of that space to be computed on demand. The optimization never leaves its linear comfort zone, but the points effectively do. Conceptually: "draw a circle classifier" becomes "fit a plane to a version of the features where circles are planes."
Historical context, honestly framed
SVMs ruled the 1990s and 2000s (before ensembles took tabular data, and long before deep networks took everything else); they're still a solid choice on small, clean, tabular datasets. The margin idea never went away, though: it resurfaced in deep-learning theory and is central to why max-margin methods and generalization are still linked in active research.
Where next: chapter 3 leaves model families behind and fixes training discipline — train/test splits.