Wiki
Advanced6 min read

Margins and support vectors

Logistic regression picks a separating line and stops. SVMs ask: which separating line is most *robust*?

Two classes separable by a straight line are separable by many straight lines — infinite families of them. Logistic regression stops at whichever one its training run happens to reach. Support vector machines (SVMs) ask a sharper question: among all the lines that classify the training set perfectly, which one leaves the widest cushion of empty space on both sides?

Why width is the point

The test point is likely near the training points. A boundary that threads close to points on both sides classifies them but with zero cushion — a whisker and the label flips. Maximum margin is built-in insurance: you're maximizing the least-confident prediction.

Tilt & shift the boundary — watch the cushion

margin 0.70 · misclassified 0

Big circles = current support vectors, the only points this boundary cares about.

Shaded band = the margin corridor the boundary bisects. Any separating position scores 0 errors, but the cushion differs: SVM’s rule is “push both sides to equal, maximal width.” Drive margin below 0.32 and a single noisy re-measurement could flip a label — that’s what the width is insurance against.

Illustrative vs real

The artifact is a hand-steered line over twelve points with the margin computed from exact distances. Real SVM training is a convex optimization solved globally — a soft-margin constant trades violations against width, and the kernel emits pairwise distances instead of a hand-drawn line. What the toy gets faithfully right: the support vectors really are the only points the boundary depends on; nudge any other point and nothing changes.

The machinery, at intuition resolution

The margin is measured by the distance from the boundary line to the nearest points on either side — the "support vectors" (they alone support the line; moving any other point slightly changes nothing, a property logistic regression decisively lacks). Training maximizes that distance under the constraint that everything classifies correctly.

The connection worth keeping: SVMs run margin maximization; logistic regression runs confidence shaping — both are convex, linear-in-weights models that differ mostly in which part of the same geometric picture they optimize. One honest difference matters in practice: logistic regression outputs probabilities you can threshold however you like; SVM outputs a side, full stop.

The one clever trick worth naming

Real data is rarely separable by anything you can draw. SVMs handle that with the kernel trick: rather than mapping data into some higher-dimensional space where a separating plane exists — a mapping which could be enormously expensive — kernels arrange for only the pairwise geometric relations of that space to be computed on demand. The optimization never leaves its linear comfort zone, but the points effectively do. Conceptually: "draw a circle classifier" becomes "fit a plane to a version of the features where circles are planes."

Historical context, honestly framed

SVMs ruled the 1990s and 2000s (before ensembles took tabular data, and long before deep networks took everything else); they're still a solid choice on small, clean, tabular datasets. The margin idea never went away, though: it resurfaced in deep-learning theory and is central to why max-margin methods and generalization are still linked in active research.

From the exam paper

Exam-modeled · End-Sem December 2024

0 / 5 answered

  1. 1In a maximum-margin classifier, which training points actually determine the position of the boundary?
    Multiple choice
  2. 2What are the data points that lie on the margin boundary called?
    Short answer
  3. 3XOR is a linearly separable function and can be implemented by a single linear decision boundary.
    True / false
  4. 4In a soft-margin SVM, increasing the penalty C on margin violations tends to:
    Multiple choice
  5. 5The kernel trick explicitly computes the coordinates of each point in a high-dimensional feature space.
    True / false

Where next: chapter 3 leaves model families behind and fixes training discipline — train/test splits.