Logistic regression
A line that cuts space in half, a sigmoid that turns distance into probability, and the cross-entropy loss that sharpens it.
Regression predicts numbers. But most questions worth asking are yes/no: will this student pass, will this email get clicked, is this mushroom toxic? Logistic regression answers those with the same tool — a straight line — read in a completely different way.
The geometry
Draw a line through a scatter of two classes. The line is the classifier: one side → class A, the other → class B. Distance from the line = confidence. That's logistic regression in one picture.
One boundary, two classes — slide it and watch confidence, errors, and margin
misclassified: 0 · margin (closest point): 1.10
ringed points sit near the boundary — low confidence. Left cloud: accent, right: foreground.
At ±2 you misclassify many points; the honest optimum balances *zero-ish errors* against a healthy margin. Cross-entropy finds a value in between automatically, by weighted distance — nothing here is one “accuracy” knob, which is exactly what made the loss chapter’s argument. This is a linear boundary in a place a curved, spiky world can trust — the ceiling is real.
How the probability works out
- Compute the line's raw score
z = w·x + b— a signed number when distance from the boundary. - Squash into 0..1 with the sigmoid:
P(class B) = 1/(1+e^(−z)). Far on the B side → near 1; far on the A side → near 0; sitting on the line → coin flip.
Why that particular squashing? Because cross-entropy (the classifier loss from chapter 1) rewards calibrated likelihood: predicted probability gets close to 1 for true-B points and to 0 for true-A points, and the gradient of that loss pushes weights by how wrong the confidence was. Trained to convergence, the boundary ends up flanked by high-confidence regions — the smooth ramp in the artifact is distance-as-confidence made visible.
Which part is regression, then?
The name confuses everyone. It's regression in the machinery (a linear score, fitted by gradient descent, one set of weights per feature) and classification in the answer (the sigmoid output is a probability, not a quantity). It is the workhorse of classical ML: cheap, convex, and its weights read off directly — "one point of lag increases churn odds by e^w" — which is why analysts, biostatisticians, and fraud engineers alike defend it against flashier rivals.
Where it hits its ceiling
Everything is decided by that single boundary; if two classes can't be separated by a straight line — the XOR curse, the spiral — no amount of training will fix it. Nearest neighbors (next lesson) will happily carve curved, disconnected territory — at other costs. This flexibility-vs-simplicity trade is the same bias/variance dial from chapter 1, wearing yet another costume.
Illustrative vs real
The artifact uses two features and a fixed sweep of boundaries. Real logistic regression fits by cross-entropy gradient descent and, again, usually with a regularization penalty — overfitting happens here too.
Where next: nearest neighbors — break the straight-line tyranny entirely.