Wiki
Core8 min read

Logistic regression

A line that cuts space in half, a sigmoid that turns distance into probability, and the cross-entropy loss that sharpens it.

Regression predicts numbers. But most questions worth asking are yes/no: will this student pass, will this email get clicked, is this mushroom toxic? Logistic regression answers those with the same tool — a straight line — read in a completely different way.

The geometry

Draw a line through a scatter of two classes. The line is the classifier: one side → class A, the other → class B. Distance from the line = confidence. That's logistic regression in one picture.

One boundary, two classes — slide it and watch confidence, errors, and margin

misclassified: 0 · margin (closest point): 1.10

ringed points sit near the boundary — low confidence. Left cloud: accent, right: foreground.

At ±2 you misclassify many points; the honest optimum balances *zero-ish errors* against a healthy margin. Cross-entropy finds a value in between automatically, by weighted distance — nothing here is one “accuracy” knob, which is exactly what made the loss chapter’s argument. This is a linear boundary in a place a curved, spiky world can trust — the ceiling is real.

How the probability works out

  1. Compute the line's raw score z = w·x + b — a signed number when distance from the boundary.
  2. Squash into 0..1 with the sigmoid: P(class B) = 1/(1+e^(−z)). Far on the B side → near 1; far on the A side → near 0; sitting on the line → coin flip.

Why that particular squashing? Because cross-entropy (the classifier loss from chapter 1) rewards calibrated likelihood: predicted probability gets close to 1 for true-B points and to 0 for true-A points, and the gradient of that loss pushes weights by how wrong the confidence was. Trained to convergence, the boundary ends up flanked by high-confidence regions — the smooth ramp in the artifact is distance-as-confidence made visible.

Which part is regression, then?

The name confuses everyone. It's regression in the machinery (a linear score, fitted by gradient descent, one set of weights per feature) and classification in the answer (the sigmoid output is a probability, not a quantity). It is the workhorse of classical ML: cheap, convex, and its weights read off directly — "one point of lag increases churn odds by e^w" — which is why analysts, biostatisticians, and fraud engineers alike defend it against flashier rivals.

Where it hits its ceiling

Everything is decided by that single boundary; if two classes can't be separated by a straight line — the XOR curse, the spiral — no amount of training will fix it. Nearest neighbors (next lesson) will happily carve curved, disconnected territory — at other costs. This flexibility-vs-simplicity trade is the same bias/variance dial from chapter 1, wearing yet another costume.

Illustrative vs real

The artifact uses two features and a fixed sweep of boundaries. Real logistic regression fits by cross-entropy gradient descent and, again, usually with a regularization penalty — overfitting happens here too.

Check yourself

Eduspheria wiki · Classical Machine Learning, Supervised learning

0 / 5 answered

  1. 1What is the role of the sigmoid in logistic regression?
    Multiple choice
  2. 2Which loss sharpens logistic regression by rewarding calibrated likelihood?
    Short answer
  3. 3The linear score is z = 0. What probability does the sigmoid output?
    Numeric answer
  4. 4Logistic regression can separate classes that are not linearly separable, such as XOR.
    True / false
  5. 5Why is logistic regression called regression if it is used for classification?
    Multiple choice

From the assignment paper

Exam-modeled · Assignment/Quiz

0 / 5 answered

  1. 1In the ADALINE model, which learning rule adjusts the weights during training?
    Multiple choice
  2. 2The Perceptron learning algorithm is guaranteed to converge only when the training data is linearly separable.
    True / false
  3. 3A Perceptron update is w_new = w + eta times (target minus output) times input. With w = 0.5, learning rate 0.1, target 1, output 0 and input 2, what is w_new?
    Numeric answer
  4. 4Which activation function does a Perceptron use at its output?
    Multiple choice
  5. 5A network trained with backpropagation fails to converge and its cost oscillates. What is the most likely cause?
    Multiple choice

Where next: nearest neighbors — break the straight-line tyranny entirely.