Wiki
Core7 min read

Linear regression, properly

The line from chapter 1, but fit by machinery: the closed form, the gradient descent view, and what 'earning' a linear model means.

Chapter 1 fit a line with your fingers. Time to make the machine do it — two different routes to the same answer, and both are worth knowing because a thousand later techniques are one of these two routes wearing a costume.

Route 1: the closed form

For squared-error lines there's an exact formula (least squares): w = Σ(xᵢ−x̄)(yᵢ−ȳ) / Σ(xᵢ−x̄)² and then b = ȳ − w·x̄. No search, no iterations — you plug in and out pops the optimum. That's rare and precious: it works because squared-loss linear regression is convex — a bowl with exactly one bottom, so "solve this equation" and "descend to the bottom" agree perfectly.

When there's no formula

The moment your model becomes non-linear (polynomial features, trees, neural nets) the closed form disappears and route 2 is all you have. Formulas are a subsidy convexity pays.

Route 2: gradient descent doing chapter 1's tax work

The iterable view: start anywhere, compute the gradient of the squared error with respect to w and b, step downhill. Watch it converge:

Gradient descent from a bad start, iterating toward w≈1.17, b≈1.31 (the closed form)

w 0.10 · b 4.00
loss 81.6

descend: run it

~220 unconscious iterations replace your two sliders. The step rule (parameters −= rate × gradient) is identical to the landscape ball — convexity just guarantees every route lands in the same place, which is why we say linear regression “has a solution” at all. Real solvers (Adam) damp smarter; the destination is the same.

Each iteration improves the line a little; near the bowl's floor the steps get small because the slope gets small — that's gradient descent self-damping, no schedule needed on a clean convex loss.

What the line means — and earns

The learned w is a claim: "each extra km of distance adds about w extra minutes, holding everything else fixed." Two honesty checks before you say it to anyone:

  1. Correlation, not causation. The line reproduces co-movement in your training data; it says nothing about what changing x would do. Confounders (weather, driver skill) live inside that w.
  2. Extrapolation is risk. A line fit between 1 and 8 km knows nothing at 80 km — the degree-12 disaster of the last chapter, in its mildest form. Interpolate with confidence, extrapolate with a lawyer.

And when a line isn't enough?

Polynomials add curvature but stay rigid. Two genuinely different escapes open chapters 2's rest: change the shape (neural-style features, later books) or change the geometry — which is where logistic regression sneaks in next: a line that cuts space in half instead of tracing it.

Illustrative vs real

Real regression also carries per-feature weights (10, 1000 columns) and usually a regularization term added to the loss (chapter 3); the closed-form formula generalizes to that case with one matrix inverse.

Check yourself

Eduspheria wiki · Classical Machine Learning, Supervised learning

0 / 5 answered

  1. 1What makes the closed-form least-squares solution possible?
    Multiple choice
  2. 2What is the exact formula-based route to the best squared-error line called?
    Short answer
  3. 3For the points (1, 2) and (3, 6), what is the least-squares slope w?
    Numeric answer
  4. 4A large learned weight proves that changing that feature causes the outcome to change.
    True / false
  5. 5The lesson warns that extrapolating a line fit between 1 and 8 km to 80 km is:
    Multiple choice

Where next: logistic regression — the same machinery turned to classification.