Linear regression, properly
The line from chapter 1, but fit by machinery: the closed form, the gradient descent view, and what 'earning' a linear model means.
Chapter 1 fit a line with your fingers. Time to make the machine do it — two different routes to the same answer, and both are worth knowing because a thousand later techniques are one of these two routes wearing a costume.
Route 1: the closed form
For squared-error lines there's an exact formula (least squares):
w = Σ(xᵢ−x̄)(yᵢ−ȳ) / Σ(xᵢ−x̄)² and then b = ȳ − w·x̄. No search, no
iterations — you plug in and out pops the optimum. That's rare and
precious: it works because squared-loss linear regression is
convex — a bowl with exactly one bottom, so "solve this equation"
and "descend to the bottom" agree perfectly.
When there's no formula
The moment your model becomes non-linear (polynomial features, trees, neural nets) the closed form disappears and route 2 is all you have. Formulas are a subsidy convexity pays.
Route 2: gradient descent doing chapter 1's tax work
The iterable view: start anywhere, compute the gradient of the squared error with respect to w and b, step downhill. Watch it converge:
Gradient descent from a bad start, iterating toward w=1.03, b=1.55 (the closed form)
w 0.10 · b 4.00
loss 81.6
descend: run it
~220 unconscious iterations replace your two sliders. The step rule (parameters −= rate × gradient) is identical to the landscape ball — convexity just guarantees every route lands in the same place, which is why we say linear regression “has a solution” at all. Real solvers (Adam) damp smarter; the destination is the same.
Each iteration improves the line a little; near the bowl's floor the steps get small because the slope gets small — that's gradient descent self-damping, no schedule needed on a clean convex loss.
What the line means — and earns
The learned w is a claim: "each extra km of distance adds about w
extra minutes, holding everything else fixed." Two honesty checks
before you say it to anyone:
- Correlation, not causation. The line reproduces co-movement in your training data; it says nothing about what changing x would do. Confounders (weather, driver skill) live inside that w.
- Extrapolation is risk. A line fit between 1 and 8 km knows nothing at 80 km — the degree-12 disaster of the last chapter, in its mildest form. Interpolate with confidence, extrapolate with a lawyer.
And when a line isn't enough?
Polynomials add curvature but stay rigid. Two genuinely different escapes open chapters 2's rest: change the shape (neural-style features, later books) or change the geometry — which is where logistic regression sneaks in next: a line that cuts space in half instead of tracing it.
Illustrative vs real
Real regression also carries per-feature weights (10, 1000 columns) and usually a regularization term added to the loss (chapter 3); the closed-form formula generalizes to that case with one matrix inverse.
Where next: logistic regression — the same machinery turned to classification.