Intro6 min read

A neuron

Logistic regression wearing a helmet: one weighted sum, one bend. Start here and depth becomes learnable instead of mysterious.

Strip a neural network down to its smallest working part and you get something you have already met. A neuron takes its inputs, multiplies each by a learned weight, adds a learned bias — and then, crucially, squashes the result through a nonlinear function before passing it on.

y = \sigma(w_1 x_1 + w_2 x_2 + \dots + w_n x_n + b)

That's it. The whole unit.

It's logistic regression in disguise

You've already fit this exact function with classical ML — the sigmoid over a weighted sum. A neuron is a logistic-regression unit whose output feeds into other units instead of stopping. Deep learning is "logistic regression, composing with itself a thousand times deep." That's not snark; it's the entire trick. There is weight-free "structure" hiding here, just a lot of small learning machines voting.

One neuron: score = w₁·x + w₂·y + b, class = red if score > 0.

w₁ (x weight)0.8w₂ (y weight)1.0b (bias)-0.5

accuracy: 100%

Outlined dots = misclassified. Perfect separation — and notice it's ONE straight crease. This neuron's full repertoire is exactly this.

The dashed line is where score = 0; slope = −w₁/w₂, offset set by the bias. Illustrative: threshold at 0 with deterministic points; training would find (w, b) by gradient descent on a loss — the artifact lets you be SGD by hand.

The part that matters: the bend

Without the nonlinearity $\sigma$ , here's the awkward truth: stacking linear layers composes straight lines into... another straight line. A hundred layers of $W_2 (W_1 x) = (W_2 W_1) x$ — one multiply. Capacity: identical to a single layer. Depth is pure overhead unless something nonlinear happens between the matmuls.

The bend is cheap. ReLU — rectified linear unit — is just $\max(0, z)$ : negative? Zero. Positive? Exact same value. That's the entire operation, and it's the default inside almost every deep net being trained today.

“Nonlinear” doesn't mean complicated

People hear nonlinearity and picture exotic transcendental functions. ReLU is a corner: $z < 0 \mapsto 0$ , else $z$ . Its virtue is that it is piecewise linear nearly everywhere — meaning big ReLU nets behave like smoothly-folded pieces of straight lines, which is exactly what the next lesson's visualization will make visible.

How it learns

Same rule you know. Compute a loss, take the gradient with respect to $w_i$ and $b$ , step downhill:

\frac{\partial y}{\partial w_i} = \sigma'(z) \cdot x_i

That chain rule — output gradient times input signal — is literally the backpropagation step from the LLM training chapter, applied to one neuron. Depth is about composing many of these; training is still this loop.

Illustrative vs real

The artifact shows one neuron separating a 2-D toy — a boundary you can see. Real neurons operate on thousands of inputs inside layers of thousands of neurons. But the math (weighted sum → nonlinearity → gradient step) is the same; only the blindfold grows.

Where next: activation-bends — why those "small cheap bends" become folds in space, and why folding is the entire superpower.

This lesson has exercises attached — mix-and-match weight/bias intuition on the decision boundary — launching once the exercises layer ships.

The part that matters: the bend #

How it learns #

The part that matters: the bend

How it learns