Wiki
Intro6 min read

A neuron

Logistic regression wearing a helmet: one weighted sum, one bend. Start here and depth becomes learnable instead of mysterious.

Strip a neural network down to its smallest working part and you get something you have already met. A neuron takes its inputs, multiplies each by a learned weight, adds a learned bias — and then, crucially, squashes the result through a nonlinear function before passing it on.

y=σ(w1x1+w2x2+⋯+wnxn+b) y = \sigma(w_1 x_1 + w_2 x_2 + \dots + w_n x_n + b)

That's it. The whole unit.

It's logistic regression in disguise

You've already fit this exact function with classical ML — the sigmoid over a weighted sum. A neuron is a logistic-regression unit whose output feeds into other units instead of stopping. Deep learning is "logistic regression, composing with itself a thousand times deep." That's not snark; it's the entire trick. There is weight-free "structure" hiding here, just a lot of small learning machines voting.

One neuron: score = w₁·x + w₂·y + b, class = red if score > 0.

accuracy: 100%

Outlined dots = misclassified. Perfect separation — and notice it's ONE straight crease. This neuron's full repertoire is exactly this.

The dashed line is where score = 0; slope = −w₁/w₂, offset set by the bias. Illustrative: threshold at 0 with deterministic points; training would find (w, b) by gradient descent on a loss — the artifact lets you be SGD by hand.

The part that matters: the bend

Without the nonlinearity σ\sigma, here's the awkward truth: stacking linear layers composes straight lines into... another straight line. A hundred layers of W2(W1x)=(W2W1)xW_2 (W_1 x) = (W_2 W_1) x — one multiply. Capacity: identical to a single layer. Depth is pure overhead unless something nonlinear happens between the matmuls.

The bend is cheap. ReLU — rectified linear unit — is just max⁡(0,z)\max(0, z): negative? Zero. Positive? Exact same value. That's the entire operation, and it's the default inside almost every deep net being trained today.

“Nonlinear” doesn't mean complicated

People hear nonlinearity and picture exotic transcendental functions. ReLU is a corner: z<0↦0z < 0 \mapsto 0, else zz. Its virtue is that it is piecewise linear nearly everywhere — meaning big ReLU nets behave like smoothly-folded pieces of straight lines, which is exactly what the next lesson's visualization will make visible.

How it learns

Same rule you know. Compute a loss, take the gradient with respect to wiw_i and bb, step downhill:

∂y∂wi=σ′(z)⋅xi \frac{\partial y}{\partial w_i} = \sigma'(z) \cdot x_i

That chain rule — output gradient times input signal — is literally the backpropagation step from the LLM training chapter, applied to one neuron. Depth is about composing many of these; training is still this loop.

Illustrative vs real

The artifact shows one neuron separating a 2-D toy — a boundary you can see. Real neurons operate on thousands of inputs inside layers of thousands of neurons. But the math (weighted sum → nonlinearity → gradient step) is the same; only the blindfold grows.

Check yourself

Eduspheria wiki · Deep Learning, From a neuron

0 / 4 answered

  1. 1A single artificial neuron applies a nonlinearity to its weighted sum before passing the result on.
    True / false
  2. 2Why is stacking purely linear layers pointless?
    Multiple choice
  3. 3Which activation function is defined as max(0, z)?
    Short answer
  4. 4The gradient for one neuron with respect to a weight is the activation derivative times the input. If the derivative is 0.25 and the input is 4, what is the gradient?
    Numeric answer

From the mid-term paper

Exam-modeled · Mid-Term October 2024

0 / 5 answered

  1. 1A perceptron with weights w1 = 1, w2 = 1 and bias b = 0 outputs +1 when the weighted sum is non-negative. On input x = (2, 1) the target is -1, so an update is made with learning rate 1. What is the new w1?
    Numeric answer
  2. 2For the same perceptron update, what is the new bias?
    Numeric answer
  3. 3A Hebb network is trained on bipolar OR with a bias. Using the Hebb rule w_i = sum of x_i times y over the four patterns, what is each of the two input weights?
    Numeric answer
  4. 4A single McCulloch-Pitts neuron can implement the XOR function.
    True / false
  5. 5What property must a dataset have for a single-layer perceptron to classify it with a straight boundary?
    Short answer

Where next: activation-bends — why those "small cheap bends" become folds in space, and why folding is the entire superpower.

This lesson has exercises attached — mix-and-match weight/bias intuition on the decision boundary — launching once the exercises layer ships.