The Wiki
Intro7 min read

Activation bends

A straight line is one decision. A ReLU stack folds space — and folds are what let a network carve an arbitrarily complicated boundary.

A neuron computes a weighted sum and then — and then — bends the number line with its activation. The bend looks like a detail. In geometry it is the whole show: each bend is a fold in space, and layers of folds chop the input space up into regions the network can label independently.

The folding paper picture

Picture the input space as a sheet of paper. One ReLU neuron is one crease: everything on one side goes flat (zero), the other side keeps its slope. With enough creases, a flat sheet can wrap itself around almost any shape — each ReLU folds it once. Complexity is just folding count, which is what the artifact below can make visible.

A red cluster wants to be *enclosed*; blue is scattered around it.

accuracy: 100%

4 creases close in — the outside crowd shrinks per extra crease.

Each crease is one ReLU neuron’s boundary; composing them carves the folded envelope shown. Illustrative: creases placed on a fixed cycle — real networks learn where to crease by SGD, and fold counts grow combinatorially with depth (that’s next lesson’s point).

Where the fold actually lives

For a ReLU neuron y=max⁡(0,wx+b)y = \max(0, w x + b), the flat side is the half-space wx+b<0w x + b < 0; the crease runs along the line wx+b=0w x + b = 0. In 2 inputs that crease is a straight line; in more inputs, a hyperplane.

Composing neurons multiplies creases. Two ReLUs stacked on the same input gives a tent function — one fold. Eight-fold piecewise-linear output approximating an S-curve comes from eight. The number of linear regions a network can carve is a real object of study: for input dimension dd with nn neurons per layer, it grows exponentially in depth and only polynomially in width. That's the honest answer to "why depth, when you could widen?"

The graveyard of activations

You will never deploy most of the activation functions in textbooks. The short list you'll actually use:

  • ReLU max⁡(0,z)\max(0, z) — the default. Cheap, sparse, folds linearly.
  • Softmax — the output squeeze of classifiers, values summing to one; not a hidden-layer activation.
  • Sigmoid/tanh — the old defaults; both saturate (flatten when far from zero), which flattens the gradient too — a special case of the training problems coming in chapter 2.
  • GELU/SiLU — smooth ReLU-cousins used inside transformers (the LLM book's architecture lesson uses GELU).

ReLU's failure mode — narratives to keep

A ReLU can "die": if its input never sits in the positive half, its output is identically zero, its gradient is identically zero, it never learns. Dying ReLUs are the reason leaky versions and smooth versions exist, and one of the reasons initialization is treated seriously in the next chapter.

Illustrative vs real

The fold picture is honest geometry for the piecewise-linear family but simplified: it uses a fixed function shape and a fold counter, not a live network training region carver. Real networks fold millions of times over, and the region structure is researched, not precomputed.

Where next: stacking-layers — what those folds buy at training time: better fits for the same data, not more tricks.

This lesson has exercises attached — predicting region counts from layer/neuron configs — launching once the exercises layer ships.