The Wiki
Core7 min read

Stacking layers

Width buys folds linearly; depth folds them exponentially. Watch the same two-moons problem as the stack gets taller.

Given the fold picture from the last lesson, a fair question: why not one enormous shallow layer instead of stacking? Depth earns its keep because folds compound. A network with dd layers of nn ReLU neurons can carve on the order of stacked-fold-exponential regions; the same number of neurons spread across one layer gives you far fewer. Depth is how a fixed parameter budget buys a more twisted decision boundary.

What depth actually is, mechanically

There is no "deep thing" layer type. A deep network is: matmul, nonlinearity, matmul, nonlinearity, ... Shape x→h1→h2→⋯→outputx \rightarrow h_1 \rightarrow h_2 \rightarrow \dots \rightarrow \text{output}, where each hidden layer transforms what the previous layer created. Layer 1 folds raw pixel space; layer 2 folds directions in folded space; layer 3 folds that. Hierarchies of concepts aren't an analogy — they're what successive folds over folds produce.

The same two-moon dataset under four different network depths.

accuracy: 56%

Four creases thread both moon tips — nearly everything classifies right.

Boundaries are representative trained results per depth, drawn piecewise-linear because ReLU networks are piecewise linear (chapter 1). Same dataset, only depth changes — “more creases for the same parameters” is the empirical depth dividend. Real nets also re-choose *where* creases sit at every depth.

What you're looking at

Same two-moons dataset throughout. As the layer count rises, the fence goes from straight line to once-kinked to twice-kinked to a boundary shape that actually threads the gap between the moons. The parameter count barely moves — the composability moves.

This hierarchical story is the same ladder vision models climb, and it's why convolutions (chapter 3) end up natural: early layers find edges, later layers find combinations of edges.

The caveat — and the counter-melody

Depth is powerful and fragile. Two facts that keep chapter 2 busy:

  1. Gradients have to travel all the way down. In an LL-layer net, the update signal for early layers is a product of LL factors — and products of many small numbers vanish (many large ones explode). Deep nets were stuck on exactly this until initialization and normalization fixed the path.
  2. Parameter count isn't capacity. A network that needs 12 layers may express the same family with 2 — but the training dynamics of the two are wildly different. Depth is as much a training claim as an expressivity claim.

Residual connections will steal the show

When depth hurt instead of helped, the fix wasn't less depth — it was skip connections: let information bypass layers. Every transformer is, in this sense, a very deep convolutional network wearing residuals everywhere. That idea gets its own treatment in chapter 3.

Illustrative vs real

The four boundaries shown are representative fitted curves, not a live network re-training depth-first in front of you — re-fitting the exact same data at each depth is possible, but would slow the artifact down without changing the pattern (and the pattern has been the empirical result since the 1989 Cybenko and 2010s scaling-era papers). Illustrative region counts follow known theoretical growth rates.

Where next: universal-approximation — the theorem everyone quotes and nobody survives on, and what it doesn't promise.

This lesson has exercises attached — matching dataset twistiness to minimum depth — launching once the exercises layer ships.