Deep Learning
The deep learning middle shelf: how a neuron learns to bend a boundary, why stacked layers keep winning, and the training hacks (initialization, normalization, dropout, optimizers, learning rates) that make depth survivable. Continues the Classical ML book's gradient descent and hands attention directly to the LLM book.
From a neuron
One artificial neuron, the nonlinearity that makes layers more than decoration, and the landscape that training actually descends on.
- A neuronLogistic regression wearing a helmet: one weighted sum, one bend. Start here and depth becomes learnable instead of mysterious. 6 min · intro
- Activation bendsA straight line is one decision. A ReLU stack folds space — and folds are what let a network carve an arbitrarily complicated boundary. 7 min · intro
- Stacking layersWidth buys folds linearly; depth folds them exponentially. Watch the same two-moons problem as the stack gets taller. 7 min · core
- Universal approximationA deep-enough ReLU net can approximate anything continuous. So why is that theorem famously unhelpful? Because it says nothing about finding the weights. 7 min · core
- The loss landscapeOne parameter, one curve, was the ML book. Deep nets put you on a surface in millions of dimensions — here's what that surface treats as hills, basins, and roads. 7 min · core
Training deep things
Why depth made gradients vanish — and the five hacks (init, normalization, dropout, optimizers, schedules) that turned deep nets from theory into practice.
- Gradient flowThe gradient tells every layer which way down is. In a deep net that signal is a long product — small factors kill it, large ones blow it up. This is *the* obstacle of depth. 8 min · core
- Init and normTwo knobs, one job: keep activation statistics steady as the signal crosses layer after layer. This fix collectively took the field twenty years. 8 min · core
- DropoutDeliberately break parts of the network on every step. It sounds like sabotage — it's the cheapest generalization tool we have. 7 min · core
- OptimizersGradient descent is the rule. Momentum, adaptive scaling, or both — three physical upgrades that decide whether training takes hours or days. 8 min · core
- The learning rateOne knob, and it dominates. Too small wastes the most expensive compute you have; too big ejects you from the basin; the schedule is everything. 7 min · core
Convolutions and vision
Sliding filters, shrinking hierarchies, the residual trick, what convolution actually costs — and why pretrained beats scratched-from-scratch almost always.
- Sliding filtersA convolution is one small pattern, stamped across the whole image at every position. It runs vision because the same edge matters everywhere. 7 min · core
- Pooling and hierarchiesShrink 224 to 11 without losing the plot: max-pooling, stride-two convs, and how 'less resolution, same meaning' is what lets deeper layers see bigger things. 6 min · core
- Residual connectionsIn 2015 depth quietly stopped working — and the fix was embarrassingly simple: let each layer output the identity plus a correction. ResNets went 100+ layers deep. 7 min · core
- Convolution costWhere the parameters and compute actually go in a conv layer — the arithmetic that steered a decade of architecture design. 6 min · core
- Transfer learningEarly conv layers learn edges nobody owns. Borrow the bottom of a pretrained network, keep it frozen or warm — and beat from-scratch with a fraction of the data. 7 min · advanced
Sequences before attention
Recurring state: how networks ordered their weights into memory — RNNs, why they forget, the LSTM's gates, and the bottleneck that attention dissolved.
- Recurring stateSame weights, looping over time: the RNN idea that let a fixed-size network read arbitrary-length sequences. 6 min · core
- The vanishing pastBy step 400, step 5 has had its say… and its gradient multiplied away to nothing. Why simple RNNs can't learn long-range dependencies. 6 min · core
- Gates and LSTMsThe LSTM decides, at every step, what to remember and what to overwrite — three learned gates that kept memory stable for hundreds of steps. 7 min · core
- The attention bridgeCompression was the plan; attention says 'just look the word up.' The end of the middle shelf — and the door straight into the LLM book. 7 min · advanced
Generative nets
From compress-reconstruct to sampling new images: autoencoders, VAEs, the GAN duel, diffusion — and what the middle shelf leaves for later shelves.
- AutoencodersSqueeze an image through a tiny bottleneck and reconstruct it — the constraint that forces the network to learn representation instead of copying. 6 min · core
- Variational autoencodersMake the bottleneck a probability distribution instead of a code — suddenly you can sample from the latent space and decode something real. 7 min · advanced
- Adversarial duelTwo networks fight: a forger against a detector. The generator learns to produce real-looking data because losing to a critic is still training signal. 7 min · advanced
- DiffusionDestroy the image with noise, learn to remove it — run that learned remover in reverse, from pure noise, and you have today's image generators. 8 min · advanced
- What DL left outThe honest map of gaps in this middle shelf — state-space models, multimodal fusion, and every place the story continues elsewhere. 6 min · core