Transfer learning
Early conv layers learn edges nobody owns. Borrow the bottom of a pretrained network, keep it frozen or warm — and beat from-scratch with a fraction of the data.
The most practically valuable lesson in the whole deep learning book costs one paragraph: nobody trains vision networks from scratch anymore. The bottom layers of a convnet trained on millions of images have learned general visual machinery — edge filters and textures, part-hunting patterns — that remain valid for nearly any new image task. Transfer learning is the discipline of reusing that bottom:
Why it works, from necessity's side
Receptive fields grow with depth (chapter 3's shrinking logic); early layers are therefore forced to know meaning at small scales — corners, strokes, gradients of color. None of that is dataset-specific: an edge is an edge whether your set is leaves or X-rays. So the early layers transfer nearly perfectly and only the deeper ones need adapt — which is exactly what the freeze dial lets you feel.
A five-block pretrained backbone + new head. Freeze / unfreeze by block, adjust dataset size, read the verdict.
trainable: 3.2M
of 3.3M total · 96% of the network
valid acc ≈ 92%
The sweet spot: enough capacity trainable, enough borrowed machinery frozen. Curves this shape are why transfer is the default.
Parameter counts and the freeze arithmetic are real (a ResNet-18-scale toy). Accuracy curves are *parameterized* illustrations of the standard transfer finding — small data favors freezing, huge data erases the pretraining gap. Don’t read curve *values* off them; read the *shape*.
The menu, from cheap to expensive
- Frozen backbone + new head — train only the last block(s). Fastest, works when the new task is close (the cat/breed detector on top of a generic backbone).
- Fine-tune everything, low learning rate — update everything gently; standard when you have moderate data and the domains differ somewhat (X-rays are not ImageNet cats).
- Progressive unfreezing / discriminative rates — unfreeze layer-by-layer as training stabilizes; the practical middle path for noisy small sets.
- Full scratch — only when the domain's visual statistics genuinely differ (medical imagery, microscopy, non-RGB sensor data) and compute/data are available.
The dark lining
Transfer to a contrasting domain (aerial photos, medical scans) can miss: edges do transfer mostly, but object priors sit deeper and are exactly what you must re-learn. And a frozen-backbone head's confidence is mis-calibrated on far-domain data — the fine caveat that shows up in production as confident nonsense. Honest evaluation discipline from the ML book matters double here: test on fresh domain data before you believe a transfer curve.
Illustrative vs real
The artifact's curves are parameterized (not live-trains): the gap-shaped relationship of dataset size × trainable fraction is drawn from the standard transfer result pattern (pretraining wins early/cold-start and disappears from the picture only at huge-data scale). The parameter-count tradeoff shown live — frozen params, trainable budget — is real arithmetic.
Where next: the last chapter of the middle shelf — sequences, and how the recursive memory idea had its heart broken by attention.
This lesson has exercises attached — choosing the freeze regime for a described task — launching once the exercises layer ships.