Core6 min read

Autoencoders

Squeeze an image through a tiny bottleneck and reconstruct it — the constraint that forces the network to learn representation instead of copying.

Generation starts from an unglamorous trick: train a network to output its own input, but force it through a bottleneck far too narrow to carry the data:

\text{in} \;\xrightarrow{\;encoder\;}\; z \;\xrightarrow{\;decoder\;}\; \hat{x}, \qquad \mathcal{L} = \|\hat{x} - x\|_2^2

If $z$ had room for a copy of $x$ , the network could memorize and reproduce — learning nothing. When $z$ is smaller than the data, the only paths to a low loss run through compressed features that generalize: edges, texture statistics, part structure. Same idea, slightly dressed, as dimensionality reduction — except nonlinear, and trained by the gradient machinery of chapter 2.

A bottleneck is a curriculum

The network's reconstruction quality tells you what survived the bottleneck. Run the slider on the artifact and watch whole object identities stay stable while decorative detail smears — dimensions of the bottleneck are voted on by the loss, and what survives indicates what your loss values. Reconstruction loss is the cheap-to-measure stand-in for "meaning".

A 4-quadrant decoder, 2 latent dims. Drag the red latent point; reconstruction follows.

decoded (4×4 reconstruction)

z₀0.30z₁0.30

Slide between codes (grey dots): near a *trained* code the reconstruction is crisp; between codes the decoder averages its neighbors and the output smears — you are inside “untrained middle”: the plain AE’s hole-ridden latent map.

Real decoder arithmetic: Gaussian-anchored Gaussian basis over a 2-D latent (a trained AE’s decoder distilled to its smallest honest caricature). Real autoencoders use conv stacks over image-sized grids; the *bottleneck geometry* (crisp at codes, mush between them) is the transferable story.

Where it's actually used

Raw autoencoders as generators — sample $z$ , decode — produce murk: the latent space has no structured density guaranteeing a "real" decode, so sampled points decode to mostly garbage between clusters. Their real jobs:

Pretraining / representation learning — bottleneck features initialize or off-the-shelf embed for transfer-style work.
Denoising masks — corrupt the input, train to restore: a self-supervised "fill-in" formulation that, as the diffusion lesson will show, scales up into the generative recipe, later.
The latent anatomy for VAEs — next lesson's entire subject is what to do about the missing density structure.

AE ≠ PCA ≠ VAE, and never confuse them

Linear bottleneck + MSE = PCA (provably); nonlinear bottleneck = autoencoder; VAE imposes a density on $z$ , changing the loss and the sampling behavior fundamentally. If your reconstruction is perfect, your bottleneck isn't small enough — overfitting through a pass-through channel is a real, common failure.

Illustrative vs real

A 2-D latent decoder with a genuinely trained tiny AE (the grid interpolates real features). Missing: convolutional decoders and high-resolution images; the bottleneck → structure relationship shown is the transferable core.

Where next: variational-autoencoder — give the latent space a probability density, and sampling becomes principled.

This lesson has exercises attached — reading reconstruction losses by bottleneck width — launching once the exercises layer ships.

Where it's actually used #

Where it's actually used