Wiki
Core6 min read

Autoencoders

Squeeze an image through a tiny bottleneck and reconstruct it — the constraint that forces the network to learn representation instead of copying.

Generation starts from an unglamorous trick: train a network to output its own input, but force it through a bottleneck far too narrow to carry the data:

in  →  encoder    z  →  decoder    x^,L=∥x^−x∥22 \text{in} \;\xrightarrow{\;encoder\;}\; z \;\xrightarrow{\;decoder\;}\; \hat{x}, \qquad \mathcal{L} = \|\hat{x} - x\|_2^2

If zz had room for a copy of xx, the network could memorize and reproduce — learning nothing. When zz is smaller than the data, the only paths to a low loss run through compressed features that generalize: edges, texture statistics, part structure. Same idea, slightly dressed, as dimensionality reduction — except nonlinear, and trained by the gradient machinery of chapter 2.

A bottleneck is a curriculum

The network's reconstruction quality tells you what survived the bottleneck. Run the slider on the artifact and watch whole object identities stay stable while decorative detail smears — dimensions of the bottleneck are voted on by the loss, and what survives indicates what your loss values. Reconstruction loss is the cheap-to-measure stand-in for "meaning".

A 4-quadrant decoder, 2 latent dims. Drag the red latent point; reconstruction follows.

decoded (4×4 reconstruction)

Slide between codes (grey dots): near a *trained* code the reconstruction is crisp; between codes the decoder averages its neighbors and the output smears — you are inside “untrained middle”: the plain AE’s hole-ridden latent map.

Real decoder arithmetic: Gaussian-anchored Gaussian basis over a 2-D latent (a trained AE’s decoder distilled to its smallest honest caricature). Real autoencoders use conv stacks over image-sized grids; the *bottleneck geometry* (crisp at codes, mush between them) is the transferable story.

Where it's actually used

Raw autoencoders as generators — sample zz, decode — produce murk: the latent space has no structured density guaranteeing a "real" decode, so sampled points decode to mostly garbage between clusters. Their real jobs:

  1. Pretraining / representation learning — bottleneck features initialize or off-the-shelf embed for transfer-style work.
  2. Denoising masks — corrupt the input, train to restore: a self-supervised "fill-in" formulation that, as the diffusion lesson will show, scales up into the generative recipe, later.
  3. The latent anatomy for VAEs — next lesson's entire subject is what to do about the missing density structure.

AE ≠ PCA ≠ VAE, and never confuse them

Linear bottleneck + MSE = PCA (provably); nonlinear bottleneck = autoencoder; VAE imposes a density on zz, changing the loss and the sampling behavior fundamentally. If your reconstruction is perfect, your bottleneck isn't small enough — overfitting through a pass-through channel is a real, common failure.

Illustrative vs real

A 2-D latent decoder with a genuinely trained tiny AE (the grid interpolates real features). Missing: convolutional decoders and high-resolution images; the bottleneck → structure relationship shown is the transferable core.

Check yourself

Eduspheria wiki · Deep Learning, Generative nets

0 / 5 answered

  1. 1If the autoencoder's bottleneck is large enough to copy the input, the network can memorize and learn nothing useful.
    True / false
  2. 2Why does sampling a raw autoencoder's latent space often produce garbage?
    Multiple choice
  3. 3A linear autoencoder with a small bottleneck and squared-error loss is provably equivalent to which classical method?
    Short answer
  4. 4The reconstruction loss is the squared L2 distance between input and output. If the input is the pair (1, 0) and the reconstruction is (0, 0), what is the squared error?
    Numeric answer
  5. 5Which real use does the lesson list for autoencoders?
    Multiple choice

Where next: variational-autoencoder — give the latent space a probability density, and sampling becomes principled.

This lesson has exercises attached — reading reconstruction losses by bottleneck width — launching once the exercises layer ships.