Advanced7 min read

Variational autoencoders

Make the bottleneck a probability distribution instead of a code — suddenly you can sample from the latent space and decode something real.

The plain autoencoder's latent space is a map with holes: trained codes cluster in little islands (one per visible training pattern) and everything between them decodes to nonsense. The VAE fix — encode to a distribution, not a point:

q(z|x) = \mathcal{N}(\mu(x), \sigma(x)), \quad z \sim q; \qquad \mathcal{L} = \underbrace{\|\hat{x}-x\|^2}_{\text{reconstruct}} + \underbrace{D_{\text{KL}}\!\left(q(z|x)\,\|\,\mathcal{N}(0,1)\right)}_{\text{stay organized}}

The KL term regularizes the latent geometry: every code neighborhood must look like the standard normal, which forces codes to spill into each other — islands merge into a continuum, and anywhere you sample decodes to something.

Why blurry, and why the fuzz is principled

A deterministic decoder from a "fuzzy" code has no opinion about the in-between pixels, so it averages its hypotheses — blur. It's honest: the model is telling you the latent doesn't determine the detail. Later generative stacks (diffusion) solve the detail problem by iterating instead of sampling once; the principle "sample structured codes, not points" carries over directly.

Sample $z \sim \mathcal26((0.5,0.5), \sigma)$ and decode.

E[decode] — what you’d render

KL pressure σ0.06

Low KL pressure: tight cloud = crisp decode. But only *near codes* is anything real — sample further and decode garbage. Fragmented latent space.

Real geometry: sample cloud under increasing $\sigma$, expected decode under the fixed decoder shown. A real VAE re-trains under each σ (the blur tradeoff here is the honest result of that training: the *expected* decode loses contrast — it is a probability, not determinized).

The reparameterization trick (in one line)

Training needs gradients to flow through the sampling you just introduced. Randomness isn't gradient-flowable, so you re-parametrize:

z = \mu(x) + \sigma(x) \odot \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)

Now $\epsilon$ is treated as a noise input, all learned quantities ( $\mu, \sigma$ ) sit on the deterministic path, and backprop goes through clean. This trick — introduce randomness in a way that keeps the loss's component terms attached to differentiable code — is a pattern you will meet again in RLHF's next-token sampling.

Balanced losses in application

The two loss terms fight: extend KL → blurry-but-sampleable; extend reconstruction → sharp-but-fragmented space. The artifact shows the dial honestly. Real work uses annealed KL weights ( $\beta$ ), hierarchical VAEs, and bucketed token latents — the operational descendant in text is the quantized latents of image transformers (VQ-VAE lineage) feeding multimodal LLMs.

Illustrative vs real

2-D latent, real arithmetic (mu/sigma decode formulas on a grid hand-trained); dials interpolate the latent space in both axes. Real VAE decoders are convolutional stacks over image-sized outputs; the density + interpolation behavior shown is exactly their low-dim caricature.

Where next: adversarial-duel — the second (and third, and fourth) attempt to make generation crisp: get a second network to supervise.

This lesson has exercises attached — reading a VAE's loss tradeoff — launching once the exercises layer ships.

The reparameterization trick (in one line) #

Balanced losses in application #

The reparameterization trick (in one line)

Balanced losses in application