Wiki
Advanced7 min read

Variational autoencoders

Make the bottleneck a probability distribution instead of a code — suddenly you can sample from the latent space and decode something real.

The plain autoencoder's latent space is a map with holes: trained codes cluster in little islands (one per visible training pattern) and everything between them decodes to nonsense. The VAE fix — encode to a distribution, not a point:

q(z∣x)=N(μ(x),σ(x)),z∼q;L=∥x^−x∥2⏟reconstruct+DKL ⁣(q(z∣x) ∥ N(0,1))⏟stay organized q(z|x) = \mathcal{N}(\mu(x), \sigma(x)), \quad z \sim q; \qquad \mathcal{L} = \underbrace{\|\hat{x}-x\|^2}_{\text{reconstruct}} + \underbrace{D_{\text{KL}}\!\left(q(z|x)\,\|\,\mathcal{N}(0,1)\right)}_{\text{stay organized}}

The KL term regularizes the latent geometry: every code neighborhood must look like the standard normal, which forces codes to spill into each other — islands merge into a continuum, and anywhere you sample decodes to something.

Why blurry, and why the fuzz is principled

A deterministic decoder from a "fuzzy" code has no opinion about the in-between pixels, so it averages its hypotheses — blur. It's honest: the model is telling you the latent doesn't determine the detail. Later generative stacks (diffusion) solve the detail problem by iterating instead of sampling once; the principle "sample structured codes, not points" carries over directly.

Sample $z \sim \mathcal26((0.5,0.5), \sigma)$ and decode.

E[decode] — what you’d render

Low KL pressure: tight cloud = crisp decode. But only *near codes* is anything real — sample further and decode garbage. Fragmented latent space.

Real geometry: sample cloud under increasing $\sigma$, expected decode under the fixed decoder shown. A real VAE re-trains under each σ (the blur tradeoff here is the honest result of that training: the *expected* decode loses contrast — it is a probability, not determinized).

The reparameterization trick (in one line)

Training needs gradients to flow through the sampling you just introduced. Randomness isn't gradient-flowable, so you re-parametrize:

z=μ(x)+σ(x)⊙ϵ,ϵ∼N(0,I) z = \mu(x) + \sigma(x) \odot \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)

Now ϵ\epsilon is treated as a noise input, all learned quantities (μ,σ\mu, \sigma) sit on the deterministic path, and backprop goes through clean. This trick — introduce randomness in a way that keeps the loss's component terms attached to differentiable code — is a pattern you will meet again in RLHF's next-token sampling.

Balanced losses in application

The two loss terms fight: extend KL → blurry-but-sampleable; extend reconstruction → sharp-but-fragmented space. The artifact shows the dial honestly. Real work uses annealed KL weights (β\beta), hierarchical VAEs, and bucketed token latents — the operational descendant in text is the quantized latents of image transformers (VQ-VAE lineage) feeding multimodal LLMs.

Illustrative vs real

2-D latent, real arithmetic (mu/sigma decode formulas on a grid hand-trained); dials interpolate the latent space in both axes. Real VAE decoders are convolutional stacks over image-sized outputs; the density + interpolation behavior shown is exactly their low-dim caricature.

Check yourself

Eduspheria wiki · Deep Learning, Generative nets

0 / 5 answered

  1. 1What does the VAE encoder output instead of a single latent point?
    Multiple choice
  2. 2The KL term pushes each encoded distribution toward a standard normal, merging isolated codes into a continuous latent space.
    True / false
  3. 3What trick rewrites the sample as mean plus standard deviation times epsilon so gradients can flow through the randomness?
    Short answer
  4. 4With mean 0.5, standard deviation 0.25 and sampled noise epsilon equal to 2, what is z = mean + standard deviation times epsilon?
    Numeric answer
  5. 5Which VAE behaviour is described as principled rather than a flaw?
    Multiple choice

From the assignment paper

Exam-modeled · Assignment/Quiz

0 / 5 answered

  1. 1What is the fundamental difference between a generative and a discriminative model?
    Multiple choice
  2. 2In a GAN, what tends to happen if the discriminator becomes too powerful early in training?
    Multiple choice
  3. 3A VAE reports a reconstruction term of 0.9 and a KL divergence of 0.4. What is the ELBO, reconstruction minus KL?
    Numeric answer
  4. 4Which loss term pushes a VAE's encoded distributions toward a standard normal so the latent space stays continuous?
    Short answer
  5. 5In a GAN, which network learns to tell real samples from generated ones?
    Multiple choice

Where next: adversarial-duel — the second (and third, and fourth) attempt to make generation crisp: get a second network to supervise.

This lesson has exercises attached — reading a VAE's loss tradeoff — launching once the exercises layer ships.