The Wiki
Advanced8 min read

Diffusion

Destroy the image with noise, learn to remove it — run that learned remover in reverse, from pure noise, and you have today's image generators.

Two failure modes sat between VAE (principled, blurry) and GAN (crisp, unstable). Diffusion (Ho et al., 2020, DDPM) sidesteps both with a recipe so blunt it took the field by surprise:

  1. Forward (fixed, no learning): corrupt a real image over TT steps by adding a little Gaussian noise each step, until nothing but noise remains.
q(xt∣xt−1)=N(αtxt−1,(1−αt) σ2I) q(x_t \mid x_{t-1}) = \mathcal{N}(\sqrt{\alpha_t} x_{t-1}, (1-\alpha_t)\,\sigma^2 I)
  1. Learn only one simple thing: a network ϵθ(xt,t)\epsilon_\theta(x_t, t) predicting the noise in a partially-corrupted xtx_t. Loss is plain MSE against the known noise:
L=Ex,t,ϵ[∥ϵ−ϵθ(αˉt x0+1−αˉtϵ,  t)∥2] \mathcal{L} = \mathbb{E}_{x, t, \epsilon} \big[ \|\epsilon - \epsilon_\theta(\sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\epsilon,\; t)\|^2 \big]
  1. Generate by running step 1 backwards: start from pure noise and subtract predicted noise, one step at a time.

Why supervising every step wins

Each denoising step is an ordinary supervised regression problem — stable, well-conditioned, and locally easy: a slightly noisy image only needs its next small un-noising, which is more image-arithmetic than imagination. Compare: a GAN must turn structureless noise into * data * in one leap without supervision; diffusion gets thousands of supervised pushes. Sampling horizons (the artifact's slider) trade faithfulness for compute.

One trained denoiser evaluated at four corruption levels t.

t = 1.0 (pure noise)
t = 0.6
t = 0.3
t = 0.0 (converged)

Each panel is the SAME sampler at a different corruption level: at t=1 the “prediction” is noise everywhere; at t=0.3 the ring’s outline is already fixed; at t=0 the reconstruction lands. That is why per-step supervised regression works — each step answers a locally-easy question, and the composition does the imagination.

Real per-step contraction toward the data ring — DDPM’s core mechanism (learned noise prediction distilled to its geometric nugget) at 2-D toy scale. Left out: the sampler-family choices (DDIM, schedulers, guidance strength), which trade steps against fidelity without changing the principle.

What modern stacks actually do

Production image models and — via their latent-token formulations — generative video are diffusion at heart, with a few industrial upgrades:

  • Latent diffusion — denoise a compressed latent (the autoencoder trick supplying the compressor), because full-pixel noise costs Quadratic economics.
  • Classifier-free guidance — a dial between "stay on the model's learned prior" and "commit to the requested condition"; the dial exists because pushing too far sharpens images past realism. The knob is why your prompts "over-follow" at strength 9.
  • Conditioning with cross-attention — the attention machinery injects text queries at every denoising step; diffusion and transformers are fused through exactly this door.

The trade list

Diffusion sampling costs many network passes (the duel: one; the VAE: one). Distillation and consistency models cut sampling to 1–4 steps at some quality cost — the standard current-data pipeline considerations apply and the cost ladder is now measured in network-evals-per-image.

Illustrative vs real

A real 2-D-to-1D toy trained on noise prediction (not image-sized, but the mechanism is exactly DDPM per-step regression with visible certainty per step). Held-back here: the modern sampler families (DDIM, scheduler choices) which change quality-speed curves without changing the principle.

Where next: what-dl-left-out — the honest "not everything worth knowing is on this shelf", then the LLM book picks the story back up.

This lesson has exercises attached — forecasting a sampler's path — launching once the exercises layer ships.