Mixture of Experts
Why frontier models pay for parameters only when they're used — routing, top-k experts, and the load-balancing game.
Every transformer so far was dense: every token flows through every parameter. MoE (mixture of experts) changes one thing — the block's MLP is replaced by several parallel MLPs (experts) plus a tiny router that picks which ones actually process each token. Capacity grows with the number of experts; per-token compute does not.
Start here
A dense model is one huge kitchen where every ingredient passes through every station. An MoE model is a food court: a greeter (the router) sends each dish to one or two relevant stalls. You can build forty stalls — cost only rises for the dishes actually routed.
The routing step
For each token, a small linear layer produces one logit per expert, and the top- experts are selected:
The token is then processed by only those experts' FFNs, and their outputs are blended (weighted by router probability). With a small fraction of the expert pool (Mixtral: 2 of 8; DeepSeek-V3: 8 of 256), per-token FLOPs stay a small fraction of the dense equivalent while total parameters grow linearly with experts. This decoupling — parameters ≠ FLOPs — is the entire commercial point: MoE models buy capacity they serve cheaply.
Watch it work:
The router sends each token to the expert(s) that process it best
Each expert caps how many tokens it accepts (the "expert capacity" knob). Overflow tokens are dropped — routed through a residual path with no expert processing, which is exactly the failure mode a load-balancing loss exists to prevent. Scores are heuristic; a real router is learned and self-organizes its experts.
Things to try in the artifact: switch top-1 → top-2 (quality knob — more expert perspectives per token, more FLOPs); then drop expert capacity to ×1 and watch tokens get dropped — routed to no expert at all, processed only by the residual path. Dropped tokens are why the loss has a second term.
The load-balancing problem
Left alone, routers collapse: one expert gets slightly better early and wins everything, the others starve and never train. Two standard fixes, usually used together:
where is the fraction of tokens that landed on expert and the mean router probability for it. The product is minimized when traffic is spread evenly — an anti-concentration penalty, the opposite spirit of most losses. Router z-losses (computed in fp32 to keep the logits numerically tame) patch the remaining instability.
Careful
MoE is not free lunch. All experts sit in GPU memory whether or not they run (capacity without FLOPs still costs RAM); training is harder to keep stable; fine-tuning is more finicky; and the "experts" are not human-legible specialists — analysis finds them splitting by token/syntax patterns, not "the biology expert". You trade engineering complexity for a better params-per-FLOP ratio.
Who actually ships MoE
| Model | Experts | Active per token | Note |
|---|---|---|---|
| Mixtral 8×7B | 8 | 2 | ~47B params, ~13B active |
| DeepSeek-V3 | 256 + 1 shared | 8 | ~671B params, ~37B active |
| GPT-4 (reported) | 8–16 | 2 | never officially confirmed |
| Llama 4 | 16–128 (varies) | varies | MoE across the lineup |
One shared "always-on" expert plus routed experts is the modern pattern — common operations don't need routing overhead at all.
Next: the other half of every block — positions and normalization.
This lesson has exercises attached — predicting where a token routes and diagnosing an expert-collapse scenario — once the exercises layer ships.