Direct preference optimization
DPO: the same preference signal as RLHF, without a reward model or RL loop — and the trade it makes.
RLHF works, but it's a four-stage pipeline: sample from the policy, label with humans, train a reward model, run RL — each stage can drift out of sync with the others. In 2023, a mathematical observation collapsed the pipeline into one ordinary supervised loss:
Start here
DPO's insight: the RLHF objective (maximize reward, stay close to the reference model) has a closed-form solution — and under it, the reward model can be solved analytically from the policy. Substituting that solution back eliminates the reward model entirely: two responses, a preference label, one binary-classification-style loss.
The loss
Read it plainly: increase the relative likelihood of the preferred response () and decrease the relative likelihood of the rejected one (), each measured against the reference model, balanced by . The reference-model comparison plays the same role as the KL penalty — it keeps the model from collapsing — and sets how strictly. Feel what β does to the gradient:
DPO loss for one preference pair — β is the leash
loss = 0.437
moderate β: real gradient pressure with a built-in leash — the sweet spot the reference implementation defaults near
The curve is the real DPO loss — play with both sliders: small β + big movement saturates (no gradient left to complain), big β flattens everything (nothing trains). β encodes the same "stay recognizably the same model" contract as RLHF's KL penalty, but *inside* the loss instead of as a separate penalty term — which is why DPO needs no reward model and no RL loop.
The beautiful part: there is no reward model to train, no sampling rollouts, no RL optimizer. It trains like SFT — forward pass, loss, backward pass — on preference pairs. The preference signal and the optimization target are identical to RLHF's; only the machinery differs.
The trade DPO makes
| RLHF (PPO-style) | DPO | |
|---|---|---|
| Components | policy + reward model + RL loop | policy only |
| Data use | online: fresh samples scored by RM | offline: fixed preference set |
| Failure mode | reward hacking, training instability | limited by offline pairs; can overfit |
| Compute | high (rollouts) | low (supervised-style) |
| Ceiling | higher with iterative fresh data | good, plateaus without refresh |
Both approaches optimize "prefer chosen over rejected." Iterative variants (generate new samples, re-label, re-train) close much of the gap; for many teams, DPO's simplicity wins outright. Frontier labs often use RLVR (RL with verifiable rewards — math/code checkers) where ground truth exists, and preference methods elsewhere.
Note
A way to hold all of this: SFT = imitate demonstrations; preference tuning (RLHF or DPO) = optimize comparisons; RLVR = optimize against checkable answers. Modern post-training is a deliberate mix of all three, sequenced like a curriculum.
What post-training as a whole achieved
Trace the arc: base model → SFT (speaks when spoken to) → preference tuning (prefers the genuinely better answer) → optionally RLVR (nails verifiable tasks). The result is an assistant whose behavior, not just knowledge, was shaped by explicit training goals.
Next: how we know any of it works — the evaluation problem, which is harder than it sounds.