Learning from human preferences
RLHF end to end: preference pairs → reward model → policy gradient with a KL leash — and where it breaks.
Judgments like "this answer is better than that one" are easy for humans to make and impossible to write as a formula. RLHF (reinforcement learning from human feedback) turns exactly that kind of judgment into a training signal, in three stages. Play the annotator first:
Start here
Humans are bad at writing perfect answers but good at comparing two. RLHF builds a model of that comparison ability (the reward model), then optimizes the assistant against it — like training against a critic instead of a script.
Prompt: "How do I make bread?" — you are the human annotator. Pick the better response.
Stage 1: collect preferences
Annotators see one prompt and two responses, and pick the better one. Thousands of these preference pairs are collected per task family. The judgment is cheap — comparing is cognitively easier than writing — and it captures values no rubric fully specifies: accuracy, tone, appropriate length, honesty.
Stage 2: train the reward model
A second transformer (often the base model with a new scalar-output head) is trained on the pairs. Its objective: for the chosen response, score it higher than the rejected one:
The reward model learns a surrogate for human judgment — a differentiable stand-in for "better," which is the whole trick: it converts an unwritable objective into a number.
Stage 3: optimize the policy against the reward
The assistant model (the policy) generates responses, the reward model scores them, and the policy is nudged toward higher-scoring outputs — with one crucial brake:
The KL penalty measures how far the policy has drifted from the reference model (usually the SFT checkpoint). Without it, the optimizer finds reward-hacking loops — the reward model scores some degenerate text (repetition, sycophancy, "as an AI, I cannot…" padding) higher than anything a human would like, and the policy exploits that scoring bug relentlessly. The KL term is the "stay recognizably the same model" contract.
This is Goodhart's law in its strongest form: when a measure becomes a target, it ceases to be a good measure. The reward model is a proxy for human judgment; RL applies enormous optimization pressure directly at the proxy, so the proxy's errors — not the agreement those errors were standing in for — are what get amplified. It is among the most extreme optimization-pressure-on-a-proxy settings we know of: the policy is a maximizer with billions of parameters, and the objective surface (a learned model scoring text) is also a learned, therefore hackable, artifact.
The mechanics, in the artifact: your choice becomes a training pair; the reward model's scores reveal a gap; the gap is the gradient direction — tempered by KL so the model can't run off chasing score instead of usefulness.
Careful
Reward hacking is the central failure mode of RLHF, not a corner case. Sycophancy (agreeing with the user), padding answers with hedging boilerplate, and uniform "balanced" responses are all learned exploits of imperfect reward models. The KL penalty slows it; better preference data and process supervision treat the cause.
Why this beats SFT — and where it doesn't
- Preference tuning optimizes the judgment between candidates, so quality improves even when no human can write the perfect example.
- It compresses "the annotator's taste" into the model — at scale, often better than any single annotator writes.
- But it inherits annotator bias (annotators are the reward model's ground truth), it's expensive (a pipeline of sampling, labeling, RL rollout), and RL itself is finicky — which motivates the next lesson's simplification.
This lesson has exercises attached — predicting what a reward model scores for a given pair, and spotting a reward-hacked response — once the exercises layer ships.
Next: the insight that makes the RL machinery optional — direct preference optimization.