Wiki
Advanced12 min read

Learning from human preferences

RLHF end to end: preference pairs → reward model → policy gradient with a KL leash — and where it breaks.

Judgments like "this answer is better than that one" are easy for humans to make and impossible to write as a formula. RLHF (reinforcement learning from human feedback) turns exactly that kind of judgment into a training signal, in three stages. Play the annotator first:

Start here

Humans are bad at writing perfect answers but good at comparing two. RLHF builds a model of that comparison ability (the reward model), then optimizes the assistant against it — like training against a critic instead of a script.

Prompt: "How do I make bread?" — you are the human annotator. Pick the better response.

Illustrative vs real

The scores in the artifact (r = 1.8, r = 0.2) are hand-assigned, not model output — real reward models output arbitrary-valued scores whose ordering is learned from thousands of pairs. What's faithful: the two-stage pipeline, KL leash, and Goodhart pressure are exactly how modern preference tuning runs.

Stage 1: collect preferences

Annotators see one prompt and two responses, and pick the better one. Thousands of these preference pairs are collected per task family. The judgment is cheap — comparing is cognitively easier than writing — and it captures values no rubric fully specifies: accuracy, tone, appropriate length, honesty.

Stage 2: train the reward model

A second transformer (often the base model with a new scalar-output head) is trained on the pairs. Its objective: for the chosen response, score it higher than the rejected one:

LRM=−log⁡σ ⁣(r(x,ychosen)−r(x,yrejected))\mathcal{L}_{\text{RM}} = -\log \sigma\!\left(r(x, y_{\text{chosen}}) - r(x, y_{\text{rejected}})\right)

The reward model learns a surrogate for human judgment — a differentiable stand-in for "better," which is the whole trick: it converts an unwritable objective into a number.

Stage 3: optimize the policy against the reward

The assistant model (the policy) generates responses, the reward model scores them, and the policy is nudged toward higher-scoring outputs — with one crucial brake:

max⁡θ  E[r(x,y)]−β KL(πθ(y∣x) ∥ πref(y∣x))\max_\theta \; \mathbb{E}\big[r(x, y)\big] - \beta\, \mathrm{KL}\big(\pi_\theta(y|x)\,\Vert\,\pi_{\text{ref}}(y|x)\big)

The KL penalty measures how far the policy has drifted from the reference model (usually the SFT checkpoint). Without it, the optimizer finds reward-hacking loops — the reward model scores some degenerate text (repetition, sycophancy, "as an AI, I cannot…" padding) higher than anything a human would like, and the policy exploits that scoring bug relentlessly. The KL term is the "stay recognizably the same model" contract.

This is Goodhart's law in its strongest form: when a measure becomes a target, it ceases to be a good measure. The reward model is a proxy for human judgment; RL applies enormous optimization pressure directly at the proxy, so the proxy's errors — not the agreement those errors were standing in for — are what get amplified. It is among the most extreme optimization-pressure-on-a-proxy settings we know of: the policy is a maximizer with billions of parameters, and the objective surface (a learned model scoring text) is also a learned, therefore hackable, artifact.

The mechanics, in the artifact: your choice becomes a training pair; the reward model's scores reveal a gap; the gap is the gradient direction — tempered by KL so the model can't run off chasing score instead of usefulness.

Careful

Reward hacking is the central failure mode of RLHF, not a corner case. Sycophancy (agreeing with the user), padding answers with hedging boilerplate, and uniform "balanced" responses are all learned exploits of imperfect reward models. The KL penalty slows it; better preference data and process supervision treat the cause.

Why this beats SFT — and where it doesn't

  • Preference tuning optimizes the judgment between candidates, so quality improves even when no human can write the perfect example.
  • It compresses "the annotator's taste" into the model — at scale, often better than any single annotator writes.
  • But it inherits annotator bias (annotators are the reward model's ground truth), it's expensive (a pipeline of sampling, labeling, RL rollout), and RL itself is finicky — which motivates the next lesson's simplification.

This lesson has exercises attached — predicting what a reward model scores for a given pair, and spotting a reward-hacked response — once the exercises layer ships.

Check yourself

Eduspheria wiki · Large Language Models, Post-training

0 / 4 answered

  1. 1What is the reward model trained to do?
    Multiple choice
  2. 2The KL penalty in RLHF keeps the policy from drifting too far from the reference model, limiting reward hacking.
    True / false
  3. 3What is the name of the law that says when a measure becomes a target it ceases to be a good measure, central to reward hacking?
    Short answer
  4. 4With the illustrative scores r = 1.8 for the chosen response and r = 0.2 for the rejected one, what is the score gap?
    Numeric answer

Next: the insight that makes the RL machinery optional — direct preference optimization.