From reward to RLHF
What happens when the reward signal is a human opinion instead of an environment signal — reward modeling, and why the LLM book's RLHF lesson is this chapter on top of a transformer.
Four lessons, one shape: an agent acts, a signal comes back, the agent updates toward the signal. Every ingredient is now on the table — credit flows backward along trajectories (lesson 1), exploration buys the experiences (lesson 2), values condense them into estimates (lesson 3), policy ascent turns them into behavior (lesson 4). This lesson changes exactly one variable: where the reward comes from.
Same loop, new judge
Replace 'the environment returned +1' with 'a human preferred this draft to that one.' Nothing about the learning loop changes — only the scoreboard is now a model of taste instead of a score counter.
Prompt: "How do I make bread?" — you are the human annotator. Pick the better response.
Why humans as the reward function
A Go engine can be given win/loss by the rules. A chat assistant has no rulebook — "helpful, honest, harmless" is a judgment call, and judgments are things humans can produce cheaply (two drafts in, one pick out) and formulas cannot. That inversion — from acting to score high to training a proxy of the scorer, then acting to score high under the proxy — is the whole construction of the LLM book's learning from human preferences lesson: preference pairs → reward model → policy gradient ascent with a KL leash. Three stages, and each one is a lesson of this chapter with the slots swapped:
| This chapter | LLM book's RLHF |
|---|---|
| agent | the LLM, drafting token-by-token |
| environment reward | a learned reward model's score |
| exploration / sampling | decoding with temperature |
| delayed reward, credit back | preference pair → ↑ chosen, ↓ rejected |
Specification gaming: the failure this chapter predicted
Lesson 2's ε=0 disaster was an agent confidently exploiting its own bad map of the world. Give the world map to a reward model learned from finite humans and the same failure class survives at scale — the literature calls it reward hacking: agents that satisfy the proxy while violating the intent (an agent told "pick up the virtual block" that first lifts the camera looking at it). The KL leash in the LLM book's update is one hedge; reward-model ensembles and adversarial probing are others. None fully close it, which is why the "limits and safety" lesson in the LLM book treats alignment as an ongoing engineering discipline, not a solved one.
Illustrative vs real
The bridge here is conceptual, not a new artifact — the interactive piece reuses the LLM book's annotator step deliberately, so the chapter closes where that book's post-training chapter opens. Real pipelines run reward models as fine-tuned transformers, RF objectives (PPO with clipped ratios or GRPO-style group baselines) against a frozen reference model, with the preference data coming from paid annotator pools and model critiques.
This lesson has exercises attached — mapping each RLHF pipeline stage back to the chapter concept it replaces — once the exercises layer ships.
Where next: the LLM book's post-training chapter, which runs this entire loop on a transformer — or, closer to home, deep learning, the missing domain between this book and that one.