Policy gradients
Skip the value function: parameterize the policy itself, sample episodes, and do gradient ascent on expected reward. The mechanics every RLHF pipeline runs on.
Values told you how good states are, and a policy fell out of them (greedily pick the best-valued neighbor). The other classical road skips the middleman: fit the policy directly, as parameters, by gradient ascent on expected return.
Tasting the soup and adding salt
A value function is chemistry: it learns the composition of every ingredient hill before deciding. Policy gradient is cooking end-to-end: make a batch, taste it, nudge the salt. Noisier per batch, but the only thing you ever optimize is the outcome.
The exact expected-return curve over the policy’s one parameter θ (the agent can’t see it). Each REINFORCE step samples episodes and does one ascent step.
episodes: 0
Per-episode noise can push θ the wrong direction on its own batches — but the ascent trend follows the curve upward. Reset and rerun to watch a different noisy walk reach the same hilltop.
The gradient is estimated from the policy’s own episodes: each pull contributes ∇ log π(a) × (response − average) and the step is 0.5 × the batch average. No value function, no transition model — only the samples. Illustrative: one-parameter softmax, exact curve drawn for teaching; real policy gradients run on networks, subtract learned baselines, and clip updates (PPO).
The estimator
The core identity — the policy gradient theorem — says the gradient of expected return is an expectation over the policy's own episodes:
Read it as the artifact does: episodes the current policy generates supply the samples; moves whose journey earned a high return get their log-probability pushed up by ∇θ log π, weighted by that return. The update never consults a value function or transition tables — sample, score the sample, nudge the recipe, re-sample.
What you pay for the directness
Two famous costs:
- High variance. R mixes the systematic effect of the move with the luck of everything after it. When steps below the horizon are subtracted later (the "baseline / advantage" trick) the trajectory noise is partially removed — the same reason ordinary supervised gradients average over batches and REINFORCE-with-baseline subtracts a baseline: estimate the typical return so the update learns how much better than typical each move was.
- Sample inefficiency. Each episode contributes one gradient sample and is then discarded, whereas a learned value model bootstraps across stored knowledge. Deep RL's PPO, TRPO, and friends are best understood as different compromises for making each episode's information stretch further — often by reusing updates in mild, clipped increments per batch.
Notice the shape: the model is generating its own training data, and the distribution that data comes from shifts every update. That feedback loop is unlike anything in supervised learning this book has touched — and it's the exact reason RLHF pipelines in the LLM book clamp updates with a KL leash before a recipe improves itself into a reward-hacking corner.
Illustrative vs real
A one-parameter softmax policy over two bandit arms, with the ascent visualized against the exact expected-return curve — a teaching simplification; genuine REINFORCE never knows that curve. Production policy gradients run on deep networks, batch many episodes per update, subtract learned baselines, and wrap the ascent in trust-region or clipped objectives (PPO being the one behind most RLHF).
This lesson has exercises attached — deriving the score-function gradient for a two-arm policy by hand from three sampled pulls — once the exercises layer ships.
Where next: what happens when the "reward" is a human preference — the closing lesson, and the bridge to the LLM book's RLHF lesson.