Learning from consequence
No labels, no correct answer — just an agent acting, an environment answering, and a reward arriving late. Why credit assignment is the hard part.
Every chapter so far had something pointing at the right answer: labels, losses, structure to preserve. Reinforcement learning is the paradigm where all of that is gone. An agent acts, an environment answers — sometimes just "you won" after two hundred moves — and the only teacher is consequence itself.
The loop in one breath
Watch the world, pick an action, get a reward, and update your tendency to repeat whatever led here. The learning problem isn't the reward — it's working out which of the two hundred moves deserved the credit.
5×5 grid: goal bottom-right (+1), two puddles (−1). The agent starts top-left — the reward lands at the end of the path, and the backward propagation does the teaching.
episodes: 0 · cells shaded by learned value
Explore a few times, then greedy. Early runs: the puddle-adjacent route still tempts. Explore more and the shortcut through the puddles fades — its cost propagated back first.
TD(0): each visited cell nudges its value toward reward + γ·(value of the next cell), so the +1 flows backward along experienced paths — nobody annotates the intermediate cells; the consequences do. Reset + explore shows new runs settle on the same shape from different trajectories. Illustrative scale; real agents run millions of steps with neural value approximators.
Credit assignment: the defining difficulty
In supervised learning, the label arrives attached to the example. Here, the +1 arrives in a corner cell of a 5×5 grid — after a chain of 8 decisions, some of which were actively counterproductive. Which move caused the win? The TD(0) updates in the artifact quietly do the classic trick: each visit nudges a state's value toward the value of what came next, so the reward propagates backward along the experienced path. The gradient didn't tell us anything; the trajectory did.
That's the shape of every RL algorithm, from Q-learning to PPO: a mechanism for distributing a delayed signal across the actions that earned it. Different algorithms are mostly different bets on how much to trust each step of the backward pass.
What this chapter is
Reinforcement is the third learning paradigm, and this chapter closes the set this book has assembled:
- Supervised (chapters 1–4): fit parameters to labeled examples.
- Unsupervised (chapter 5): find structure with no labels at all.
- Reinforcement (this chapter): learn from a reward sequence your own actions generate.
It's also where classical ML touches the LLM book for real: the RLHF lesson there assumes this chapter. But first the two classical foundations — choosing to search (next lesson) and the values that make search smart (the lesson after).
Illustrative vs real
The artifact is a hand-rolled TD(0) sweep with a fixed reward layout — illustrative scale. Real gridworlds and their big cousins (Atari, robotics, games) involve millions of steps, function approximation instead of a value table, and the notorious instability the "deadly triad" of bootstrapping + off-policy + deep function approximators introduces.
This lesson has exercises attached — hand-computing a TD(0) update along a 4-step trajectory to watch credit flow backward — once the exercises layer ships.
Where next: exploration vs exploitation — the tension that decides whether the agent ever even visits good states.