Value functions
The Bellman equation: the value of a state is its reward plus the discounted value of what comes next. Why 'what comes next' must be discounted — and how values flow backward.
"Which arms paid off" was a one-step question — reward on the spot. Real agents face differently-shaped problems: the reward lands at the goal, and the question is what any individual position along the way is worth. A value function is the answer: for each state, an estimate of the future reward it can expect — and its defining equation is recursive.
Property values
A house isn't valuable because of its rooms; it's valuable because of where it lets you go. States work the same way — one step's own reward, plus the market price of the neighborhood it leads into.
A corridor with one +1 at the goal cell (marked ✓). Bars show each cell’s fixed-point value after 200 Bellman sweeps.
Drop γ and the far cells go dark — same goal, different horizon priced. Raise it and value floods backward, but each hop costs a factor γ: the exponential decay of hope.
Every cell bootstraps: its value = reward stepping toward the goal + γ × (next cell’s value), which leverages the neighbor’s own estimate. The sweeps settle at the Bellman fixed point without anyone counting distance to the goal — recurrence alone. Illustrative: deterministic corridor, tabular values; real value learning is asynchronous, often off-policy, and table-free.
Why discount at all
The γ slider is not decoration. Small γ means near-term rewards dominate — the goal's value decays almost to nothing before it reaches the first cell. Large γ lets value travel far back, but only γ < 1 keeps the recursion convergent and the total promise finite: beyond some horizon, a far-away +1 is worth essentially zero. The first corridor cell at γ=0.99 carries nearly the same value as the goal cell itself; at γ=0.3 it is barely above zero. Same corridor, same future — the time horizon you price is a modeling decision, not an observation.
The recursion that bootstraps value backward
The Bellman equation for values under a policy:
Every arc in the artifact's corridor is that equation drawn at a state: the value here equals the reward next, plus the discounted value there — which itself encodes reward plus discounted value further along, and so on. "Bootstrapping" is used literally: each estimate bootstraps off its own current best guess, and the sweeps settle only when every state is at a fixed point. Notice what the recursions' solution buys: the goal's value needs no sensory access to the goal at all — one step of propagation anywhere in the corridor picks up exactly as much of the +1 as γ's decay allows.
The last lesson's Q-learning did this too — its artifact's
value += α(r + γ·V(next) − value) line is this equation flipped into
an update — but here the discount's effect is made visible rather
than assumed.
Illustrative vs real
A deterministic corridor, 200 sweeps of plainly visible fixed-point value iteration, and hand-drawn hallway arcs. Real value computation may be asynchronous (sweeps interleaved with acting), off-policy (Q-learning, updating toward max even on non-greedy steps), and over function approximators rather than a table — the extrapolation errors under that approximation are the "deadly triad" warning from the first lesson.
This lesson has exercises attached — chasing a discounted value one hop at a time with γ = 0.5 to see it halve per step — once the exercises layer ships.
Where next: policy gradients — the other classical route to a decision rule, where the policy is fit directly instead of built from values.