Exploration vs exploitation
To maximize you must first know — to know you must sacrifice reward. The bandit dilemma, ε-greedy, and why an apparently optimal agent can be permanently wrong.
The last lesson's agent learned because it wandered. Dampen that wander too hard and it locks onto the first decent thing it stumbles on. Push it too hard and it never stops sampling the useless arms. This is the exploration–exploitation dilemma, and it isn't a tuning nuisance — it is the definitional tension of acting to learn.
The restaurant you go to twice
Exploit: the known-good restaurant. Explore: the new one, which will probably be worse — but might not be, and you'll never find out without paying for the experiment once. Optimal play prices in the information the experiment buys, not just tonight's meal.
Four one-armed bandits, true win rates hidden. Your agent sees only its own pulls.
pulls: 0
dashed line: true rate (you can see it; the agent can’t)
Try ε = 0 with a couple of resets: unlucky first pulls can lock the agent onto a mediocre arm — forever, since nothing ever proves it wrong.
ε is the fraction of pulls spent as deliberate waste — information bought with reward. At ε = 0 nothing re-opens a closed question; at high ε estimates never settle. Illustrative: stationary Bernoulli arms and hand-rolled ε-greedy; real systems use UCB / Thompson sampling to explore curiosity-shaped rather than blindly.
The dilemma, made unforgiving
Set ε to 0 and watch the artifact: with unlucky first pulls (random seeds make this recurring) the agent commits to arm 0 forever and its average reward flatlines far below the best arm. The per-step regret clock keeps running. Pure greedy fails structurally, not unluckily — no amount of more pulling on the wrong arm reopens the comparison.
Push ε high and the opposite failure appears: the estimate jitter never settles because the agent keeps paying for experiments it no longer needs. ε-greedy's trick is that exploration is avowedly temporary: each random pull is a deliberate waste whose return is a tightened estimate.
The classical fixes beyond ε
ε-greedy explores uniformly — it is as likely to re-sample a proven loser as a barely-tried arm. The classical upgrades make exploration curiosity-shaped:
- Optimism under uncertainty (UCB): score each arm by estimate plus a bonus that shrinks as its sample count grows. "Pull what could be best, weighted by how sure you are."
- Thompson sampling: keep a distribution over beliefs about each arm and sample from them — arms you're uncertain about naturally get tried.
Both connect back to chapter 3's discipline: the bonus UCB pays is a bias–variance trade in action — willingly accepting 0.3's noise because it purchases information about 0.8.
Illustrative vs real
Four stationary Gaussian arms and a hand-rolled ε-greedy here. Real deployments may be nonstationary, context-sensitive (the best arm depends on the visitor — contextual bandits, the one-episode version of RL), and explored with one of these curiosity-shaped policies. The dilemma itself, though, they all pay.
This lesson has exercises attached — computing UCB scores for the four arms at a given pull count and predicting which arm it explores next — once the exercises layer ships.
Where next: value functions — turning "which pulls paid off?" into numbers you can compute a policy from.