Eduspheria Wiki
Advanced12 min read

Reasoning models

From asking a model to think out loud to training it to — RL on verifiable rewards, test-time compute, and what a 'thinking' budget buys.

The previous lesson treated chain-of-thought as prompting — asking a frozen model to use tokens as scratch space. The defining 2024–2026 shift is that the thinking itself is now trained: reasoning models (o1, DeepSeek-R1, Gemini Thinking, Claude extended thinking) learn to produce long deliberation before answering, because it demonstrably raises accuracy on hard problems. The distinction matters:

  • Prompted CoT — same weights, different context. Gains are real but capped by what the base model already does reliably.
  • Trained reasoning — post-training rewards successful long deliberation, so the model learns strategies (setting up equations, backtracking, self-checking) it never produced before.

Start here

A reasoning model isn't a smarter next-token predictor — it's the same predictor allowed, and trained, to spend 1000× more tokens on a question than the question's answer needs. Where normal decoding buys the answer, thinking-buying buys verification: the extra tokens are mostly the model checking its own work.

Training it: RL on verifiable rewards

Preference data (the RLHF lesson) can't grade reasoning quality — humans can't score 2000 tokens of deliberation reliably. The unlock: choose domains where correctness is mechanically checkable —

  • math: the final answer is a number you can compare,
  • code: run the tests,
  • formal logic: check the proof.

So the reward is no longer a learned reward model but the environment itself (RLVR — RL with verifiable rewards). The policy generates many candidate solutions per problem; those that pass get their thinking reinforced, including the patterns of thinking, not just answers.

The standard algorithm is a simplified PPO called GRPO (group relative policy optimization): for each prompt, sample a group of GG responses, score them, and use the group mean as the baseline instead of a learned critic network:

A^i=ri−mean(r1..rG)std(r1..rG)\hat{A}_i = \frac{r_i - \text{mean}(r_1..r_G)}{\text{std}(r_1..r_G)}

The gradient pushes up responses that beat their group-mates. It's RLHF's machinery minus the reward model and critic — and the emergence of long self-checking chains under this training (DeepSeek-R1 was the loud public demonstration) surprised even its authors.

Test-time compute: the new scaling axis

Watch what a thinking budget does to one hard problem:

One hard problem, one model — only the thinking budget changes

thinking tokens spent →accuracy50%

Problem: "A farmer has 24 apples in three baskets. The second basket has twice the first; the third equals the first. How many in each?"

sets up equations

a + b + c = 24, b = a + c, and 'twice' means b = 2a → c = a, so 4a = 24 → a = 6, b = 12, c = 6… but let me check the 'twice' reading before committing — could also mean b = 2a only, with c free…

The characteristic shape: steep early gains as the budget buys actual setup and self-checking, then a plateau — extra thinking rarely rescues a model past its ceiling. Curve and samples are illustrative; on easy problems the curve is flat from token zero, and the cost is paid in latency whether or not accuracy moves.

This is the second scaling law: pretraining scales capability with compute spent before deployment; reasoning models spend compute at inference, per question, and their accuracy-vs-thinking curve behaves like a (noisier) training curve. It reframes serving economics: the same model is a fast cheap assistant at zero thinking budget and a slower expensive expert at a large one — the user chooses the point on the curve.

Three practitioner notes:

  • Match budget to difficulty. On easy questions the curve is flat from zero tokens — thinking mode is pure latency and cost waste there. Routing easy/simple queries to non-thinking modes is now a standard product pattern.
  • The visible chain is not the full computation. What's shown can be summarized or selectively edited; treat it as evidence of process, not a proof transcript (the prompting lesson's caveat, industrialized).
  • Trained CoT can be hacked too. RL on verifiable rewards finds exploits when rewards are subtly wrong (a buggy test suite is a gradient toward passing that suite) — the reward-hacking story of the RLHF lesson, transplanted.

Distillation closes the loop

R1's second headline: long-horizon reasoning distills. Small models fine-tuned on the reasoning model's outputs recover much of the behavior without any RL — the efficient-adaptation lesson's tooling applied to a new capability. This is why cheap local models got dramatically better at math in one release cycle: frontier compute bought the reasoning once; everyone else copied the traces.

Next: giving the thinking model knowledge it doesn't have — retrieval.

This lesson has exercises attached — predicting which problems benefit from a thinking budget and spotting a reward-hacked chain — once the exercises layer ships.