Reasoning models
From asking a model to think out loud to training it to — RL on verifiable rewards, test-time compute, what a 'thinking' budget buys, and the opposite trade: System One decision models.
The previous lesson treated chain-of-thought as prompting — asking a frozen model to use tokens as scratch space. The defining 2024–2026 shift is that the thinking itself is now trained: reasoning models (o1, DeepSeek-R1, Gemini Thinking, Claude extended thinking) learn to produce long deliberation before answering, because it demonstrably raises accuracy on hard problems. The distinction matters:
- Prompted CoT — same weights, different context. Gains are real but capped by what the base model already does reliably.
- Trained reasoning — post-training rewards successful long deliberation, so the model learns strategies (setting up equations, backtracking, self-checking) it never produced before.
Start here
A reasoning model isn't a smarter next-token predictor — it's the same predictor allowed, and trained, to spend 1000× more tokens on a question than the question's answer needs. Where normal decoding buys the answer, thinking-buying buys verification: the extra tokens are mostly the model checking its own work.
Training it: RL on verifiable rewards
Preference data (the RLHF lesson) can't grade reasoning quality — humans can't score 2000 tokens of deliberation reliably. The unlock: choose domains where correctness is mechanically checkable —
- math: the final answer is a number you can compare,
- code: run the tests,
- formal logic: check the proof.
So the reward is no longer a learned reward model but the environment itself (RLVR — RL with verifiable rewards). The policy generates many candidate solutions per problem; those that pass get their thinking reinforced, including the patterns of thinking, not just answers.
The standard algorithm is a simplified PPO called GRPO (group relative policy optimization): for each prompt, sample a group of responses, score them, and use the group mean as the baseline instead of a learned critic network:
The gradient pushes up responses that beat their group-mates. It's RLHF's machinery minus the reward model and critic — and the emergence of long self-checking chains under this training (DeepSeek-R1 was the loud public demonstration) surprised even its authors.
Test-time compute: the new scaling axis
Watch what a thinking budget does to one hard problem:
One hard problem, one model — only the thinking budget changes
Problem: "A farmer has 24 apples in three baskets. The second basket has twice the first; the third equals the first. How many in each?"
sets up equations
The characteristic shape: steep early gains as the budget buys actual setup and self-checking, then a plateau — extra thinking rarely rescues a model past its ceiling. Curve and samples are illustrative; on easy problems the curve is flat from token zero, and the cost is paid in latency whether or not accuracy moves.
This is the second scaling law: pretraining scales capability with compute spent before deployment; reasoning models spend compute at inference, per question, and their accuracy-vs-thinking curve behaves like a (noisier) training curve. It reframes serving economics: the same model is a fast cheap assistant at zero thinking budget and a slower expensive expert at a large one — the user chooses the point on the curve.
Three practitioner notes:
- Match budget to difficulty. On easy questions the curve is flat from zero tokens — thinking mode is pure latency and cost waste there. Routing easy/simple queries to non-thinking modes is now a standard product pattern.
- The visible chain is not the full computation. What's shown can be summarized or selectively edited; treat it as evidence of process, not a proof transcript (the prompting lesson's caveat, industrialized).
- Trained CoT can be hacked too. RL on verifiable rewards finds exploits when rewards are subtly wrong (a buggy test suite is a gradient toward passing that suite) — the reward-hacking story of the RLHF lesson, transplanted.
System One and System Two
Cognitive science splits thought into System 1 — fast, automatic, intuitive — and System 2 — slow, deliberate, effortful (Kahneman's Thinking, Fast and Slow). It maps onto models more cleanly than most metaphors: a plain chat model answers on its first impulse, while a reasoning model spends tokens to deliberate. Keep it a metaphor — nothing in a transformer is literally "intuitive" — but it names the two trades this lesson is about.
The names collide
s1 is a model (Stanford's minimal test-time-scaling replication).
"System One" is a category — and TypeSafe AI's Jev is a System One
model, not "the s1 model." Different things that share a numeral.
s1 — the cheapest System-2 model. The s1 model is the smallest
demonstration of the second scaling law: supervised fine-tuning of
Qwen2.5-32B on just 1,000 curated reasoning traces (s1K) — 26
minutes on 16 H100s — plus budget forcing at inference, which
suppresses the end-of-thinking token and appends "Wait" to make the model
keep deliberating (or forces an early stop to cap it). It matches
o1-preview on competition math and extrapolates on AIME24 (50%→57%) by
thinking longer. The takeaway: the reasoning was already latent in
pretraining; a sliver of traces plus a decoding trick activates and
scales it. (Its successor is s1.1.)
Jev — a System One model. In September 2026 TypeSafe AI released Jev, first of a class it calls System One models: models that don't generate text at all. You send a state (text or JSON) and typed questions — pick one, score a level, answer yes/no — and get back typed values with calibrated probabilities in a single parallel pass. The vendor reports 40–200× lower latency and roughly 400× lower cost than frontier LLMs on these classification-shaped tasks, and the "can't hallucinate" claim is literal: it never emits a string, so there is no string to invent.
Careful
Read Jev's numbers as vendor-reported, on classification-shaped tasks, from an early-access, text-only, proprietary model — not a benchmark. "Can't hallucinate" means "can't fabricate prose," not "always right": a calibrated 0.94 is still wrong about 6% of the time. A decision model is a complement to a reasoning LLM — routing, extraction, guardrails — not a replacement for open-ended reasoning.
Same decision, two shapes of model
Ticket #4821 — "I was charged twice for the same subscription this month, please refund the duplicate."
Question: which team should handle this ticket?
Customer reports a duplicate subscription charge…
…and explicitly asks to refund the duplicate.
choice
No chain of thought, no sentence to parse — one pass, one typed answer with a probability your code can branch on.
response time (illustrative)
System Two buys accuracy on hard problems by spending tokens to deliberate; System One buys speed on easy, high-volume ones by spending none. The two are complements: route and gate with a decision model, reason with an LLM. System One figures are vendor-reported and illustrative — not a head-to-head benchmark.
The two axes are opposite trades. System 2 spends more tokens to buy accuracy on hard problems; System 1 spends none to buy speed on easy, high-volume ones. A production stack increasingly uses both: a fast decision model to route and gate, a slow reasoning model for whatever survives the gate.
Distillation closes the loop
R1's second headline: long-horizon reasoning distills. Small models fine-tuned on the reasoning model's outputs recover much of the behavior without any RL — the efficient-adaptation lesson's tooling applied to a new capability. This is why cheap local models got dramatically better at math in one release cycle: frontier compute bought the reasoning once; everyone else copied the traces.
Check yourself
Eduspheria wiki · Large Language Models, Frontier
0 / 5 answered
Next: giving the thinking model knowledge it doesn't have — retrieval.
This lesson has exercises attached — predicting which problems benefit from a thinking budget and spotting a reward-hacked chain — once the exercises layer ships.