Wiki
Advanced14 min read

Reasoning models

From asking a model to think out loud to training it to — RL on verifiable rewards, test-time compute, what a 'thinking' budget buys, and the opposite trade: System One decision models.

The previous lesson treated chain-of-thought as prompting — asking a frozen model to use tokens as scratch space. The defining 2024–2026 shift is that the thinking itself is now trained: reasoning models (o1, DeepSeek-R1, Gemini Thinking, Claude extended thinking) learn to produce long deliberation before answering, because it demonstrably raises accuracy on hard problems. The distinction matters:

  • Prompted CoT — same weights, different context. Gains are real but capped by what the base model already does reliably.
  • Trained reasoning — post-training rewards successful long deliberation, so the model learns strategies (setting up equations, backtracking, self-checking) it never produced before.

Start here

A reasoning model isn't a smarter next-token predictor — it's the same predictor allowed, and trained, to spend 1000× more tokens on a question than the question's answer needs. Where normal decoding buys the answer, thinking-buying buys verification: the extra tokens are mostly the model checking its own work.

Training it: RL on verifiable rewards

Preference data (the RLHF lesson) can't grade reasoning quality — humans can't score 2000 tokens of deliberation reliably. The unlock: choose domains where correctness is mechanically checkable —

  • math: the final answer is a number you can compare,
  • code: run the tests,
  • formal logic: check the proof.

So the reward is no longer a learned reward model but the environment itself (RLVR — RL with verifiable rewards). The policy generates many candidate solutions per problem; those that pass get their thinking reinforced, including the patterns of thinking, not just answers.

The standard algorithm is a simplified PPO called GRPO (group relative policy optimization): for each prompt, sample a group of GG responses, score them, and use the group mean as the baseline instead of a learned critic network:

A^i=ri−mean(r1..rG)std(r1..rG)\hat{A}_i = \frac{r_i - \text{mean}(r_1..r_G)}{\text{std}(r_1..r_G)}

The gradient pushes up responses that beat their group-mates. It's RLHF's machinery minus the reward model and critic — and the emergence of long self-checking chains under this training (DeepSeek-R1 was the loud public demonstration) surprised even its authors.

Test-time compute: the new scaling axis

Watch what a thinking budget does to one hard problem:

One hard problem, one model — only the thinking budget changes

thinking tokens spent →accuracy50%

Problem: "A farmer has 24 apples in three baskets. The second basket has twice the first; the third equals the first. How many in each?"

sets up equations

a + b + c = 24, b = a + c, and 'twice' means b = 2a → c = a, so 4a = 24 → a = 6, b = 12, c = 6… but let me check the 'twice' reading before committing — could also mean b = 2a only, with c free…

The characteristic shape: steep early gains as the budget buys actual setup and self-checking, then a plateau — extra thinking rarely rescues a model past its ceiling. Curve and samples are illustrative; on easy problems the curve is flat from token zero, and the cost is paid in latency whether or not accuracy moves.

This is the second scaling law: pretraining scales capability with compute spent before deployment; reasoning models spend compute at inference, per question, and their accuracy-vs-thinking curve behaves like a (noisier) training curve. It reframes serving economics: the same model is a fast cheap assistant at zero thinking budget and a slower expensive expert at a large one — the user chooses the point on the curve.

Three practitioner notes:

  • Match budget to difficulty. On easy questions the curve is flat from zero tokens — thinking mode is pure latency and cost waste there. Routing easy/simple queries to non-thinking modes is now a standard product pattern.
  • The visible chain is not the full computation. What's shown can be summarized or selectively edited; treat it as evidence of process, not a proof transcript (the prompting lesson's caveat, industrialized).
  • Trained CoT can be hacked too. RL on verifiable rewards finds exploits when rewards are subtly wrong (a buggy test suite is a gradient toward passing that suite) — the reward-hacking story of the RLHF lesson, transplanted.

System One and System Two

Cognitive science splits thought into System 1 — fast, automatic, intuitive — and System 2 — slow, deliberate, effortful (Kahneman's Thinking, Fast and Slow). It maps onto models more cleanly than most metaphors: a plain chat model answers on its first impulse, while a reasoning model spends tokens to deliberate. Keep it a metaphor — nothing in a transformer is literally "intuitive" — but it names the two trades this lesson is about.

The names collide

s1 is a model (Stanford's minimal test-time-scaling replication). "System One" is a category — and TypeSafe AI's Jev is a System One model, not "the s1 model." Different things that share a numeral.

s1 — the cheapest System-2 model. The s1 model is the smallest demonstration of the second scaling law: supervised fine-tuning of Qwen2.5-32B on just 1,000 curated reasoning traces (s1K) — 26 minutes on 16 H100s — plus budget forcing at inference, which suppresses the end-of-thinking token and appends "Wait" to make the model keep deliberating (or forces an early stop to cap it). It matches o1-preview on competition math and extrapolates on AIME24 (50%→57%) by thinking longer. The takeaway: the reasoning was already latent in pretraining; a sliver of traces plus a decoding trick activates and scales it. (Its successor is s1.1.)

Jev — a System One model. In September 2026 TypeSafe AI released Jev, first of a class it calls System One models: models that don't generate text at all. You send a state (text or JSON) and typed questions — pick one, score a level, answer yes/no — and get back typed values with calibrated probabilities in a single parallel pass. The vendor reports 40–200× lower latency and roughly 400× lower cost than frontier LLMs on these classification-shaped tasks, and the "can't hallucinate" claim is literal: it never emits a string, so there is no string to invent.

Careful

Read Jev's numbers as vendor-reported, on classification-shaped tasks, from an early-access, text-only, proprietary model — not a benchmark. "Can't hallucinate" means "can't fabricate prose," not "always right": a calibrated 0.94 is still wrong about 6% of the time. A decision model is a complement to a reasoning LLM — routing, extraction, guardrails — not a replacement for open-ended reasoning.

Same decision, two shapes of model

Ticket #4821 — "I was charged twice for the same subscription this month, please refund the duplicate."

Question: which team should handle this ticket?

System Two — reasoning LLMgenerates text

Customer reports a duplicate subscription charge…

…and explicitly asks to refund the duplicate.

"This looks like a billing issue — the customer was charged twice and wants the duplicate refunded."
System One — decision modeltyped values only

choice

billing
0.94
technical
0.04
account
0.02
choice: "billing" · confidence: 0.94

No chain of thought, no sentence to parse — one pass, one typed answer with a probability your code can branch on.

response time (illustrative)

System Two
0.95s
System One
0.11s

System Two buys accuracy on hard problems by spending tokens to deliberate; System One buys speed on easy, high-volume ones by spending none. The two are complements: route and gate with a decision model, reason with an LLM. System One figures are vendor-reported and illustrative — not a head-to-head benchmark.

The two axes are opposite trades. System 2 spends more tokens to buy accuracy on hard problems; System 1 spends none to buy speed on easy, high-volume ones. A production stack increasingly uses both: a fast decision model to route and gate, a slow reasoning model for whatever survives the gate.

Distillation closes the loop

R1's second headline: long-horizon reasoning distills. Small models fine-tuned on the reasoning model's outputs recover much of the behavior without any RL — the efficient-adaptation lesson's tooling applied to a new capability. This is why cheap local models got dramatically better at math in one release cycle: frontier compute bought the reasoning once; everyone else copied the traces.

Check yourself

Eduspheria wiki · Large Language Models, Frontier

0 / 5 answered

  1. 1What reward signal do reasoning models like R1 train on in RLVR?
    Multiple choice
  2. 2The s1 model was fine-tuned on how many curated reasoning traces, its s1K set?
    Numeric answer
  3. 3In GRPO, the baseline for a response's advantage is the mean score of its sampled group, not a learned critic network.
    True / false
  4. 4What is the name of the second scaling axis where compute is spent at inference per question?
    Short answer
  5. 5Why is thinking mode wasteful on easy questions?
    Multiple choice

Next: giving the thinking model knowledge it doesn't have — retrieval.

This lesson has exercises attached — predicting which problems benefit from a thinking budget and spotting a reward-hacked chain — once the exercises layer ships.