Wiki
Core9 min read

Generating text

How a model turns next-token probabilities into words — and where temperature and top-p come from.

After the last block, the model holds one number per vocabulary entry — a logit for every possible next token. Softmax turns those into probabilities. But probabilities are not text: somewhere, a decision must be made. That decision is called decoding, and it's a product choice layered on top of the model, not part of the model itself.

Start here

The model proposes a distribution ("'sat' 40%, 'sleeps' 22%, …"). Decoding decides how to pick from it: always the safest token, or a weighted roll of the dice. The same model can sound boring or wild purely by changing how it samples.

Greedy: the safe but repetitive choice

The simplest strategy is greedy decoding — always pick the highest probability token. It's locally optimal and globally dull: models decoded greedily loop, repeat phrases, and never take the slightly risky-but-better word. Real assistants sample instead.

Temperature: sharpening or flattening the dice

Temperature re-divides the logits by a value TT before softmax:

pi=exp⁡(li/T)∑jexp⁡(lj/T)p_i = \frac{\exp(l_i / T)}{\sum_j \exp(l_j / T)}
  • T<1T < 1 sharpens: differences between logits grow, the top token dominates. Confident and repetitive.
  • T>1T > 1 flattens: probabilities spread out, unlikely tokens get real chances. Creative and more often wrong.
  • T→0T \to 0 approaches greedy; T→∞T \to \infty approaches uniform random.

Try it — same logits, slider moves the temperature:

Next-token distribution for The cat ___

sat
49.8%
sleeps
20.2%
is
15.0%
will
8.2%
ate
6.1%
photon
0.6%
pick a temperature, then sample

moderate settings: a few plausible tokens share the chance · probabilities are real softmax math on illustrative logits

Notice what low temperature does to "photon" (it vanishes) and what high temperature does (it becomes possible). There is no "correct" temperature — it trades reliability for variety, which is why API products expose it as a user setting.

Top-p: cutting the tail honestly

Temperature alone has a failure mode: with TT high, even a 1-in-1000 token occasionally fires, and 1-in-1000 tokens are usually gibberish. Top-p (nucleus) sampling fixes the tail, not the temperature: sort tokens by probability, keep the smallest set whose probabilities sum to at least pp (say 0.9), and zero out the rest before sampling.

Toggle it in the playground: at top-p 0.5, "photon" is cut entirely while the plausible candidates still compete. Combined with a moderate temperature, this is the standard recipe behind most chat products.

Note

Decoding settings change nothing about the model's weights — the same checkpoint can answer a math question at near-greedy temperature or brainstorm story ideas at 1.3. When output quality varies wildly, decode settings are often the difference, not the model.

Check yourself

Eduspheria wiki · Large Language Models, Foundations

0 / 4 answered

  1. 1What does raising the temperature above 1 do to the next-token distribution?
    Multiple choice
  2. 2Temperature divides the logits by T before softmax. If T = 0.5, logits are effectively multiplied by what factor?
    Numeric answer
  3. 3Changing temperature or top-p changes the model's weights.
    True / false
  4. 4Which sampling method keeps the smallest set of tokens whose probabilities sum to at least p and zeroes out the rest?
    Short answer

Foundations complete

You can now explain the full journey: text → tokens → embeddings with positional signals → a stack of attention + feed-forward blocks → probabilities → decoding into words. The next chapter starts where this one ends: how such a stack is actually trained — the objective, the data, and the scale that makes it all work.