Generating text
How a model turns next-token probabilities into words — and where temperature and top-p come from.
After the last block, the model holds one number per vocabulary entry — a logit for every possible next token. Softmax turns those into probabilities. But probabilities are not text: somewhere, a decision must be made. That decision is called decoding, and it's a product choice layered on top of the model, not part of the model itself.
Start here
The model proposes a distribution ("'sat' 40%, 'sleeps' 22%, …"). Decoding decides how to pick from it: always the safest token, or a weighted roll of the dice. The same model can sound boring or wild purely by changing how it samples.
Greedy: the safe but repetitive choice
The simplest strategy is greedy decoding — always pick the highest probability token. It's locally optimal and globally dull: models decoded greedily loop, repeat phrases, and never take the slightly risky-but-better word. Real assistants sample instead.
Temperature: sharpening or flattening the dice
Temperature re-divides the logits by a value before softmax:
- sharpens: differences between logits grow, the top token dominates. Confident and repetitive.
- flattens: probabilities spread out, unlikely tokens get real chances. Creative and more often wrong.
- approaches greedy; approaches uniform random.
Try it — same logits, slider moves the temperature:
Next-token distribution for The cat ___
moderate settings: a few plausible tokens share the chance · probabilities are real softmax math on illustrative logits
Notice what low temperature does to "photon" (it vanishes) and what high temperature does (it becomes possible). There is no "correct" temperature — it trades reliability for variety, which is why API products expose it as a user setting.
Top-p: cutting the tail honestly
Temperature alone has a failure mode: with high, even a 1-in-1000 token occasionally fires, and 1-in-1000 tokens are usually gibberish. Top-p (nucleus) sampling fixes the tail, not the temperature: sort tokens by probability, keep the smallest set whose probabilities sum to at least (say 0.9), and zero out the rest before sampling.
Toggle it in the playground: at top-p 0.5, "photon" is cut entirely while the plausible candidates still compete. Combined with a moderate temperature, this is the standard recipe behind most chat products.
Note
Decoding settings change nothing about the model's weights — the same checkpoint can answer a math question at near-greedy temperature or brainstorm story ideas at 1.3. When output quality varies wildly, decode settings are often the difference, not the model.
Check yourself
Eduspheria wiki · Large Language Models, Foundations
0 / 4 answered
Foundations complete
You can now explain the full journey: text → tokens → embeddings with positional signals → a stack of attention + feed-forward blocks → probabilities → decoding into words. The next chapter starts where this one ends: how such a stack is actually trained — the objective, the data, and the scale that makes it all work.