Generating text

After the last block, the model holds one number per vocabulary entry — a logit for every possible next token. Softmax turns those into probabilities. But probabilities are not text: somewhere, a decision must be made. That decision is called decoding, and it's a product choice layered on top of the model, not part of the model itself.

Greedy: the safe but repetitive choice #

The simplest strategy is greedy decoding — always pick the highest probability token. It's locally optimal and globally dull: models decoded greedily loop, repeat phrases, and never take the slightly risky-but-better word. Real assistants sample instead.

Temperature: sharpening or flattening the dice #

Temperature re-divides the logits by a value $T$ before softmax:

$$p_i = \frac{\exp(l_i / T)}{\sum_j \exp(l_j / T)}$$

• $T < 1$ sharpens: differences between logits grow, the top token dominates. Confident and repetitive.
• $T > 1$ flattens: probabilities spread out, unlikely tokens get real chances. Creative and more often wrong.
• $T \to 0$ approaches greedy; $T \to \infty$ approaches uniform random.

Try it — same logits, slider moves the temperature:

Notice what low temperature does to "photon" (it vanishes) and what high temperature does (it becomes possible). There is no "correct" temperature — it trades reliability for variety, which is why API products expose it as a user setting.

Top-p: cutting the tail honestly #

Temperature alone has a failure mode: with $T$ high, even a 1-in-1000 token occasionally fires, and 1-in-1000 tokens are usually gibberish. Top-p (nucleus) sampling fixes the tail, not the temperature: sort tokens by probability, keep the smallest set whose probabilities sum to at least $p$ (say 0.9), and zero out the rest before sampling.

Toggle it in the playground: at top-p 0.5, "photon" is cut entirely while the plausible candidates still compete. Combined with a moderate temperature, this is the standard recipe behind most chat products.

Foundations complete #

You can now explain the full journey: text → tokens → embeddings with positional signals → a stack of attention + feed-forward blocks → probabilities → decoding into words. The next chapter starts where this one ends: how such a stack is actually trained — the objective, the data, and the scale that makes it all work.