Positions and normalization
How the transformer sees word order — and the plumbing that keeps a deep stack trainable.
Attention has a hidden weakness: by itself, it has no idea what order tokens come in. The attention formula compares every token against every other token — nothing in that computation knows whether "dog" came before "bites" or after it. This lesson covers the fix, plus the other piece of plumbing every deep stack needs.
Start here
Attention treats its input like a bag of vectors — it computes the same thing no matter how you shuffle them. To restore word order, the model adds a positional signal to each token's embedding before anything else runs.
The order-blindness proof
Swap the words in a sentence and watch what happens to attention:
Same three words — swap the order and watch attention
No positions: both orders produce the exact same attention grid — the model literally cannot tell 'dog bites man' from 'man bites dog'. Toy vectors, but the order-blindness is real.
Without the positional toggle, both orders produce the identical attention grid — and identical next-token predictions. "Dog bites man" and "man bites dog" would be the same event. With the positional signal added to each embedding, the vectors differ per position and attention picks up on order.
How positions are encoded
Three approaches you'll meet, in increasing modernity:
- Sinusoidal (original paper) — fixed sine/cosine waves of different frequencies added to each position. No learned parameters; clever, but rigid.
- Learned positions — a lookup table of one vector per position, trained like the embeddings (GPT-2 style). Simple and effective up to the maximum length the table was trained for.
- Rotary (RoPE) — instead of adding a position vector, each attention head's query/key vectors are rotated by an angle that depends on position. Dot products then naturally encode relative distance, and models can sometimes generalize to longer contexts than they were trained on. The dominant choice in modern LLMs.
The rotation deliverable:
Rotate by position — the dot product only sees the offset
-0.63
attention score = cos(Δθ)
Δθ = 2.25 rad (pos gap 3)
One dimension pair: RoPE rotates q and k by angle × (position × frequency), so the dot product is cos of the rotation *difference* — a function of relative offset only. "Slow pair / fast pair" shows real RoPE's multi-frequency trick: each dimension pair spins at a different rate, so nearby positions differ in fast dims (fine-grained distance) while slow dims keep long-range information.
Note
You don't need the trigonometry — the takeaway is architectural: position information is added (or baked into attention) before the first block, and every later layer inherits it through the residual stream.
Keeping magnitudes sane: normalization
The second piece of plumbing solves a quieter problem. Each block adds and multiplies vectors dozens of times; across layers, vector magnitudes can drift — some components exploding, others shrinking. Feeding wildly-scaled vectors into softmax or gradients is how deep networks destabilize.
LayerNorm fixes this per token: subtract the mean of the vector's components, divide by the standard deviation, then rescale with two learned parameters ( to stretch, to shift):
Modern LLMs mostly use RMSNorm, a cheaper variant that skips the mean-subtraction and just rescales by the root-mean-square — same stabilizing effect, slightly less compute.
In the block from the transformer-architecture lesson, normalization
runs before each
sub-layer (pre-norm: x + Attn(LayerNorm(x))). That ordering is a
refinement found by later experiments — post-norm transformers (the
original paper's order) train less reliably at depth.
Careful
Normalization and residuals don't add intelligence — they remove obstacles. You could describe a transformer without them, but no serious model ships without them, because the 20-block stack simply won't train.
Where you are
You now have the complete architecture: tokens → embeddings with positional signals → a stack of (attention + feed-forward, wrapped in residuals and norms) → probabilities over the vocabulary. One detour remains — multimodality — and then, in the chapter's last lesson, the question behind every user-facing knob: when the model outputs a probability distribution, how does it actually choose the next token? That's the generating-text lesson, and it's where settings like "temperature" come from.