The transformer block

Attention tells one token how to gather information from the others. But a token that has gathered information still has to process it, and a single round of both isn't enough to model language. The transformer's answer is almost comically simple: do those two moves over and over, in the same shape, dozens of times.

One block, two moves #

Here is the whole block, in code you could run today:

def transformer_block(x):
    x = x + Attention(LayerNorm(x))
    x = x + FeedForward(LayerNorm(x))
    return x

Read it as a loop over two sub-steps:

1. Attention — each token builds a new vector by blending in information from the other tokens (the lesson on attention).
2. Feed-forward — each token, now context-enriched, is processed independently by a small two-layer network. No token talks to another here; it's per-token "thinking time".

The order is the point. Attention is the only place tokens exchange information; the feed-forward network is where each token digests what it collected. A useful mental model: attention is communication, feed-forward is computation.

The feed-forward network #

The FFN is not fancy — it expands each token's vector to roughly 4× its size, applies a nonlinearity, and projects back:

with $W_1 \in \mathbb{R}^{d \times 4d}$ and $W_2 \in \mathbb{R}^{4d \times d}$. The GELU nonlinearity is essential — without it, two stacked linear layers would collapse into one big linear layer and the block could learn nothing new.

Why the additions matter: residuals #

Each sub-layer wraps its input in x = x + f(x) instead of replacing it. That small rewrite — a residual connection — is what makes deep stacks trainable.

During training, gradients flow backward from the loss to the earliest layers. Through an unbroken chain of matrix multiplies, those gradients shrink toward zero (the vanishing gradient problem). A residual connection gives them a shortcut: the gradient of x + f(x) with respect to x includes a plain +1 path, so information can flow through the whole stack untouched. LayerNorm before each sub-layer (computed: subtract the mean, divide by the standard deviation, rescale with learned weights) keeps the vector magnitudes from drifting layer after layer.

This lesson has exercises attached — counting the parameters of a block by hand and predicting which weights double when the model width doubles — once the exercises layer ships.

Stacking the blocks #

A real model applies the block $N$ times — each with its own learned weights, same shape:

Click through the stack: embeddings at the bottom (with positional information added — that's the other missing piece, covered in the positions-and-normalization lesson), $N$ identical blocks in the middle, and the LM head on top. Every block receives the previous block's output; the vector flowing up through these blocks is often called the residual stream — each sub-layer reads from it and writes back to it.

The output head: from vector to next token #

After the last block, the model takes the final vector for the last position and projects it to one score per vocabulary entry:

Softmax turns the raw scores into a probability distribution over the whole vocabulary — "the" 12%, "cat" 3%, and so on. How the model picks among those probabilities when writing text is its own lesson (generating-text).

Putting real numbers on it #

GPT-2 small, the classic reference model:

Scale that shape up — bigger $d$, more blocks, more heads — and the parameter count grows roughly with $N \cdot d^2$, which is why model sizes balloon so fast: double the width and you quadruple the parameters per block. The scale-and-scaling-laws lesson in the training chapter covers what that buys you.

What's still missing #

Two pieces are deliberately unexplained so far. First: attention itself is order-blind — "dog bites man" and "man bites dog" produce identical attention patterns without an extra signal for position, which is why the embedding stage adds one. Second: nothing here says how the whole stack is trained. Positional signals are covered in the positions-and-normalization lesson; the other fix — training — is the next chapter. But first, one upgrade to the block you just built: what if most of it didn't run for most tokens?

Next: mixture of experts — paying for parameters only when they run.