The Wiki
Advanced7 min read

The attention bridge

Compression was the plan; attention says 'just look the word up.' The end of the middle shelf — and the door straight into the LLM book.

The seq2seq recipe (Sutskever et al., 2014): encoder RNN reads the input and squeezes it into one vector; decoder RNN writes the output from that vector. Everything in the source sentence — 4 words or 400 — must survive in a single hLh_L. The length cost is brutal:

source⟶hL    (fixed size)⟶output \text{source} \longrightarrow h_L \;\; \text{(fixed size)} \longrightarrow \text{output}

Attention (Bahdanau et al., 2014) was invented for this exact gap: let the decoder skip the bottleneck and consult the encoder's step by step, weighted:

αt=softmax(qtK⊤),ctxt=∑iαt,i vi \alpha_t = \text{softmax}(q_t K^\top), \qquad \text{ctx}_t = \sum_i \alpha_{t,i}\, v_i

From remembering to looking up

The recurrent contract is remember: compress, carry, refresh. Attention says: don't remember — index. The decoder keeps a key ("subject-verb agreement") and matches it against every stored position in parallel; the needed information travels along a path of length 1, not a path of length 400. Gradient-wise, it's this chapter's vanishing-past lesson resolved by architecture.

Decoder query: “who exactly was magnificent?” — via the bottleneck, or via lookup.

thechefwhotrainedinpariswasfamous

① plain seq2seq

entire sentence → one vector h_L → answer. Usable info per source word decays like 58% of any word during a 8-word sentence — and *all* info about ’chef’ must flow through the same vector it arrived in.

② with attention lookup

α-weights point directly at “chef” — information path length 1 from the position that holds the answer, no matter the sentence length.

Different query words point at different source positions — that per-query dynamic distribution *is* attention, at seq2seq-bridge scale. Note the lookup doesn’t compress anything: every stored position stays addressable throughout.

The α weights are a dot-product-plus-softmax exactly as in the equations (values shown hand-set to demonstrate the pivot, rather than trained). What’s real about the shift: path-length O(1) instead of O(n), no single-vector bottleneck, fully parallel compute — the three reasons attention won.

Why attention demolished the alternatives

By 2017 the arithmetic tipped transparently:

  1. Path length is constant. Any two positions interact in one hop — no depth-scaled products, no gated babysitting.
  2. Fully parallel. No loop: every position's query ships simultaneously, and wall-clock training gets big data as a serial-time winner, whereas RNNs spend real per-step time.
  3. No fixed-size summary. Capacity grows with sequence: a longer input just means more stored slots — the recurrent bottleneck is dropped as a category, not patched.

The transformer then kept only attention — dropping recurrence entirely — and scaled it into today's whole LLM story. Its specific costs (the quadratic O(L2)O(L^2), counteracted with efficient variants) and audio and image coverage (patches work like words) are the LLM book's concerns — this chapter's job was the why.

The middle shelf closes

Walk the arc one last time: a neuron is logistic regression revisited (a-neuron) → folds are capacity (activation-bends) → depth is the efficient way to earn folds (stacking) → the theorem validates the target, the training hacks make it survivable (training chapter) → convs make it cheap and hierarchical for images (vision chapter) → sequences demanded memory, compression failed, attention won. Now continue the story where the LLM book began:

→ tokens and embeddings

Illustrative vs real

The artifact replays the original bottleneck framing: fixed-length summary that overruns vs direct indexed lookup. Computation shown is a real similarity product; transformer internals (multi-head, query expanding, roles) are the LLM book's chapter — only the strategy pivot lives here.

This lesson has exercises attached — the seq2seq vs lookup routing comparison — launching once the exercises layer ships.