Embeddings and similarity
Everything in this book ran on hand-picked features. The classical end of the line is learning the features too — and that idea goes all the way up.
Every model so far leaned on features you chose: age, distance, spend. The classical final boss is dropping that too: learn the representation itself — map each item into a vector space where "similar meaning" becomes "nearby point," with nobody writing down what the axes mean.
The exchange you're making
You give up being able to name the features ("coordinate 47 is, uh, customer-loyalty-ish?") and get back a similarity function you never had to hand-engineer. Ugly to explain to a human, fantastic to compute with.
A learned vector space: meaning = direction, similarity = angle
Colors: royal / animal / food. Note clusters formed purely from the vectors’ own geometry.
Cosine similarity — angle, not distance — is the readout, because in learned spaces magnitude is often meaningless. Try “king − man”: no one wrote the rule that it lands near “queen”; the gradient descent that built this space couldn’t help arranging meaning geometrically. That’s the whole classical story — learned representation first, then the chapter-2 algorithms — and the LLM book starts exactly here, at scale.
How the space gets built — classical edition
You learn vectors by giving the model a task about relationships and letting gradients do the feature work. The classical exemplar is matrix factorization for recommendations: users × items interactions are one giant sparse matrix; finding two small matrices (user-vectors × item-vectors) whose product best explains the observed ratings is learned PCA — the vectors' nearness in the learned space is the similarity. Word2vec did the same trick to text (predict a word from its neighbors → words that fill the same role land near each other), and it's the reason "king − man + woman ≈ queen" became a party trick: directions in representation space carry meaning, not just position.
Ugly in one way, universal in another
The coordinates are famously uninterpretable — the LLM book's "superposition" chapter factors into this exactly. But interpretability was traded away knowingly, because what you get in return is composability:
- k-NN earlier ran on your hand-built feature space; run it on an embedding space and it becomes similarity search over anything with a representation.
- Nearest-neighbor retrieval in embeddings became the retrieval half of RAG ("give me document chunks near this query vector") — the classical ideas plus scale, nothing else.
- The LLM book's "tokens and embeddings" lesson is this lesson done in billions of dimensions: the embedding IS the learned features.
The reorientation worth leaving with
Chapter 1 said models fit parameters to data. The classical arc of this book ends here with the fuller sentence: the feature space itself is also something you can fit — and once you know that, "machine learning" stops being a list of algorithms and becomes one practice: choose a geometry where your task becomes easy, then fit inside it.
Illustrative vs real
Hand-made vectors with interpretability aid here. Real embedding spaces are 100s-1000s of dimensions, trained by contrastive objectives, and evaluations happen through downstream tasks — never by staring at coordinates.
Where next: the LLM domain closes the loop — its first lesson starts from this exact sentence.