Tokens and embeddings

A language model never sees words. It sees numbers — specifically, a list of integers, and then a list of vectors. Everything else in this wiki builds on that one fact, so it's worth getting comfortable with before anything else.

Splitting text into tokens #

Real tokenizers (like BPE or SentencePiece) learn their vocabulary from a huge corpus of text, merging frequent character pairs into larger and larger chunks. The playground below uses a much simpler rule — splitting on whitespace and punctuation — so you can see the shape of the idea before the data-and-tokenization lesson in the training chapter gets into the mechanics.

Try typing a made-up word, or a word in a language other than English. Notice that whatever you type still gets split into some set of pieces — a real tokenizer's vocabulary is fixed, so it never fails, it just falls back to smaller and smaller chunks (down to individual bytes if it has to).

From tokens to vectors #

Once text is a sequence of token IDs, the model looks each ID up in an embedding table — literally a big matrix where row $i$ is the vector for token $i$:

where $V$ is the vocabulary size (often 50,000–200,000) and $d$ is the embedding dimension (often 1,000–10,000+ in large models). Looking up a token is just indexing a row:

# token_ids: list[int], shape (sequence_length,)
# E: embedding table, shape (vocab_size, d_model)
embeddings = E[token_ids]  # shape (sequence_length, d_model)

That's it — no computation yet, just a lookup. The vectors themselves start out random and are learned during training, so that tokens used in similar contexts end up with similar vectors. That's the whole basis for the famous "king − man + woman ≈ queen" observation: it's a byproduct of training, not something anyone hand-designed.

Next: how the model lets tokens look at each other — the attention mechanism.