Tokens and embeddings
How raw text becomes numbers a model can actually compute with.
A language model never sees words. It sees numbers — specifically, a list of integers, and then a list of vectors. Everything else in this wiki builds on that one fact, so it's worth getting comfortable with before anything else.
Start here
If you remember nothing else from this lesson: tokenization chops text into pieces, and embeddings turn each piece into a point in space. Every later chapter — attention, training, alignment — is just operations on those points.
Splitting text into tokens
Real tokenizers (like BPE or SentencePiece) learn their vocabulary from a
huge corpus of text, merging frequent character pairs into larger and
larger chunks — the mechanics of that learning process get their own
lesson later (training/data-and-tokenization). The playground below
uses a much simpler rule — splitting on whitespace and punctuation — so
you can see the shape of the idea first.
Try typing a made-up word, or a word in a language other than English. Notice that whatever you type still gets split into some set of pieces — a real tokenizer's vocabulary is fixed, so it never fails, it just falls back to smaller and smaller chunks, down to individual bytes if it has to. That's why you'll never see a real tokenizer error out on unexpected input — there's no such thing as an out-of-vocabulary word once bytes are the fallback.
A rough budget: characters per token
You don't need to run a tokenizer to estimate cost. For English text, a useful rule of thumb is about 4 characters per token (roughly ¾ of a word per token). A 3,000-word English blog post is therefore around 4,000 tokens, not 3,000 — worth knowing before you paste something into a context window and wonder why it didn't fit.
Tokens are what you pay for and what fits
API pricing and context-window limits are both counted in tokens, not words or characters. "This model has a 200K context window" means 200K tokens — at ~4 characters each, that's roughly 150,000 words, but the exact number depends on what you're feeding it (see below).
Every model counts differently
There is no single universal tokenizer. GPT-family, Llama-family, and Claude-family models each ship their own vocabulary, trained on their own data mixture. The same paragraph can cost meaningfully different token counts depending on which model's tokenizer you run it through — a sentence that's 40 tokens in one vocabulary might be 55 in another. If you're comparing context budgets or per-token pricing across providers, you're implicitly comparing different tokenizers too, not just different prices.
Not all text is equally expensive
Non-English languages and source code tend to tokenize into more pieces per unit of meaning than English prose, because the vocabulary was learned mostly from English-heavy web data. A paragraph of Hindi or a block of Python can cost noticeably more tokens than an equivalent paragraph of English — the same context window buys you less of it.
Numbers are a special headache
Tokenizers don't reliably split digits at consistent boundaries. Depending
on the surrounding text, "1234" might become one token, or split as
"12" + "34", or "123" + "4" — and the split can change if you add
a comma or a space nearby. The model never sees digits the way you do,
as individual place values; it sees whatever chunks the tokenizer happened
to produce. This is a large part of why language models are
historically unreliable at multi-digit arithmetic: the input
representation itself obscures place value before any reasoning even
starts. (Some newer tokenizers special-case digits for exactly this
reason.)
Special tokens
Real vocabularies reserve a handful of IDs for structure rather than
text. A <BOS>/<EOS> pair marks where a sequence starts and ends; chat
models add role markers so the model can tell "this text came from the
user" apart from "this text came from me"; some vocabularies keep a
padding token for batching sequences of different lengths together. None
of these correspond to anything you typed — they're scaffolding the model
was trained to expect, and every prompt you send is quietly wrapped in
some of them before it reaches the model.
From tokens to vectors
Once text is a sequence of token IDs, the model looks each ID up in an embedding table — literally a big matrix where row is the vector for token :
where is the vocabulary size (often 50,000–200,000) and is the embedding dimension (often 1,000–10,000+ in large models). Looking up a token is just indexing a row:
# token_ids: list[int], shape (sequence_length,)
# E: embedding table, shape (vocab_size, d_model)
embeddings = E[token_ids] # shape (sequence_length, d_model)That's it — no computation yet, just a lookup. The vectors themselves start out as small random numbers and are learned during training, so that tokens used in similar contexts end up with similar vectors.
Just how big is this table?
With and , stored at 2 bytes per number, alone is — before the model has done a single layer of reasoning. That's why the embedding table is often the single largest set of parameters tied directly to vocabulary size rather than to "how smart" the model is.
The other end: tied weights
The embedding table isn't only used going in. At the very end of the model, something has to turn a final vector back into a probability over the whole vocabulary — that's a matrix multiply against a matrix, producing one score per possible next token. Many models simply reuse (transposed) for this instead of learning a second, separate matrix — a trick called weight tying. It roughly halves the vocabulary-sized parameter cost and bakes in a reasonable assumption: the same vector space used to represent "what token is this" going in is a sensible space for "how much does this look like each token" coming out.
Explore the space
Click a word to see its nearest neighbors in embedding space
Toy 2D layout for illustration — real embeddings have thousands of dimensions, and this plot was hand-placed so the analogy lands exactly on "queen". Real vector arithmetic on a trained model is far noisier than this and only works cleanly on a minority of relationships.
Click a word above to see which other words land nearest to it, or hit
the analogy button to watch the classic king − man + woman ≈ queen
vector arithmetic play out. This plot is a hand-placed 2D toy, not a
real model's embeddings — real ones live in thousands of dimensions and
can't be drawn on a page. But the two structural ideas it's showing are
real: nearby points tend to share meaning or usage, and some of the
directions between points correspond to consistent relationships (here,
"add royalty" or "swap gender").
The analogy demo is a best-case cherry pick
"king − man + woman ≈ queen" is real and reproducible on real embeddings, but it's also one of the cleanest examples that exists, and it gets quoted precisely because it works so well. Most token pairs don't have a clean, single direction separating them, and raw embedding vectors aren't as evenly spread through space as the toy plot above suggests — real embedding spaces are known to be anisotropic: they tend to bunch in a narrow cone rather than spreading out isotropically, which is part of why comparing two embeddings with plain cosine similarity is less trustworthy than it first appears. Treat "vectors encode meaning" as directionally true, not as a precise measuring instrument.
At this point the vectors know nothing about order — E[token_ids]
would produce the identical set of vectors no matter what order the
tokens came in, and some architectures additionally scale these vectors
(for example, by ) before anything else touches them. Where
each token sits in the sequence gets added on next — that's the subject
of the positions lesson later in this chapter.
Check yourself
Eduspheria wiki · Large Language Models, Foundations
0 / 4 answered
Next: how the model lets tokens look at each other — the attention mechanism.