Wiki
Core10 min read

Data and tokenization

How a tokenizer vocabulary is learned, and how trillions of tokens get selected before training starts.

A model can only learn from the vocabulary it's given. That vocabulary isn't hand-written — it's learned from the data itself by an algorithm called Byte-Pair Encoding (BPE), and the same selection discipline applies to the training text too. This lesson covers both halves of the fuel problem: how text becomes tokens, and which text is worth training on.

Start here

BPE starts with single characters and repeatedly merges the most frequent adjacent pair. Common words fuse into single tokens; rare words stay as pieces. The vocabulary ends up sized to the data.

Watching BPE learn

Step through the actual algorithm on a tiny corpus — every merge is the real BPE rule, just on a small enough corpus to see:

BPE learning a vocabulary from a tiny corpus (low, lower, slowest, slowly, lowest)

merge 0/6
low·
lower·
slowest·
slowly·
lowest·
next: (l, o) appears 5×

Each step merges the most frequent adjacent pair across the whole corpus — the real algorithm, on a one-word-family corpus so every merge is visible. · marks the end-of-word boundary.

Three things to notice. First, "low" fuses into one token quickly — frequency is all BPE cares about. Second, shared endings ("-est", "-ly") emerge as natural sub-tokens, which is how real vocabularies handle morphology. Third, the special end-of-word marker (·) matters: it's why "low" and the "low" inside "lower" can be different tokens — the tokenizer learns word-boundary-sensitive pieces.

From toy to real

Real tokenizers run this exact loop over large samples of the corpus (gigabytes to hundreds of gigabytes — the sample, not the full crawl, is what trains the vocabulary), merging until the vocabulary reaches a target size (commonly 50k–200k tokens). Two consequences worth internalizing:

  • No out-of-vocabulary failures. Worst case, a word falls back to bytes — every possible text is encodable.
  • Non-English and code pay a token tax. Rare languages and code tokenize into many more pieces per meaning, so the same paragraph costs more context (more positions to attend across). Vocabulary choices have real downstream effects.

That fallback is worth seeing directly. Below, the merge table is learned from a tiny corpus with the same algorithm, then applied to whatever you type — so familiar pieces fuse and unfamiliar ones break down to single characters:

tokenizers·tokenize·unfamiliar·words·
18 tokens · vocabulary 30 · 50 merges learned

Real BPE: the merge table is learned from a 32-word built-in corpus with the actual algorithm, then applied greedily to your text — the same way a production tokenizer applies its frozen merges. The vocabulary here is tiny, so unfamiliar words fall back to single characters (dashed), the character-level version of a real tokenizer's byte fallback. · marks the end-of-word boundary.

The data funnel

Pretraining corpora start larger than they end. The funnel:

  1. Gather — web crawls, books, code, papers: petabytes of raw text.
  2. Filter — deduplicate (near-exact and fuzzy), drop spam and boilerplate, score quality, balance domains. Labs commonly keep a small fraction of what they crawl.
  3. Mix — decide proportions: how much code, how much math, how much conversation. The mixture is a major capability decision — it shapes what the final model is good at.
  4. Tokenize — BPE over the mix; everything becomes token IDs.

Note

Frontier labs are publicly cagey about data mixtures because the mixture is capability. Two labs with the same architecture and compute can produce very different models purely from step 3.

Why this comes before training

The tokenizer is frozen before the model trains: the embedding table one row per token, the LM head one column per token. Change the vocabulary later and both must be rebuilt. That's why data and tokenization are decided first — the objective in the next lesson then only ever sees token IDs.

Check yourself

Eduspheria wiki · Large Language Models, Training

0 / 4 answered

  1. 1What algorithm learns a tokenizer vocabulary by repeatedly merging the most frequent adjacent pair?
    Short answer
  2. 2Why is the tokenizer frozen before model training begins?
    Multiple choice
  3. 3A corpus of 500 GB of raw text is filtered down to 2 percent. How many GB remain?
    Numeric answer
  4. 4In the data funnel, the mix step decides proportions like how much code, math, and conversation, which shapes what the model is good at.
    True / false

Next: the training objective itself — next-token prediction and the loss that drives everything.