Eduspheria Wiki
Core10 min read

Data and tokenization

How a tokenizer vocabulary is learned, and how trillions of tokens get selected before training starts.

A model can only learn from the vocabulary it's given. That vocabulary isn't hand-written — it's learned from the data itself by an algorithm called Byte-Pair Encoding (BPE), and the same selection discipline applies to the training text too. This lesson covers both halves of the fuel problem: how text becomes tokens, and which text is worth training on.

Start here

BPE starts with single characters and repeatedly merges the most frequent adjacent pair. Common words fuse into single tokens; rare words stay as pieces. The vocabulary ends up sized to the data.

Watching BPE learn

Step through the actual algorithm on a tiny corpus — every merge is the real BPE rule, just on a small enough corpus to see:

BPE learning a vocabulary from a tiny corpus (low, lower, slowest, slowly, lowest)

merge 0/6
low·
lower·
slowest·
slowly·
lowest·
next: (l, o) appears 5×

Each step merges the most frequent adjacent pair across the whole corpus — the real algorithm, on a one-word-family corpus so every merge is visible. · marks the end-of-word boundary.

Three things to notice. First, "low" fuses into one token quickly — frequency is all BPE cares about. Second, shared endings ("-est", "-ly") emerge as natural sub-tokens, which is how real vocabularies handle morphology. Third, the special end-of-word marker (·) matters: it's why "low" and the "low" inside "lower" can be different tokens — the tokenizer learns word-boundary-sensitive pieces.

From toy to real

Real tokenizers run this exact loop over large samples of the corpus (gigabytes to hundreds of gigabytes — the sample, not the full crawl, is what trains the vocabulary), merging until the vocabulary reaches a target size (commonly 50k–200k tokens). Two consequences worth internalizing:

  • No out-of-vocabulary failures. Worst case, a word falls back to bytes — every possible text is encodable.
  • Non-English and code pay a token tax. Rare languages and code tokenize into many more pieces per meaning, so the same paragraph costs more context (more positions to attend across). Vocabulary choices have real downstream effects.

The data funnel

Pretraining corpora start larger than they end. The funnel:

  1. Gather — web crawls, books, code, papers: petabytes of raw text.
  2. Filter — deduplicate (near-exact and fuzzy), drop spam and boilerplate, score quality, balance domains. Labs commonly keep a small fraction of what they crawl.
  3. Mix — decide proportions: how much code, how much math, how much conversation. The mixture is a major capability decision — it shapes what the final model is good at.
  4. Tokenize — BPE over the mix; everything becomes token IDs.

Note

Frontier labs are publicly cagey about data mixtures because the mixture is capability. Two labs with the same architecture and compute can produce very different models purely from step 3.

Why this comes before training

The tokenizer is frozen before the model trains: the embedding table one row per token, the LM head one column per token. Change the vocabulary later and both must be rebuilt. That's why data and tokenization are decided first — the objective in the next lesson then only ever sees token IDs.

Next: the training objective itself — next-token prediction and the loss that drives everything.