Large Language Models
How large language models actually work — starting from raw text, through training and post-training, out to the frontier, and finally inside the network's internals. Every lesson leads with intuition, then a visualization you can play with, then the underlying math.
Foundations
The building blocks every later chapter assumes you already have.
- Tokens and embeddingsHow raw text becomes numbers a model can actually compute with. 6 min · intro
- The attention mechanismHow a token decides which other tokens in the sequence actually matter to it. 10 min · core
- The transformer blockHow attention and feed-forward layers stack into the architecture behind every modern LLM. 12 min · core
- Mixture of ExpertsWhy frontier models pay for parameters only when they're used — routing, top-k experts, and the load-balancing game. 9 min · core
- Positions and normalizationHow the transformer sees word order — and the plumbing that keeps a deep stack trainable. 9 min · core
- MultimodalityPatches, spectrograms, projectors — how one transformer reads images and audio by turning them into tokens. 10 min · core
- Generating textHow a model turns next-token probabilities into words — and where temperature and top-p come from. 9 min · core
Training
How the stack from Foundations actually gets built: the data funnel, the objective and its gradients (backpropagation), compute-optimal scale, and the engineering that makes it affordable.
- Data and tokenizationHow a tokenizer vocabulary is learned, and how trillions of tokens get selected before training starts. 10 min · core
- The training objectiveNext-token prediction as a differentiable game: cross-entropy loss, gradients, and the learning-rate dance. 10 min · core
- BackpropagationHow the gradient actually gets computed — the chain rule walked backward through the computational graph. 10 min · core
- Scale and scaling lawsHow loss falls as a power law in parameters, data, and compute — and how labs plan trainings with that math. 11 min · advanced
- Making it efficientThe engineering tricks that let a 100B-parameter model train on thousands of GPUs and serve millions of users. 9 min · advanced
Post-training
From text-completer to assistant: instruction tuning, human preferences, and honest evaluation.
- Instruction tuningHow a text-completer becomes an assistant — supervised fine-tuning on demonstrations, and why it's not enough. 9 min · core
- Learning from human preferencesRLHF end to end: preference pairs → reward model → policy gradient with a KL leash — and where it breaks. 12 min · advanced
- Direct preference optimizationDPO: the same preference signal as RLHF, without a reward model or RL loop — and the trade it makes. 9 min · advanced
- Evaluating modelsWhy benchmark numbers lie less than you fear but more than you hope — contamination, benchmarks vs. reality, and human eval. 10 min · core
Frontier
Using the trained assistant: prompting, reasoning, retrieval, agents, adaptation — and where it all still breaks.
- Prompting and reasoningHow to steer a trained model — and why asking it to think out loud actually changes what it can do. 9 min · intro
- Reasoning modelsFrom asking a model to think out loud to training it to — RL on verifiable rewards, test-time compute, and what a 'thinking' budget buys. 12 min · advanced
- Retrieval-augmented generationGiving the model knowledge it doesn't have — and why citing sources changes what 'trustworthy' means. 10 min · core
- Agents and toolsFrom answering to acting: the ReAct loop, tool choice, and everything that breaks when a model takes actions. 10 min · core
- Efficient adaptationLoRA: reshaping a 100B model without retraining it — the math of low-rank updates and when full fine-tuning still wins. 9 min · advanced
- Limits, safety, and what's nextWhat LLMs still get wrong, how safety engineering works in practice, and the frontier directions worth tracking. 11 min · core
Interpretability
Opening the box: what the model actually learned internally, how superposition hides it, and how far we can read — and steer — a network's mind.
- What models learn insideThe outputs are the easy part — how probing reads a network's hidden states, and why 'one neuron, one meaning' is mostly false. 9 min · core
- Superposition and sparse autoencodersNetworks store more features than they have dimensions — the geometry that makes neurons polysemantic, and how SAEs split them apart. 11 min · advanced
- Steering and applicationsFrom reading to controlling: activation steering, activation patching, model edits — and the safety work this enables. 10 min · advanced