The training objective

Everything expensive about LLMs — GPUs for weeks, trillions of tokens — boils down to one repeated move: guess the next token, measure how wrong you were, nudge the weights to be less wrong. This lesson makes that move precise.

The loss #

For one position, the model produced probability $p_i$ for the true next token $i$. The loss is:

$$\mathcal{L} = -\log p_{\text{true token}}$$

The negative log has exactly the shape you want: $p = 1$ → loss 0; $p = 0.01$ → loss ≈ 4.6; $p \to 0$ → loss explodes. Being confidently wrong costs more than being uncertain, and there is no floor on how bad a miss can be — the model is always pushed toward the truth.

Where the formula comes from #

It's not arbitrary — it's maximum likelihood, walked backward. A language model is a probability model; "training it" means choosing parameters that make the training data as probable as possible. For one next-token prediction, the probability the model assigns to the true token is $p_{\text{true}}$; maximizing that is equivalent to minimizing its negative log:

$$\max_\theta \; p_\theta(\text{true token}) \quad\Longleftrightarrow\quad \min_\theta \; -\log p_\theta(\text{true token})$$

Writing it over the whole vocabulary with a one-hot vector $y$ (all zeros except the true token) gives the textbook form — cross-entropy:

$$\mathcal{L} = -\sum_i y_i \log p_i = -\log p_{\text{true}}$$

The sum collapses to a single term because $y$ is one-hot. (When labels aren't one-hot — label smoothing, soft distillation targets — the sum becomes a weighted average of $\log p_i$, which is where those techniques get their effect.) And it's called cross-"entropy" because it compares the data's true entropy against the entropy of encoding it with the model's distribution — cross-entropy = true entropy + KL divergence, so minimizing cross-entropy minimizes the model's mismatch with the data. With natural logs the unit is nats, which is why perplexity is $e^{\mathcal{L}}$.

Two companion readings worth memorizing:

• Perplexity $= e^{\mathcal{L}}$ (for natural-log loss): "the model is effectively choosing among $N$ equally likely tokens." Loss 2.0 ⇔ perplexity ≈ 7.4. This is the number people quote when comparing pretraining runs.
• The gradient is soft. $-\log p$ has derivative $-1/p$: the worse the model's estimate of the true token, the harder it's pushed. Corrections scale with the miss — no manual per-example reweighting needed.

During pretraining, every position of every sequence is a training example at once: given "The cat sat on the ___", the model is scored on "mat" — and simultaneously on "The" given context, "cat" given "The", and so on. One sequence of 1000 tokens yields ~1000 next-token predictions. This density is why next-token prediction is such a cheap supervision signal: no human labels, the text labels itself.

From loss to updated weights #

The nudging is calculus the model never sees explicitly:

1. Forward pass — predict next tokens for a batch of sequences.
2. Backward pass — compute, for every one of billions of parameters, how much it contributed to the loss (the gradient) — the next lesson opens this up.
3. Update — move each parameter a small step against its gradient: $\theta \leftarrow \theta - \eta \, \nabla_\theta \mathcal{L}$. The step size $\eta$ is the learning rate.

Repeat over billions of positions. The optimizer in practice is AdamW, which also tracks per-parameter momentum — details matter less than the shape of the loop.

The learning-rate dance #

Watch what the step size does to training:

• Too high — each update overshoots; loss bounces or diverges. The classic early-training crash.
• Too low — progress is glacial; you pay full compute for little improvement.
• Good — a smooth power-law decay: huge early gains, then slow, steady refinement. Real runs combine a short warmup (ramp from 0) with a long decay (gradually shrinking steps so the model settles into a minimum instead of bouncing around it).

What next-token prediction does and doesn't buy #

Two facts worth holding onto:

• It buys everything at once. Grammar, facts, style, some reasoning — all compressed into "predict the next token" over trillions of varied tokens. There is no separate grammar module.
• It does not align the model. Next-token prediction produces a text-completer, not an assistant. A base model asked "How do I make bread?" might respond with "…and why is bread important in history?" — the statistically plausible continuation, not the helpful one. Fixing that mismatch is post-training, later in this wiki.

Next: the backward pass itself — how the gradient reaches every one of those billions of parameters in a single walk.