Wiki
Core10 min read

The training objective

Next-token prediction as a differentiable game: cross-entropy loss, gradients, and the learning-rate dance.

Everything expensive about LLMs — GPUs for weeks, trillions of tokens — boils down to one repeated move: guess the next token, measure how wrong you were, nudge the weights to be less wrong. This lesson makes that move precise.

Start here

Training is next-token prediction played millions of times. The "measure how wrong" is cross-entropy: the negative log of the probability the model assigned to the token that actually came next. Predict confidently and correctly → tiny loss. Miss badly → large loss.

The loss

For one position, the model produced probability pip_i for the true next token ii. The loss is:

L=−log⁡ptrue token\mathcal{L} = -\log p_{\text{true token}}

The negative log has exactly the shape you want: p=1p = 1 → loss 0; p=0.01p = 0.01 → loss ≈ 4.6; p→0p \to 0 → loss explodes. Being confidently wrong costs more than being uncertain, and there is no floor on how bad a miss can be — the model is always pushed toward the truth.

Where the formula comes from

It's not arbitrary — it's maximum likelihood, walked backward. A language model is a probability model; "training it" means choosing parameters that make the training data as probable as possible. For one next-token prediction, the probability the model assigns to the true token is ptruep_{\text{true}}; maximizing that is equivalent to minimizing its negative log:

max⁡θ  pθ(true token)⟺min⁡θ  −log⁡pθ(true token)\max_\theta \; p_\theta(\text{true token}) \quad\Longleftrightarrow\quad \min_\theta \; -\log p_\theta(\text{true token})

Writing it over the whole vocabulary with a one-hot vector yy (all zeros except the true token) gives the textbook form — cross-entropy:

L=−∑iyilog⁡pi=−log⁡ptrue\mathcal{L} = -\sum_i y_i \log p_i = -\log p_{\text{true}}

The sum collapses to a single term because yy is one-hot. (When labels aren't one-hot — label smoothing, soft distillation targets — the sum becomes a weighted average of log⁡pi\log p_i, which is where those techniques get their effect.) And it's called cross-"entropy" because it compares the data's true entropy against the entropy of encoding it with the model's distribution — cross-entropy = true entropy + KL divergence, so minimizing cross-entropy minimizes the model's mismatch with the data. With natural logs the unit is nats, which is why perplexity is eLe^{\mathcal{L}}.

Two companion readings worth memorizing:

  • Perplexity =eL= e^{\mathcal{L}} (for natural-log loss): "the model is effectively choosing among NN equally likely tokens." Loss 2.0 ⇔ perplexity ≈ 7.4. This is the number people quote when comparing pretraining runs.
  • The gradient is soft. With respect to the softmax input, the cross-entropy gradient is p−yp - y (predicted minus true): the worse the model's estimate of the true token, the harder it's pushed. Corrections scale with the miss — no manual per-example reweighting needed.

During pretraining, every position of every sequence is a training example at once: given "The cat sat on the ___", the model is scored on "mat" — and simultaneously on "The" given context, "cat" given "The", and so on. One sequence of 1000 tokens yields ~1000 next-token predictions. This density is why next-token prediction is such a cheap supervision signal: no human labels, the text labels itself.

From loss to updated weights

The nudging is calculus the model never sees explicitly:

  1. Forward pass — predict next tokens for a batch of sequences.
  2. Backward pass — compute, for every one of billions of parameters, how much it contributed to the loss (the gradient) — the next lesson opens this up.
  3. Update — move each parameter a small step against its gradient: θ←θ−η ∇θL\theta \leftarrow \theta - \eta \, \nabla_\theta \mathcal{L}. The step size η\eta is the learning rate.

Repeat over billions of positions. The optimizer in practice is AdamW, which also tracks per-parameter momentum — details matter less than the shape of the loop.

The learning-rate dance

Watch what the step size does to training:

Training loss vs steps — illustrative curve, not real training

02468losstraining steps →

good: smooth power-law decay — most of the progress happens early

  • Too high — each update overshoots; loss bounces or diverges. The classic early-training crash.
  • Too low — progress is glacial; you pay full compute for little improvement.
  • Good — a smooth long decay: huge early gains, then slow, steady refinement. Real runs combine a short warmup (ramp from 0) with a long decay (commonly cosine-shaped, gradually shrinking steps so the model settles into a minimum instead of bouncing around it).

Careful

The loss curve is the single most-watched chart in a pretraining run. It decays smoothly when healthy — if it spikes, the first suspects are the learning rate and a corrupted data batch, in that order.

What next-token prediction does and doesn't buy

Two facts worth holding onto:

  • It buys everything at once. Grammar, facts, style, some reasoning — all compressed into "predict the next token" over trillions of varied tokens. There is no separate grammar module.
  • It does not align the model. Next-token prediction produces a text-completer, not an assistant. A base model asked "How do I make bread?" might respond with "…and why is bread important in history?" — the statistically plausible continuation, not the helpful one. Fixing that mismatch is post-training, later in this wiki.

Check yourself

Eduspheria wiki · Large Language Models, Training

0 / 5 answered

  1. 1For natural-log cross-entropy, what is perplexity in terms of the loss L?
    Multiple choice
  2. 2A model assigns probability 0.01 to the true next token. Using natural log, what is the loss?
    Numeric answer
  3. 3With respect to the softmax input, the cross-entropy gradient is predicted minus true, so bigger misses produce bigger corrections.
    True / false
  4. 4What is the name of the step size eta in the update rule that moves each parameter against its gradient?
    Short answer
  5. 5Why is next-token prediction considered cheap supervision?
    Multiple choice

Next: the backward pass itself — how the gradient reaches every one of those billions of parameters in a single walk.