Scale and scaling laws

The most consequential empirical discovery of the LLM era is that pretraining loss falls as a smooth power law in three variables: parameters $N$, data $D$, and compute $C \approx 6ND$. Not in jumps, not chaotically — predictably, across six-to-seven orders of magnitude. This lesson is about what that buys and how to spend it.

The law #

In log-log space, the relationship is nearly a straight line:

$$\mathcal{L}(N, D) \approx aN^{-\alpha} + bD^{-\beta} + \mathcal{L}_\infty$$

with exponents in the tens-of-percent range — every 10× of parameters buys a predictable chunk of loss reduction, even when the improvement looks invisible step to step. Two landmark results:

• Kaplan et al. (2020) — loss follows clean power laws in $N$, $D$, and $C$; bigger models are more sample-efficient (they need less data per parameter to reach the same loss).
• Hoffmann et al., "Chinchilla" (2022) — at a fixed compute budget, the best loss comes from scaling parameters and data together, roughly 20 tokens of data per parameter. The field had been over-investing in parameters and under-training them.

Spending a budget wisely #

The Chinchilla insight is easiest to see as a budget problem: every training run spends compute as roughly $C = 6ND$. For one fixed budget, you can buy a big under-fed model or a smaller well-fed one — the loss surface picks the sweet spot:

Move the budget slider and the optimum shifts — but its tokens per parameter stays in a narrow band. That stability is the practical takeaway labs distilled into planning tools (in the real fit, the band centers near 20 tokens per parameter).

Emergence: the discontinuous surprise #

The smooth law hides a stranger phenomenon: some abilities — multi-step arithmetic, following multi-part instructions — appear near nonexistent at smaller scale and jump sharply past a threshold. The practical reading: scaling buys qualitative new behavior, not just quantitative refinement.

But hold the claim at arm's length — the debate it triggered is instructive. Schaeffer et al. (2023) showed many "emergent" abilities measure exactly what you'd predict from a smooth underlying curve viewed through a discontinuous metric: on exact-match scoring (all digits right or zero credit), a model improving steadily per-digit still scores 0% until the last digit comes online — the jump is in the ruler, not necessarily the capability. Switch to a smooth metric (partial credit, per-token likelihood of the right answer) and most "emergences" decompose into gradual gains.

The honest split to carry:

• Per-token loss is smooth in every regime measured directly. Data-constrained regimes (heavy epoch reuse) require amendments the original law didn't anticipate, but the smooth shape holds.
• Task-level metrics can jump — partly real (some abilities do require integrated multi-step competence, and partial-credit progress doesn't help a user who needs the whole chain), partly metric artifact. "Emergence" is partly physics, partly how we graded the exam.

The practical consequence survived the debate either way: small proxy models do not reliably tell you which tasks a big model will do well — they tell you the loss. Planning capability (not just loss) needs task-level evaluation at multiple scales.

Scale's side-effects #

Scaling is not free capability:

• Costs concentrate — a frontier training run is months of an entire GPU cluster; the loss curve must be trusted before launch.
• Inference inherits the size — a bigger model is slower and costlier to serve forever, which is why efficiency tricks (next lesson) and distillation matter commercially.
• Data becomes the binding constraint — Chinchilla-optimal training of a 1T-parameter model would need more tokens than exist in publicly crawlable text. Quality, synthetic data, and reuse (more epochs than Chinchilla assumed) are the modern adaptations.

Next: the tricks that make big models cheap to build and serve — FlashAttention, mixed precision, and friends.