Making it efficient
The engineering tricks that let a 100B-parameter model train on thousands of GPUs and serve millions of users.
Scaling laws say what to buy; efficiency engineering says how to afford it. None of the ideas here change what the model computes — they change what it costs to compute. Three families: faster attention, smaller numbers, and bigger effective clusters.
Start here
The transformer's compute bill is dominated by two things: attention over long sequences and matrix multiplies over huge weight matrices. Each efficiency trick attacks one of them without changing what the model learns.
FlashAttention: same math, better memory choreography
Standard attention materializes the full score matrix — for a 100k-token context that is numbers, most of the cost being moving them between GPU memory tiers, not computing them. FlashAttention computes the identical attention output by tiling the computation through fast on-chip memory and never materializing the big matrix. It's an IO-scheduling insight, not an approximation:
Exact output, several times faster, and it unlocks much longer contexts — this is one of the quiet reasons context windows grew from 2k to 128k+. Watch the choreography:
One attention head's score matrix — materialize it all, or stream it tile by tile?
naive: materialize 2048×2048
8 MB per head, per layer
FlashAttention: stream through SRAM
2% streamed
The trick: softmax can be computed incrementally — each tile's scores are multiplied, max-tracked, and reduced on-chip, then discarded, so GPU memory holds O(T) not O(T²) (the full 32k matrix here is ~2 GB × heads × layers; nothing like it is ever materialized). Same math, same exact result — the win is not computing less, it is moving numbers between memory tiers less. A tile of T = 2048 here needs ≈128 KB — the real kernel tunes tile shapes exactly like this.
Fewer heads that cover the same ground: MQA/GQA
Each attention head has its own and projections — expensive to store during decoding. Multi-Query Attention shares one head across all query heads; Grouped-Query shares within groups. Quality loss is small; memory savings are large, which is why most modern open models ship with GQA.
Smaller numbers: mixed precision
GPUs compute far faster on fp16/bf16 than fp32, and lower precision means less memory bandwidth. Training runs keep a master fp32 copy of the weights, compute most of the forward/backward in bf16, and accumulate carefully so rounding errors don't accumulate. Inference pushes further: fp8 and int4 quantization shrink serving costs several-fold, usually with small quality loss after careful calibration — and fp8 has recently moved into training at the frontier as well.
Sharding: one model, many GPUs
No single GPU holds a 100B-parameter model in memory, so training splits it:
| Strategy | Split | Cost |
|---|---|---|
| Data parallel | every GPU holds full weights, processes different batches | all-reduce gradients each step |
| Tensor parallel | one layer's matrices split across GPUs | communication inside every forward pass |
| Pipeline parallel | different layers on different GPUs | "bubbles" from sequential stages |
| ZeRO/FSDP | shard weights/optimizer state, gather on demand | heavy network traffic |
| Activation checkpointing | discard stored activations, recompute during backward | ~33% more compute for large memory savings |
Real clusters compose these. Activation checkpointing pairs with the backpropagation lesson's memory tax: backprop needs the forward pass's intermediates, and recomputing them is often cheaper than the memory to keep them. The efficiency frontier is as much systems engineering as modeling.
Careful
These tricks are why "how big is the model" is a worse question than "how was it trained and served". Two models with the same parameter count can differ 10× in real cost depending on precision, attention kernel, and cluster strategy.
Where training ends
You now know how a base model gets built: vocabulary from BPE, the next-token objective, learning-rate choreography, compute-optimal scale, and the engineering that keeps it all affordable. But the result is a text-completer, not an assistant. The next chapter is the bridge from "predicts plausible text" to "helps people" — post-training.