Backpropagation
How the gradient actually gets computed — the chain rule walked backward through the computational graph.
The previous lesson said "compute how much every parameter contributed to the loss" and moved on. This lesson is that step: backpropagation — the chain rule executed in reverse order through the computational graph. It is the single algorithm all deep learning runs on, and its efficiency (not intelligence) is what made the field possible.
Start here
A forward pass is a factory line: inputs flow left to right through stations, each transforming what it receives. Backprop is the audit that runs the line in reverse, asking at every station: "given how badly the product missed, how much was my transformation to blame?" — and the answer for each parameter falls out of one walk.
The computational graph
Training doesn't see "a model" — it sees a graph of simple operations. Here is the whole idea on a toy graph; step through it:
Forward builds the values; backward walks them in reverse, applying the chain rule
x
input
1
× 2
linear
2
× 3
linear
6
y
output
6
(y − t)²
loss
36
press run — forward pass computes every node left to right
Each node stores only its local derivative; the backward pass multiplies the incoming gradient by it and sends the product leftward. A real network does exactly this — with matrix-Jacobian products as the local derivatives and one backward pass per batch, no matter how many billions of parameters share the graph.
The pattern to hold onto: every node knows only its local derivative — how its output changes if its input wiggles (the ×2 node knows its output is 2× its input; that's all it stores). The backward pass sends a gradient right-to-left, and each node multiplies the incoming gradient by its local derivative and passes the product on. The composition of local rules is the chain rule:
In the artifact: , then the ×3 node contributes its 3, the ×2 node its 2 — so , every factor accounted for.
Why it's cheap, and why that's the whole story
The naive alternative — perturb each parameter slightly and re-measure the loss — needs one forward pass per parameter. A 70B model would need 70 billion forward passes per step. Backprop computes every gradient in the graph with:
- one forward pass (store intermediate values),
- one backward pass — roughly 2× the FLOPs of the forward.
That ~3× total is where the famous compute estimate lives: pretraining costs — 2N per token for the forward, 4N for the backward. Backprop is why trillion-parameter training is arithmetic, not fantasy. Reverse-mode automatic differentiation is just this algorithm implemented by your framework: you write the forward code, autodiff derives the backward walk for free.
Note
Same walk, many names: backprop (algorithm), reverse-mode autodiff (its generalization), "the backward pass" (the loop step). Frameworks like PyTorch record the graph as your forward code runs and replay it in reverse — you never write a backward function.
Two honest caveats
- Memory is the tax. Storing intermediates for the reverse walk is what makes training memory-bound — activation checkpointing (recomputing instead of storing) trades the compute-time/memory tradeoff explicitly (the efficiency lesson returns to this).
- It's a local signal. Backprop says which direction each weight should move for this batch — nothing about whether the batch is representative, whether the step is too big, or whether the objective is the one you care about. It's the engine, not the steering.
Next: what happens when you scale this loop up by six-to-seven orders of magnitude — the scaling laws.
This lesson has exercises attached — hand-computing the backward pass on a two-node graph, and predicting what happens to when a layer's weights double — once the exercises layer ships.