Loss functions
The score you optimize secretly decides the model you get. Choose it like you mean it.
Last lesson ended with a choice hiding in plain sight: squared error. Why squared? Nobody forcing you. The loss function is yours to pick, and picking it is not a technical detail — it is the definition of "wrong", and the model will take you literally.
The one-line version
The loss function is how you write down what you care about. Optimize a careless loss and you get a model that is excellent at the wrong thing.
One dataset, three losses, three honest answers — live
Not hypothetical: below are eight real delivery times and the two classic regression losses applied to them. The marked values are the optimum the loss would converge to. Now let one corrupt reading in, and watch only one of the markers care:
Eight deliveries. Two loss functions. One bad sensor reading.
clean data: markers agree — the two losses disagree only on pain
Why these numbers: the value minimizing the squared error of a dataset is *by definition* its mean; for absolute error it’s the median. Outliers drag means hard and medians barely — that’s the whole trade between the two losses, visible in one strip.
Suppose you predict delivery time given distance, and a predicted value that equals the true value scores zero. How should a 10-minute error hurt?
- Squared error
(pred − true)²— a 10-minute error hurts 100. Squaring makes big errors catastrophic, so the model prioritizes avoiding the worst case. If your data has a 50-minute outlier, squared loss goes and chases it — at the cost of small, everyday errors. - Absolute error
|pred − true|— a 10-minute error hurts 10. Flat response to error size. The model instead plays it safe around the middle of the data: absolute-loss regression (technically, it estimates the median) ignores outliers almost completely. - Cross-entropy (for categories) — the loss works differently:
the model outputs a probability for each class, and the loss is
-log(probability of the true class). Predict "spam" with 0.9 confidence and it was spam → tiny loss − log(0.9) ≈ 0.1. Predict with 0.01 → loss − log(0.01) ≈ 4.6. Confidently wrong hurts logarithmically more the wronger you are — the standard loss for every classifier, including language models: next-token training in the LLM book is cross-entropy over the vocabulary.
Regression vs classification in one line
Predicting a number (price, minutes, temperature) is regression — use squared/absolute on the difference. Predicting a label (spam/not, cat/dog) is classification — use cross-entropy on the probability.
Convergence of the two intuitions
Two failure patterns are worth naming early, because every project meets both eventually:
- Outliers rewrite squared-loss models. One sensor glitch can move your regression visibly. Cure: robust loss or clean the data — but "clean the data" is a human judgment the model cannot make for you.
- Absolute/cross-entropy losses can't tell you "close". A 0.0001 miss vs 0.5 miss can score the same once a threshold is crossed — which is fine for labels and terrible for physical quantities.
The deeper point: loss choice is where human values enter the system. "Reckless" regression (squared loss) outruns "safe" regression (absolute loss) on the same data — you choose which model to ship by choosing what to score. It's the same lesson the LLM book repeats as reward models: training objectives encode preferences, and preferences are always someone's.
The honest caveat
In practice you'll often reach a default (MSE for regression, cross-entropy for classification) because the data decides more than the loss formula does. That's fine — but defaults are choices too. When a model's behavior surprises you, first ask: what loss did it minimize, and was that the thing I actually wanted?
Illustrative vs real
Real practice adds regularization and validation choices on top (chapters 3), and loss functions get modified in interesting ways (huber loss, focal loss) — but every one remains the same idea: one number, one opinion about wrongness.
Where next: gradient-descent replaces sliders with an algorithm that can actually find the minimum of your chosen loss.