Train/test splits
The model scores itself on data it studied. Grading has to come from somewhere else — and setting that up wrong is how careers are lost.
Chapter 1's trap deserves its own lesson, because the fix has teeth: the score you can compute (training loss) is not the score you care about (real-world loss). The whole discipline of practical ML stands on one move — hold some data out and never let the model see it while learning.
The exam metaphor, taken literally
Training data = homework with answers. Test data = the exam you haven't seen. A model graded on its homework will find a way to ace it. The test set is not "more data" — it is your only honest scoreboard, and it is single-use.
100 rows. Three ways to score a model. Two of them are lying.
reported test RMSE: 0.270
honest re-measure: 0.265
Accent dots train; foreground dots test; orange = leaked rows (near-duplicates sitting on both sides of the line). Every leaky and peeking path reports *optimistically* — the gap between reported and honest is the exact size of your self-deception. At this toy scale the gap is small; on real, buggy pipelines it is enough to publish an embarrassing claim.
The two sins the artifact demonstrates
- Optimizing on test. Split once, then try model variant after model variant, keeping the one with the best test score. Each comparison learns from the test set — after a handful, your "test score" is a training score wearing a disguise (and it's always optimistic). Real headlines say "our model gets 94%": ask what data chose the model.
- Leakage. Test information flowing into training — most commonly through preprocessing fit before the split (scaling, imputation, target encoding). The most virulent form: duplicated or near-duplicate rows spanning both sides, so "unseen" data was partially memorized. Look for suspiciously tidy scores before you look for bugs in the model.
The fraction to hold out has no magic constant: 20% is fashion, 30% is fine, single-digit percentages are fine when data is huge (millions of rows don't need giant evaluation samples — they need trustworthy evaluation pipelines).
Why this is the non-negotiable rule
Everything else in this chapter — cross-validation, scaling, regularization — is machinery for making the held-out estimate more accurate or the model less overfit. If the held-out protocol is broken, all those refinements are decorations on a fake number. When a classical model and a deep model disagree, the referee is the same.
Illustrative vs real
100 rows here so you can watch every dollar of the split. Real splits also stratify (keep class proportions equal per side), and time-series data gets stricter still: you may only test on data after the training window, or your model gets to time travel.
Where next: cross-validation — what to do when one split has too much variance to trust.