Wiki
Core7 min read

Train/test splits

The model scores itself on data it studied. Grading has to come from somewhere else — and setting that up wrong is how careers are lost.

Chapter 1's trap deserves its own lesson, because the fix has teeth: the score you can compute (training loss) is not the score you care about (real-world loss). The whole discipline of practical ML stands on one move — hold some data out and never let the model see it while learning.

The exam metaphor, taken literally

Training data = homework with answers. Test data = the exam you haven't seen. A model graded on its homework will find a way to ace it. The test set is not "more data" — it is your only honest scoreboard, and it is single-use.

100 rows. Three ways to score a model. Two of them are lying.

reported test RMSE: 0.270
honest re-measure: 0.265

Accent dots train; foreground dots test; orange = leaked rows (near-duplicates sitting on both sides of the line). The peeking gap is real, computed arithmetic (best-of-30 noisy looks vs. one honest look). The leak gap is a stand-in: a 2-parameter line is too rigid to actually memorize six duplicate rows, so the effect shown is illustrative of what leakage does to flexible, memorization-prone models (trees, nearest-neighbors, deep nets) — not something this particular toy model can be tricked into doing. Either way, the lesson holds: the gap between reported and honest is the size of your self-deception, and on real, buggy pipelines with real models it is enough to publish an embarrassing claim.

The two sins the artifact demonstrates

  1. Optimizing on test. Split once, then try model variant after model variant, keeping the one with the best test score. Each comparison learns from the test set — after a handful, your "test score" is a training score wearing a disguise (and it's always optimistic). Real headlines say "our model gets 94%": ask what data chose the model.
  2. Leakage. Test information flowing into training — most commonly through preprocessing fit before the split (scaling, imputation, target encoding). The most virulent form: duplicated or near-duplicate rows spanning both sides, so "unseen" data was partially memorized. Look for suspiciously tidy scores before you look for bugs in the model.

The fraction to hold out has no magic constant: 20% is fashion, 30% is fine, single-digit percentages are fine when data is huge (millions of rows don't need giant evaluation samples — they need trustworthy evaluation pipelines).

Why this is the non-negotiable rule

Everything else in this chapter — cross-validation, scaling, regularization — is machinery for making the held-out estimate more accurate or the model less overfit. If the held-out protocol is broken, all those refinements are decorations on a fake number. When a classical model and a deep model disagree, the referee is the same.

Illustrative vs real

100 rows here so you can watch every dollar of the split. Real splits also stratify (keep class proportions equal per side), and time-series data gets stricter still: you may only test on data after the training window, or your model gets to time travel.

Check yourself

Eduspheria wiki · Classical Machine Learning, Learning in practice

0 / 5 answered

  1. 1Why is a held-out test set needed?
    Multiple choice
  2. 2What is the name for test information flowing into training, often through preprocessing fitted before the split?
    Short answer
  3. 3Repeatedly choosing model variants by their test score keeps the test score honest.
    True / false
  4. 4You hold out 20% of 100 rows. How many rows are in the test set?
    Numeric answer
  5. 5Why must time-series data be split by time rather than at random?
    Multiple choice

Where next: cross-validation — what to do when one split has too much variance to trust.