Cross-validation
One split, one glance at luck. K-fold resamples that glance — the standard way to know a score isn't a coin flip.
A held-out test set measures honestly — but once. With a small dataset, 20% of the rows can be lucky or unlucky, and your one test score inherits that luck. Cross-validation (k-fold CV) removes the luck by averaging over multiple splits of the same data.
The recipe
Split data into k equal folds. Train on k−1 folds, evaluate on the remaining one. Rotate which fold is held out, k times. Average the k scores — and look at their spread, not just the mean.
5-fold CV, two candidates — mean isn’t the only number the folds are telling you
A: mean 0.81 · spread 0.04
B: mean 0.79 · spread 0.29
Means are (near) tied; spreads are not. The bars ARE the information folds exist to give you — a mean without a spread is a claim without its error. In production, B’s best case (0.95) and worst case (0.66) are both “the same model,” which is rarely what the person asking for it meant. When means differ by less than a spread, report a tie.
What to actually read from it
- The mean is your honest generalization estimate — better than any single split, biased slightly pessimistic (each model trains on less data than you'd deploy).
- The spread (the bars) is the noise estimate in your scores. Two models whose means differ less than their spreads differ are indistinguishable — picking the "winner" is the peeking sin in a lab coat. Read the spread before you read the mean.
- Fold-by-fold flips — a model that wins fold 1 and loses fold 5 sharply — usually means the model family is fine but the dataset slice structure (clusters, class skew) dominates. Fix the data protocol, not the model.
Cost, and when to skip it
K-fold costs k trainings. On tiny classical data that's minutes; on deep models it's prohibitive, which is why big-data practice collapses back to one large test set — affordable only because there the sample size, not the split, carries the accuracy. Small data with big stakes is exactly CV's territory (medicine, credit, churn).
Illustrative vs real
The artifact re-fits the same tiny model across folds so every number is visible. Real CV adds: stratified folds (class balance per fold), grouped folds (all rows of one patient/city together — another leakage species), and nested CV when model selection itself must be scored.
Where next: scaling and feature hygiene — the part that silently decides k-NN and SVM quality before any fitting starts.