Scaling and features
Before any model sees your data, you decide what its numbers mean. Get that wrong and the model dutifully learns your mistake.
Models compute. Distances, dot products, splits — all arithmetic on whatever numbers your columns carry. Nobody in that arithmetic knows that "height_cm = 170" and "height_m = 1.7" are the same fact declared in different units. Scaling is not a nicety; it is part of defining your data.
Where the units vote
k-NN's distance: with raw columns, age-in-years (0-99) is ~50× louder than height-in-meters (0-2) — the bigger-unit feature becomes the distance. SVMs and PCA behave likewise; tree models don't care (a split threshold just shifts, nothing else changes).
k=3 query: age 45, height 1.88 m — which neighbors win depends entirely on units
k=3 vote → class 0 (height pattern wins)
Truth: the two clouds split by height (accent = tall class), so the query is accent. In raw units the age spread (±40 years) dwarfs the height spread (1.55-1.93 m) — the three nearest are whatever-neighbors-age-45, and the vote flips.
Toggle and watch the three connecting lines teleport. Same data, same k — one answer is simply *wrong*, made from unit choice. Scalers are fit on training rows only; treating the test set’s mean/std as available is leakage. Trees shrug at this: their split thresholds just shift, no distance ever computed.
The two transforms everyone uses
- Standardize (z-score): subtract mean, divide by std → mean 0, std 1. Default for anything that uses distances/gradients (k-NN, SVM, PCA, gradient descent).
- Normalize (min-max): squeeze columns to [0,1]. Fine companion, but one extreme value can compress every other observation.
Both belong to the fit-on-train-only rule of the last chapter: fit the scaler on training rows, apply it to both — fitting on everything is the quietest leakage violation there is (test set info, via mean/std, entered the preprocessing).
Features are choices, not facts
Scaling is the cheapest of a whole class of decisions called feature engineering: which columns to keep, log-transforms for skewed amounts, age-groups vs raw ages, ratios instead of magnitudes. Two honest rules survive:
- Garbage magnitudes beat garbage columns: a useless feature is dead weight; a wrong-scaled useful one is worse than useless — it hijacks the distance.
- In the LLM book, tokens+embeddings are exactly this step — representing raw reality (text) as numbers arranged so that similarity is computable. The representation IS most of the model.
Illustrative vs real
Two features, a k-NN style distance, one toggle. Real pipelines additionally clip outliers (see the outlier tails before scaling), impute missing values, and encode categorical columns — all fit-per-fold, never globally.
Where next: metrics that lie (accuracy on the imbalance problem), then regularization — the deliberate handicap.