PCA and dimensionality
You can't see 50-dimensional data. But you can find the two axes that carry most of what's going on — and know what you gave up.
Most datasets carry more columns than meaning. Age, income, years at company, spend, churn risk — often three of those five columns move together, because they measure something underneath: tenure-ishness. Principal component analysis (PCA) finds those hidden axes — directions in which the data is most spread out — and re-expresses everything along them.
What you're doing
Rotate the coordinate system so the data's natural axes align with the plot. The first axis catches as much variance as a single line can; the second, at right angles, catches the most of the remainder — and so on. Keep the top few; the tail is mostly noise you were storing.
Same cloud, two view angles — “good” axes make the data look simple
variance captured by axis 1: 71%
axis 2: 29%
Align the view with data’s spread and the ellipse flattens: the two original columns were one thing (“size-ish”) plus noise. That realignment is PCA — a rotation, nothing more.
The rotated view is what “keeping the top component” means: describe points only along the solid line. You’d compress 2 numbers into 1 and lose almost nothing here — the discarded axis holds the noise wiggle. In 500 dimensions the same economics crush inputs a thousand-fold, but as the readout honestly shows, “high variance” is only sometimes “signal.”
What the math is, in one paragraph
PCA finds the directions of maximal variance. Mechanically: standardize the columns (chapter 3 again — PCA is a distance/dot product method and raw units dominate it), compute the covariance matrix (how every column co-varies with every other), and take its eigenvectors — the principal axes. Project data onto the top two and plot; the readout above shows the fraction of total variance each of the kept axes carries.
The honest part: what the compression throws away
PCA is a lossy projection, and the retained-variance readout is the price tag. 94% kept sounds great — but the discarded 6% of variance sometimes contains the entire signal you cared about (the class difference can be small in variance but absolute in consequence; think of one column that only differs on customers who churn). Variance is not importance — PCA optimizes spread, not downstream usefulness. And unlike k-means, PCA is unsupervised: it never sees your labels, so it can never be blamed for aligning with them by luck.
Where this idea goes next
Every model in the LLM book runs on exactly this logic: embeddings are dense representations where a few hundred axes encode input; attention and later t-SNE/UMAP projections are either PCA's mathematical siblings or its offspring — "find the geometry in which this data becomes simple." Dimensionality reduction is the quietest, most universal idea in the book: representation is compression.
Illustrative vs real
Two correlated features, one rotation, an on-screen readout. Real PCA handles hundreds of columns at once, and its variants (Kernel PCA, incremental PCA, autoencoders) trade linearity for power.
Where next: anomalies — the tail you can no longer see by eye.