What models learn inside
The outputs are the easy part — how probing reads a network's hidden states, and why 'one neuron, one meaning' is mostly false.
Everything in this wiki so far treated the model as text in, text out. This chapter opens the box. The question: the model learned something from trillions of tokens — can we see what it actually is? The first step is reading hidden states, and the tool for reading is the probe.
Start here
At every layer, a token sits as a vector you can intercept. If some piece of information ("is this word plural?") is in that vector, a small classifier should be able to read it out. Train the classifier, measure accuracy — you've just taken an fMRI of a specific thought.
Probing: reading without changing
A linear probe is a trained map from a hidden state to a label:
The recipe: run the model on thousands of sentences, record the residual-stream vector at a chosen position and layer, train the probe on labeled features, and interpret accuracy. Three ground rules make results meaningful:
- Controls, or it's noise. A probe that also succeeds on shuffled data or a randomly-initialized model has learned nothing about the model — it learned the dataset's shortcuts. Probe accuracy is only evidence if it beats a control baseline.
- The probe can't be too strong. A deep nonlinear probe can extract anything from anything; the claim "information is present" is strongest when a linear readout suffices (the standard is selectivity plus simplicity).
- Presence ≠ use. A probe can read information the model itself never consults when producing output. Probing shows what's stored; causal methods (the steering lesson) test what's used.
What probing has found, consistently: syntax encoded early (part-of-speech, dependencies), semantic and entity information in middle layers, task-oriented features late; board-game-trained models build linear internal boards (the Othello-GPT result — the model represented the board state, not just move sequences); and simple "truthfulness directions" that separate the model's considered-true from considered-false claims. The layer-by-layer shape:
Linear probe accuracy vs layer depth
part-of-speech is linearly readable almost immediately
Layer positions of the peaks are the qualitative shape reported by probing studies, not a specific model's chart (real curves are noisier, and only exist where someone trained the probe). Two ground rules apply: accuracy only counts above the control baseline, and presence ≠ use — causal methods (steering) test what the model actually consults.
The feature problem
You can read some things out. But interpretation at scale hits a wall immediately: neurons are not features. Inspect a real unit and it fires on Persian poetry, sycophantic praise, and the letter "s" in English — unrelated things sharing one channel. Try it:
Hidden units firing across one sentence — pick a unit to inspect
Patterns are constructed for the demonstration — in a real model you'd record activations with a hook and find what this unit responds to by probing across thousands of sentences. The polysemantic unit is the rule, not the exception: real units routinely mix unrelated features.
If single units are polysemantic, "read off the neurons" stops working as an explanation strategy — and the natural next question is why a network would store unrelated things together. That's not an accident; it's a deliberate, economical compression — superposition — and it's the next lesson.
Note
Why interpretability matters even if you never touch a model internally: it is the only method that can verify claims like "the model is not using a backdoor feature," "this unlearning actually removed the capability," or "this failure traces to concept X" — behavior testing can't distinguish fixed from merely suppressed.
Next: the geometry that makes neurons polysemantic on purpose.
This lesson has exercises attached — predicting probe-vs-control results for three scenarios — once the exercises layer ships.