Wiki
Advanced10 min read

Steering and applications

From reading to controlling: activation steering, activation patching, model edits — and the safety work this enables.

Probing reads; the final lesson of this chapter touches. Once a feature exists as a clean direction, three moves open up: adding it to activations (steering), swapping activations between runs (patching), and editing weights (model editing). This is interpretability graduating from microscope to — sometimes — control panel.

Start here

If the model's behavior is mediated by identifiable directions in its activations, then at inference time you can turn a knob on those directions — no training, no prompt engineering: subtract the "sycophancy" direction and watch agreement-bias drop.

Steering: inference-time behavior control

Add a scaled feature vector to the residual stream (every layer, or a chosen layer) during generation:

x′=x+α vx' = x + \alpha \, v

Try the dose–response:

Add α × (formal-register vector) to every residual stream, then read the output

user: Can you fix the bug in this function?

Sure — the bug is on line 3. Replace `i < n` with `i <= n` and it should pass the tests.

α = 0: the model's default register — casual, direct.

One vector, dose-dependent effect, no retraining — the essence of activation steering. Continuations are curated for the demonstration; in a real model the vector comes from a SAE feature or the mean activation difference over a feature-relevant corpus, and dose–response is measured, not guessed.

Where does vv come from? Two standard recipes: a SAE feature's decoder direction (the previous lesson's output), or difference-in-means — record activations on prompts where the feature is present vs absent, subtract the means. Then tune α\alpha: small values shift style, large ones degrade coherence (the artifact's overshoot regime) — steering is dose-dependent, not binary.

The honest limits: effects bleed into adjacent behaviors (one group's "reduced sycophancy" was another group's "blunter, less helpful answers"); the right layer matters as much as the vector; and strong steering can push the model off-distribution, where nothing is predictable.

Patching: causal localization

Activation patching is probing with a causal twist: run the model on input A and input B, then transplant one activation from the A-run into the B-run and see how much the output moves. This is the standard method for where is the answer computed questions, and it produced the most famous result in model editing:

  • ROME (Rank-One Model Editing), Meng et al. (2022) first localized a factual association ("Eiffel Tower is in Paris") via causal tracing — running input A/B patching across layers to find where the fact lives (a single MLP layer, in a mid-range GPT model) — then showed in the same work how to edit the weights at that site with a rank-one update: rewrite "Paris" → "Rome" for the subject's entity vector, and the model's answers change downstream — without touching anything else.

The ROME debates matter as much as the result: later work showed such edits often break other facts sharing the subject (localization yes, surgical editing harder), which is the recurring lesson of this chapter — reading is robust, controlling is fragile.

What it's for

The applications are the reason to fund the science:

  • Auditing safety claims. Is a model's refusal real or a fragile persona? Does the "we removed capability X" claim hold when you search the weights for X's features? Behavior testing can't distinguish removed from suppressed — internals can.
  • Unlearning with verification. Deleting hazardous knowledge (CBW synthesis, etc.) and proving the feature is gone, not just masked by a guardrail prompt.
  • Deception/monitoring. Features that fire on deceptive reasoning-during-honest-appearing-output are candidates for runtime honesty monitors.
  • Debugging capability. "Why does the model fail at decades math?" is answerable: find the century-conversion feature, watch where the chain breaks.

Careful

Interpretability's standing gaps: clean results cluster in toy settings and narrow features; industrial-scale SAEs are expensive and partial; and interventions don't yet compose (steer on two features and interactions surprise you). The field's trajectory is real — treat current capabilities as diagnostic, not yet as a control plane.

Check yourself

Eduspheria wiki · Large Language Models, Interpretability

0 / 4 answered

  1. 1In activation patching, what is transplanted between runs?
    Multiple choice
  2. 2What method obtains a steering vector by subtracting mean activations where a feature is present from where it is absent?
    Short answer
  3. 3Steering is dose-dependent: small alpha shifts style while large alpha can degrade coherence.
    True / false
  4. 4What did ROME use causal tracing to localize?
    Multiple choice

This closes the wiki's interpretability arc — except for one postscript worth its own lesson: what all this internal machinery looks like when we stop trying to read it as prose. The next lesson asks whether a model's real "thought" ever lives in the tokens at all.