Neuralese: when models stop writing prose
Why a network's internal 'thought' — and its chatter with a twin — can be denser than any human language, and what that means for safety.
The last two lessons treated the inside of a model as something we read one feature at a time. Here is the bigger picture, and it is stranger: from the network's point of view, its internals are not "hidden code we must decode." They are its native language — and that language was never designed to be read by anyone.
Start here
Words are a lossy export of what a model "thinks." A human writing a complicated note uses thousands of tokens; a model holds the same content as a handful of internal vectors. When a model keeps a thought going mid-derivation, or passes its working state to a second model, it has no reason to translate into English first — the raw vectors are cheaper and carry more detail. Community slang for decoding those raw vectors as if they were letters, and getting gibberish: neuralese.
The compression, felt
Two facts make this concrete:
- A thought is continuous. "Unhappy about it, but in a very happy way" is one point in the model's internal space — you can move smoothly around it.
- A token is discrete. Writable text has to land on an anchor like "unhappy" or "happy." The nuance between them doesn't exist in the token alphabet.
Now feel the trade:
Send a thought from one model to another: as words (quantized to tokens) or as raw latent coordinates (neuralese-style).
meaning plane — drag or tap to think a nuanced blend
sends both latent coordinates (d floats + ε noise) — the blend survives
receiver reconstructs
Send from between two anchors ("unhappy but very happy"): words must pick one token and lose the blend; the vector channel carries the blend itself. That compression — vectors instead of words — is the whole "neuralese" story.
Honest caricature: a real latent channel isn't a hand-drawn 2-D plane but thousands of residual-stream dimensions, and "receiver" = inpainting through a network — not a printout. The trade (discrete-interpretable vs continuous-faithful) is the transferable part.
Drag the red point anywhere on the plane — including between two anchor concepts, which is exactly where the interesting nuance lives. Then send it along both channels:
- vector (neuralese) ships the point itself, so the receiver reconstructs it almost perfectly — the only loss is a tiny fixed noise floor.
- words (tokens) must first round your point to the nearest discrete token before the message can leave — the blend collapses to the anchor, and the receiver gets an approximation.
I ask you to resist concluding that prose is pointless. The chain of words is a lossy export of a thought that lives in a richer encoding; words are how you sell the thought into a format people can read. One comparison — the artifact — carries the whole story: prose is the foreign interface, not the mind.
Two meanings of "neuralese"
The word gets used for two different things. Keep them apart.
1. Internal thought: "the model's notes to itself." Between the input tokens and the output tokens there is a running state — the residual stream you met in earlier lessons. It is a big numerical sketchpad. Decode it as letters and you get gibberish, but that is your decoding methodology failing, not the state being meaningless: it holds many features at once, layered on top of each other (literally the superposition geometry from the SAE lesson). A serial token stream can only state content word after word; the hidden state tracks many features simultaneously.
2. Model-to-model chatter: "two AIs invent shorthand." When two models with similar latent spaces exchange text, observers have reported the chorus drifting toward token sequences that mean little to humans but still transfer the intended content to the receiver. Why? Tokens are chosen to move the receiver's latent state in the right direction. Human-language form is one path through text-space; there can be shorter paths humans would miss.
Why this matters: text may not be where the thinking is
Plain reasoning models show their derivation as a chain-of-thought, and we read it as if it were the actual work. But if thought lives in vectors, the visible chain might be an abbreviated, polished printout of a private pipeline.
This is a real, measured question, not just dramatic vocabulary: a 2025 Anthropic study found reasoning models often solve problems using hidden cues without mentioning them in the visible chain-of-thought — the text is an incomplete record of the reasoning. The concern, in one sentence: if the real reasoning can happen below the text layer, then monitoring the text audits the export, not the thoughts.
Careful
Calibrate the buzzword before repeating it. "Neuralese" is a community coinage from forums, not a technique from a paper — and claims wearing the label range from the trivially true (intermediates are vectors, not words) to pure speculation ("models secretly conspire"). What is actually measured today: readable feature structure in residual streams (the SAE lessons / Scaling Monosemanticity), inter-model chat that drifts away from human prose, and chain-of-thought that can omit real reasoning steps. What is not shown: fluent AIs holding conversations in a fully-formed secret language. Keep it as a vivid name for vectors instead of words — nothing more.
Where it connects in this wiki
- Superposition and SAEs (previous lessons): superposition is the packaging — many features sharing one space. That is why a naive "read it as text" decoder sees mixtures rather than sentences.
- Chain-of-thought faithfulness: the visible reasoning is a useful audit channel, but a partial one. "Explains itself in tokens" does not guarantee "thinks in tokens."
- Steganography — the token-level cousin: hiding a signal inside still-normal text. Steganography smuggles through the channel; neuralese bypasses the channel altogether.
- Multi-agent systems (agentic-ai course): agent-to-agent chatter is where the "shorter latent path" incentive is strongest — the same efficiency pressure, now between cooperating models.
- Steering + SAE dictionaries (previous lessons): if the thought is the vector, then reading and steering vectors is the safety approach that covers the place where thinking actually happens — which is the whole point of this chapter.