hawkes.aiKey ideas
- The one idea — Interpretability reads the model's internal state: by probing the residual stream — the vector carrying information through the layers — you find concepts represented as directions, so you can detect what the model is internally "thinking," like whether it treats a statement as true or false.
- How it is shown — A simple probe trained on the activations lighting up for a concept — "this is about France," or "the model represents this claim as false" — read straight from the internal vector.
- The trap to avoid — Assuming the model is a black box we can only judge by its outputs — its internal activations carry readable structure; you can often find a concept as a direction inside.
- What it sets up — If concepts are directions inside, is each neuron one concept? no — it's messier: superposition.
Researchers can read parts of an AI's internal state, including whether it treats a statement as true or false — before it outputs a single word.
The one idea
Interpretability reads the model's internal state: by probing the residual stream — the vector carrying information through the layers — you find concepts represented as directions, so you can detect what the model is internally "thinking," like whether it treats a statement as true or false.
So far we've judged the model from the outside — its words, its confidence. Now something stranger: can we open it up and read what it's actually representing inside? That field is interpretability, and the model is far less of a black box than you'd expect. Recall the residual stream — the running vector of numbers that carries information through the model's layers, read and updated at every step. That stream is the model's working state, its "mind" mid-thought. Interpretability asks: what's written in there, and can we learn to read it? The basic tool is a probe. You take the activations at some layer and train a simple classifier to predict a property — is this text French? Is the sentiment negative? Is this claim true?
How it works — the demo
A simple probe trained on the activations lighting up for a concept — "this is about France," or "the model represents this claim as false" — read straight from the internal vector.
The logic: if a simple probe can read it off the activations, the model must be representing that concept, clearly enough to find. And what they find is striking: concepts live as directions in that space. There's a direction for "French," one for a sentiment, even one that tracks whether the model treats a statement as true. Move along it, and you turn the concept up or down. Meaning has a geometry — and you can both read it and push on it. This connects straight back to hallucination. Sometimes the model internally represents a claim as false — its "truth" direction reads negative — and states it anyway, confidently. The knowledge was in there; the output didn't reflect it. So reading the mind can catch what the words hide: a model that, in some sense, knew better. The trap: treating the model as a pure black box we can only judge by its outputs.
The trap to avoid
Assuming the model is a black box we can only judge by its outputs — its internal activations carry readable structure; you can often find a concept as a direction inside.
Why it matters — and what’s next
If concepts are directions inside, is each neuron one concept? no — it's messier: superposition.
It isn't fully opaque. Its activations carry real, readable structure, and probing recovers a surprising amount. Not perfectly — interpretability is early and hard — but "we can only see the outputs" is already false. So interpretability reads the model's mind: probe the residual stream, and concepts show up as directions you can detect and even steer. But here's the catch that makes it hard. There appear to be far more concepts than neurons to hold them. How does that fit? Next: superposition.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
So far we've judged the model from the outside — its words, its confidence. Now something stranger: can we open it up and read what it's actually representing inside? That field is interpretability, and the model is far less of a black box than you'd expect.
Recall the residual stream — the running vector of numbers that carries information through the model's layers, read and updated at every step. That stream is the model's working state, its "mind" mid-thought. Interpretability asks: what's written in there, and can we learn to read it?
The basic tool is a probe. You take the activations at some layer and train a simple classifier to predict a property — is this text French? Is the sentiment negative? Is this claim true? The logic: if a simple probe can read it off the activations, the model must be representing that concept, clearly enough to find.
And what they find is striking: concepts live as directions in that space. There's a direction for "French," one for a sentiment, even one that tracks whether the model treats a statement as true. Move along it, and you turn the concept up or down. Meaning has a geometry — and you can both read it and push on it.
This connects straight back to hallucination. Sometimes the model internally represents a claim as false — its "truth" direction reads negative — and states it anyway, confidently. The knowledge was in there; the output didn't reflect it. So reading the mind can catch what the words hide: a model that, in some sense, knew better.
The trap: treating the model as a pure black box we can only judge by its outputs. It isn't fully opaque. Its activations carry real, readable structure, and probing recovers a surprising amount. Not perfectly — interpretability is early and hard — but "we can only see the outputs" is already false.
So interpretability reads the model's mind: probe the residual stream, and concepts show up as directions you can detect and even steer. But here's the catch that makes it hard. There appear to be far more concepts than neurons to hold them. How does that fit? Next: superposition.