More ideas than neurons (superposition).

Here's a paradox from inside neural networks: they represent more concepts than they have neurons. The solution is weird — concepts overlap, sharing the same neurons.

The one idea

Models pack far more concepts than they have neurons by storing them in superposition — overlapping directions that share neurons — so a single neuron isn't one concept; it's many. That's why "one neuron = one idea" fails, and why dictionary learning exists to untangle them.

Last episode, concepts were directions you could read inside the model. Now the twist that makes interpretability hard: a model represents far more concepts than it has neurons — thousands of neurons, but tens of thousands of things to represent. How do you fit more ideas than you have parts? Here's the puzzle. A layer might have a few thousand neurons, but it needs to represent tens of thousands of distinct concepts — languages, topics, tones, facts. The numbers don't add up if each neuron holds one idea; you'd run out immediately. So the model doesn't work that way — it packs them in, overlapping. The trick is called superposition. Concepts are stored as directions that share neurons — many features overlaid in the same space, each using a slightly different combination of neurons. It works because any given input activates only a few features at once, so the directions rarely collide.

How it works — the demo

One neuron firing for several unrelated things (a "polysemantic" neuron), and a sparse autoencoder pulling the tangle apart into many clean, single-meaning features.

The model exploits that sparsity to pack more in. This kills a tempting idea: one neuron for "France," one for your grandmother, one clean neuron per concept. There isn't. A single neuron fires for many unrelated things — it's polysemantic. So reading one neuron tells you almost nothing; the concept is a pattern across many neurons, not a single cell. That's why naive neuron-by-neuron interpretability fails. So how do you recover clean concepts? Dictionary learning — sparse autoencoders. You train a tool to re-express the tangled activations as a large dictionary of features, each monosemantic: a single, clear meaning. It pulls the superposition apart into thousands of readable features.

The trap to avoid

Expecting each neuron to mean one thing (the "grandmother neuron") — features are superposed and share neurons; you need dictionary learning to recover clean concepts.

Why it matters — and what’s next

We can now find, read, and disentangle features inside — which sets up steering, and the harder truth about what's structurally fixable.

This is a live frontier of interpretability, and it's working. The trap: expecting one neuron to mean one thing, and reading neurons directly. That fails, because of superposition. The unit of meaning isn't the neuron; it's the feature — a direction across many neurons — and finding those features takes real work. Single neurons look like noise; features make the model legible. So there are more ideas than neurons: features live in superposition, sharing neurons, and dictionary learning untangles them into clean concepts. We can now find, read, and separate what's inside the model. Which sets up the next move — using those features to steer it, and facing what's honestly fixable. Next.

This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.