hawkes.aiKey ideas
- The one idea — The feed-forward block stores learned facts and patterns — attention routes information; the MLP is where it's kept.
- How it is shown — The vault as key→value memory: a pattern-detector fires, an association gets written to the stream; a single fact surgically edited.
- The trap to avoid — "Attention is the smart part" — attention moves; the MLP knows. Two-thirds of the weights live here.
- What it sets up — Fact-editing and pattern-lookup.
Researchers have located a specific fact inside a model and edited it in place. Change the right numbers, and it confidently reports the new version — everything else intact.
The one idea
The feed-forward block stores learned facts and patterns — attention routes information; the MLP is where it's kept.
Attention moves information around. It doesn't know anything. This hall — the quiet one next door — is where the knowing lives. Every floor of the machine has two halls. After the library, each token's stream enters the MLP — a plain bank of neurons from episode twenty-nine, arranged in two ranks: expand wide, squash, contract back. No cross-token arcs here. Each token thinks alone, about what it just gathered. Researchers who opened this hall found a beautiful structure: it behaves like a memory. The first rank holds pattern detectors — stencils that fire on things like Eiffel-Tower-ness. Each detector is paired with a payload in the second rank — and when the detector fires, its payload gets written to the stream: Paris. France. Iron.
How it works — the demo
The vault as key→value memory: a pattern-detector fires, an association gets written to the stream; a single fact surgically edited.
Detect a pattern, retrieve its associations, add them to the conveyor. How sure are we it's storage? Sure enough to operate. Researchers have located specific facts in these ranks and edited them in place — rewrite the right payload, and the model now confidently reports the new version, with everything else intact. You can't do surgery on a metaphor. And here's the budget surprise: the vault is where most of the model is. Roughly two-thirds of a standard model's weights live in these MLP ranks; attention's lenses are the smaller share. That multi-gigabyte file from episode three? Most of it is drawers. The trap: attention gets the fame — the papers, the name, the metaphors — so people assume it's the smart part. But attention only routes. The MLP holds the books.
The trap to avoid
"Attention is the smart part" — attention moves; the MLP knows. Two-thirds of the weights live here.
Why it matters — and what’s next
Fact-editing and pattern-lookup.
A librarian with no library is just pointing at things. The two halls are a team: one decides what's relevant, the other knows things about it. Notice the choreography. Alone, "bank" fires generic drawers. But after the library visit — river-cargo installed — it fires the river-bank detectors instead. Gather first, look up second, every floor. Context changes which knowledge gets retrieved. That's the loop that makes the machine feel like it understands. Two halls, one conveyor through both. That's everything a floor contains — and next episode we snap it together into the brick the entire model is built from.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Attention moves information around. It doesn't know anything. This hall — the quiet one next door — is where the knowing lives.
Every floor of the machine has two halls. After the library, each token's stream enters the MLP — a plain bank of neurons from episode twenty-nine, arranged in two ranks: expand wide, squash, contract back. No cross-token arcs here. Each token thinks alone, about what it just gathered.
Researchers who opened this hall found a beautiful structure: it behaves like a memory. The first rank holds pattern detectors — stencils that fire on things like Eiffel-Tower-ness. Each detector is paired with a payload in the second rank — and when the detector fires, its payload gets written to the stream: Paris. France. Iron. Detect a pattern, retrieve its associations, add them to the conveyor.
How sure are we it's storage? Sure enough to operate. Researchers have located specific facts in these ranks and edited them in place — rewrite the right payload, and the model now confidently reports the new version, with everything else intact. You can't do surgery on a metaphor.
And here's the budget surprise: the vault is where most of the model is. Roughly two-thirds of a standard model's weights live in these MLP ranks; attention's lenses are the smaller share. That multi-gigabyte file from episode three? Most of it is drawers.
The trap: attention gets the fame — the papers, the name, the metaphors — so people assume it's the smart part. But attention only routes. The MLP holds the books. A librarian with no library is just pointing at things. The two halls are a team: one decides what's relevant, the other knows things about it.
Notice the choreography. Alone, "bank" fires generic drawers. But after the library visit — river-cargo installed — it fires the river-bank detectors instead. Gather first, look up second, every floor. Context changes which knowledge gets retrieved. That's the loop that makes the machine feel like it understands.
Two halls, one conveyor through both. That's everything a floor contains — and next episode we snap it together into the brick the entire model is built from.