hawkes.aiKey ideas
- The one idea — Multi-head attention runs several independent attention patterns in parallel — separate lenses, separate budgets, separate blends — recombined into one update.
- How it is shown — One layer's visit splitting into eight parallel libraries, each igniting a different pattern over the same sentence.
- The trap to avoid — Heads aren't assigned jobs — specialization emerges; and many heads turn out redundant.
- What it sets up — What do they choose to become?
One question per word was never enough. Grammar, pronouns, the story so far — so the layer splits into heads, each free to ask a different kind of question.
The one idea
Multi-head attention runs several independent attention patterns in parallel — separate lenses, separate budgets, separate blends — recombined into one update.
The machine doesn't look one way. It looks eight ways at once — sometimes thirty-two, sometimes a hundred. The final flourish of the attention arc: heads. Why isn't one library enough? Because one visit spends one budget on one question — and a word arrives needing several answered at once. Which noun does this adjective touch? Where does this pronoun point? What's the story so far? Forcing one budget to serve all of them buys mush. So the layer splits the work.
How it works — the demo
One layer's visit splitting into eight parallel libraries, each igniting a different pattern over the same sentence.
Eight heads each press the full stream through their own slimmer lenses — each with its own scores, its own budget, its own blend. Eight complete attention ceremonies, run in parallel, each free to ask a different kind of question. Those parallel units are the heads. Watch one sentence through the stack and the point lands: one head lights a tight diagonal — each word studying its immediate neighbor. Another throws long arcs across the whole context. Another clumps locally around phrases. Eight incompatible readings of the same words, all computed, all kept. Then the reunion: the eight blends are joined side by side and pressed through one last learned lens into a single update — which is added to the stream, exactly as episode thirty promised. From outside, the layer made one contribution. Inside, eight specialists filed their findings.
The trap to avoid
Heads aren't assigned jobs — specialization emerges; and many heads turn out redundant.
Why it matters — and what’s next
What do they choose to become?
The numbers: GPT-2 small ran twelve heads on each of twelve layers — a hundred forty-four independent lookers. Frontier models run dozens of heads across dozens of layers: thousands of small libraries, every floor convening a fresh committee over the day's text. The trap: nobody assigns those committees their jobs. No engineer wrote "head five, track pronouns." The lenses start random, training tunes them, and roles emerge — unevenly: some heads become sharp specialists, many end up redundant, and famously, quite a few can be removed with barely a scratch. Which makes the real question irresistible. When roles do emerge — what do they become? Next episode: researchers crack the rooms open. What they found inside is the best detective story in this act.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
The machine doesn't look one way. It looks eight ways at once — sometimes thirty-two, sometimes a hundred. The final flourish of the attention arc: heads.
Why isn't one library enough? Because one visit spends one budget on one question — and a word arrives needing several answered at once. Which noun does this adjective touch? Where does this pronoun point? What's the story so far? Forcing one budget to serve all of them buys mush.
So the layer splits the work. Eight heads each press the full stream through their own slimmer lenses — each with its own scores, its own budget, its own blend. Eight complete attention ceremonies, run in parallel, each free to ask a different kind of question. Those parallel units are the heads.
Watch one sentence through the stack and the point lands: one head lights a tight diagonal — each word studying its immediate neighbor. Another throws long arcs across the whole context. Another clumps locally around phrases. Eight incompatible readings of the same words, all computed, all kept.
Then the reunion: the eight blends are joined side by side and pressed through one last learned lens into a single update — which is added to the stream, exactly as episode thirty promised. From outside, the layer made one contribution. Inside, eight specialists filed their findings.
The numbers: GPT-2 small ran twelve heads on each of twelve layers — a hundred forty-four independent lookers. Frontier models run dozens of heads across dozens of layers: thousands of small libraries, every floor convening a fresh committee over the day's text.
The trap: nobody assigns those committees their jobs. No engineer wrote "head five, track pronouns." The lenses start random, training tunes them, and roles emerge — unevenly: some heads become sharp specialists, many end up redundant, and famously, quite a few can be removed with barely a scratch. Which makes the real question irresistible. When roles do emerge — what do they become? Next episode: researchers crack the rooms open. What they found inside is the best detective story in this act.