hawkes.aiKey ideas
- The one idea — Cracked open, models reveal real specialist heads — previous-token heads, reference-tracking heads, and induction heads that notice "this happened before" and copy what came next.
- How it is shown — The pronoun head drawing its arc from "it" back to the right noun; the induction pattern A-B…A→B completing.
- The trap to avoid — Over-reading head stories — labels are post-hoc human narratives; circuits are messy, distributed, and superposed.
- What it sets up — Induction = the machinery of learning-from-your-prompt.
Why does pasting three examples teach an AI your format? Researchers found the gears: induction heads, which notice 'this happened before' and bet on what followed last time.
The one idea
Cracked open, models reveal real specialist heads — previous-token heads, reference-tracking heads, and induction heads that notice "this happened before" and copy what came next.
Inside the model, there is a head whose emergent specialty is tracking what pronouns refer to. Nobody built it. Researchers found it. Tonight, the detective story. How do you learn a head's job? Two tools. Watch it: run thousands of sentences and record where its budget goes. And knock it out: silence it, see which abilities limp. Pattern plus ablation — the microscope and the scalpel of a young field called interpretability. The gallery of finds is real and replicated. Previous-token heads — budget always on the word just behind.
How it works — the demo
The pronoun head drawing its arc from "it" back to the right noun; the induction pattern A-B…A→B completing.
Anchor heads parked on the sequence's start. Phrase-local heads. And the showpieces: heads whose arcs reliably connect a name to its later mentions — machinery for who's-who, self-organized out of gradient descent. And then there's the star: the induction head. Its move: when the current token appeared before in the context, look up what followed it last time — and bet on that following again. A-then-B, seen once; meet A again, predict B. "This happened before; copy what came next." Because induction is plausibly why prompting works at all. It's why an invented name from paragraph one survives to paragraph nine, and why pasting three examples teaches it your format — the heads catch the rhythm and continue it. These heads emerge in a sudden burst early in training, right as models get dramatically better at learning from context. When a later episode marvels at in-context learning, you've already met its gears. Now the trap, stated with respect: those tidy labels are human stories told after the fact.
The trap to avoid
Over-reading head stories — labels are post-hoc human narratives; circuits are messy, distributed, and superposed.
Why it matters — and what’s next
Induction = the machinery of learning-from-your-prompt.
Real heads moonlight. Functions smear across several heads with no single owner. And most rooms in the building defy naming entirely. The clean specimens are real, and they are the exceptions. Why minds resist labels — superposition — is a whole episode later. And with that, the attention arc is complete: words find each other, read each other, rewrite each other — in parallel, under a mask, eight ways at once. But attention only moves information around. Where does the model keep what it knows? The other half of every floor has been waiting. Next: the MLP — where knowledge lives.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Inside the model, there is a head whose emergent specialty is tracking what pronouns refer to. Nobody built it. Researchers found it. Tonight, the detective story.
How do you learn a head's job? Two tools. Watch it: run thousands of sentences and record where its budget goes. And knock it out: silence it, see which abilities limp. Pattern plus ablation — the microscope and the scalpel of a young field called interpretability.
The gallery of finds is real and replicated. Previous-token heads — budget always on the word just behind. Anchor heads parked on the sequence's start. Phrase-local heads. And the showpieces: heads whose arcs reliably connect a name to its later mentions — machinery for who's-who, self-organized out of gradient descent.
And then there's the star: the induction head. Its move: when the current token appeared before in the context, look up what followed it last time — and bet on that following again. A-then-B, seen once; meet A again, predict B. "This happened before; copy what came next."
Because induction is plausibly why prompting works at all. It's why an invented name from paragraph one survives to paragraph nine, and why pasting three examples teaches it your format — the heads catch the rhythm and continue it. These heads emerge in a sudden burst early in training, right as models get dramatically better at learning from context. When a later episode marvels at in-context learning, you've already met its gears.
Now the trap, stated with respect: those tidy labels are human stories told after the fact. Real heads moonlight. Functions smear across several heads with no single owner. And most rooms in the building defy naming entirely. The clean specimens are real, and they are the exceptions. Why minds resist labels — superposition — is a whole episode later.
And with that, the attention arc is complete: words find each other, read each other, rewrite each other — in parallel, under a mask, eight ways at once. But attention only moves information around. Where does the model keep what it knows? The other half of every floor has been waiting. Next: the MLP — where knowledge lives.