hawkes.aiKey ideas
- The one idea — A word's meaning depends on other words — the machine needs learned, content-based, any-to-any communication between token streams.
- How it is shown — "River bank" vs "money bank": one stranded point, two destinations, resolvable only by reading neighbors.
- The trap to avoid — Learning attention formulas before the problem — QKV feels arbitrary until you know what it's for.
- What it sets up — The requirements checklist.
The word that changes everything might be twenty tokens away, or two. One 'not' flips everything after it. Now design a machine for that. Attention is the answer.
The one idea
A word's meaning depends on other words — the machine needs learned, content-based, any-to-any communication between token streams.
"Bank" means nothing until you see the word next to it. That sentence is why the most famous mechanism in AI exists. Feel the problem first. Last episode left us stranded. Each token's stream rides the machine alone — and its starting point, the embedding, was fixed at the door, one point per menu item, blind to context. So "bank" is trapped mid-fork forever — unless streams can read each other. Look at what language demands. A pronoun points back across a whole sentence — and which word "it" means depends on meaning, not distance. A single "not" flips everything after it. The word that resolves an ambiguity might be twenty tokens away, or two. The connections are long-range, content-dependent, and different for every sentence ever written.
How it works — the demo
"River bank" vs "money bank": one stranded point, two destinations, resolvable only by reading neighbors.
The obvious fixes were tried for years. Pass a running summary down the line, token to token — but a summary squeezed through every step degrades like a rumor, and the line can't be computed in parallel. Or give each word a fixed window of neighbors — cheap, but blind to the word just outside the frame. Both hit walls. Write the wishlist explicitly. Any token must be able to reach any other — no distance limit. Matches must be made by content — "it" finds "trophy" because of what they mean, not where they sit. The matching rules must be learned from data, not written by hand. And it must run in parallel, every token at once, or training at scale dies. Here's the shape of the answer, before the mechanism. Let each word ask.
The trap to avoid
Learning attention formulas before the problem — QKV feels arbitrary until you know what it's for.
Why it matters — and what’s next
The requirements checklist.
Let its stream reach out to every other stream at once, weight each one by relevance — learned relevance — and pull back a blend of what it found. "Bank" reads the room, finds "river" glowing hot, drinks from it, and its point moves to the delta. That behavior is attention. The trap — and it's how attention gets taught almost everywhere: meeting the formula before the problem. Queries, keys, values, dot products, softmax — presented cold, it's arbitrary machinery. Held against the wishlist, every piece has a visible job. You now have the wishlist. You're immune. Next episode, the mechanism itself — and we're doing it as a library. Every word walks in, files a request slip, scans the spines, and walks out changed. Three minutes, and the most famous idea in modern AI is yours.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
"Bank" means nothing until you see the word next to it. That sentence is why the most famous mechanism in AI exists. Feel the problem first.
Last episode left us stranded. Each token's stream rides the machine alone — and its starting point, the embedding, was fixed at the door, one point per menu item, blind to context. So "bank" is trapped mid-fork forever — unless streams can read each other.
Look at what language demands. A pronoun points back across a whole sentence — and which word "it" means depends on meaning, not distance. A single "not" flips everything after it. The word that resolves an ambiguity might be twenty tokens away, or two. The connections are long-range, content-dependent, and different for every sentence ever written.
The obvious fixes were tried for years. Pass a running summary down the line, token to token — but a summary squeezed through every step degrades like a rumor, and the line can't be computed in parallel. Or give each word a fixed window of neighbors — cheap, but blind to the word just outside the frame. Both hit walls.
Write the wishlist explicitly. Any token must be able to reach any other — no distance limit. Matches must be made by content — "it" finds "trophy" because of what they mean, not where they sit. The matching rules must be learned from data, not written by hand. And it must run in parallel, every token at once, or training at scale dies.
Here's the shape of the answer, before the mechanism. Let each word ask. Let its stream reach out to every other stream at once, weight each one by relevance — learned relevance — and pull back a blend of what it found. "Bank" reads the room, finds "river" glowing hot, drinks from it, and its point moves to the delta. That behavior is attention.
The trap — and it's how attention gets taught almost everywhere: meeting the formula before the problem. Queries, keys, values, dot products, softmax — presented cold, it's arbitrary machinery. Held against the wishlist, every piece has a visible job. You now have the wishlist. You're immune.
Next episode, the mechanism itself — and we're doing it as a library. Every word walks in, files a request slip, scans the spines, and walks out changed. Three minutes, and the most famous idea in modern AI is yours.