hawkes.aiKey ideas
- The one idea — Attention's output is a weighted sum of value vectors, added to the token's stream — context physically installed, meaning physically moved.
- How it is shown — The budget executing over the books; "bank's" point sliding to the river delta in the starfield.
- The trap to avoid — Attention doesn't "look" — nothing gazes; vector content gets copied and summed. The spotlight is arithmetic.
- What it sets up — One blend per visit is one question — what if you need several at once?
The most famous mechanism in AI fits in one sentence: content-addressed copying between token streams. Find by meaning, weight by relevance, copy, add. Every diagram is that sentence, drawn.
The one idea
Attention's output is a weighted sum of value vectors, added to the token's stream — context physically installed, meaning physically moved.
The word rewrites itself using a blend of everything it looked at. This is the step the whole mechanism exists for. Watch it land. The arithmetic is almost anticlimactic: every book — every value vector — gets multiplied by its budget share, and they're all summed. Out comes one new vector, the same size as any single book: a weighted average of what the room offered, dominated by what mattered, seasoned by everything else. And where does the blend go? Onto the stream — added, in exactly episode thirty's manner. The original "bank" rides untouched underneath; the river-flavored blend layers on top. Nothing erased, context installed.
How it works — the demo
The budget executing over the books; "bank's" point sliding to the river delta in the starfield.
The conveyor carries both upward. Now look at the map. "Bank's" point — stranded at its fork since the embeddings act — slides into the river delta. In a different sentence, the same starting point slides to the vaults instead. One frozen menu-item, becoming a different meaning in every sentence it visits. The thing fixed embeddings couldn't do is done. You can now say the whole mechanism in one sentence: attention is content-addressed copying between token streams. Find by meaning, weight by relevance, copy by share, add to the stream. Every diagram you'll ever see of it is that sentence, drawn.
The trap to avoid
Attention doesn't "look" — nothing gazes; vector content gets copied and summed. The spotlight is arithmetic.
Why it matters — and what’s next
One blend per visit is one question — what if you need several at once?
The trap, one last time before we scale up: nothing is looking at anything. There's no gaze to follow, no beam swiveling between words. "Attention" is copying and summing — arithmetic with a theatrical name. Keep the theater for intuition; keep the arithmetic for truth. And the ending is a beginning: one visit spends one budget — one question per word per layer. But a word needs its grammar resolved, its pronouns traced, its story followed — simultaneously. One library can't ask them all. Next group: the machine runs many libraries at once. They're called heads.
'Attention' is a theatrical name for arithmetic. Nothing gazes, no beam swivels between words. Vectors get copied and summed — and a word's meaning physically moves.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
The word rewrites itself using a blend of everything it looked at. This is the step the whole mechanism exists for. Watch it land.
The arithmetic is almost anticlimactic: every book — every value vector — gets multiplied by its budget share, and they're all summed. Out comes one new vector, the same size as any single book: a weighted average of what the room offered, dominated by what mattered, seasoned by everything else.
And where does the blend go? Onto the stream — added, in exactly episode thirty's manner. The original "bank" rides untouched underneath; the river-flavored blend layers on top. Nothing erased, context installed. The conveyor carries both upward.
Now look at the map. "Bank's" point — stranded at its fork since the embeddings act — slides into the river delta. In a different sentence, the same starting point slides to the vaults instead. One frozen menu-item, becoming a different meaning in every sentence it visits. The thing fixed embeddings couldn't do is done.
You can now say the whole mechanism in one sentence: attention is content-addressed copying between token streams. Find by meaning, weight by relevance, copy by share, add to the stream. Every diagram you'll ever see of it is that sentence, drawn.
The trap, one last time before we scale up: nothing is looking at anything. There's no gaze to follow, no beam swiveling between words. "Attention" is copying and summing — arithmetic with a theatrical name. Keep the theater for intuition; keep the arithmetic for truth.
And the ending is a beginning: one visit spends one budget — one question per word per layer. But a word needs its grammar resolved, its pronouns traced, its story followed — simultaneously. One library can't ask them all. Next group: the machine runs many libraries at once. They're called heads.