hawkes.aiKey ideas
- The one idea — Relevance is a dot product between query and key — the same instrument as word similarity and the neuron — scaled by √d to keep the race fair.
- How it is shown — One slip swept against all labels, a raw score materializing per pair; the third-appearance reveal of the instrument.
- The trap to avoid — Raw scores aren't attention yet — unbounded, messy, not comparable until normalized (next episode).
- What it sets up — The scores need to become a budget.
One piece of arithmetic measures word similarity, powers every neuron, and now decides which words matter to which. Same instrument, three jobs — the whole machine is built from it.
The one idea
Relevance is a dot product between query and key — the same instrument as word similarity and the neuron — scaled by √d to keep the race fair.
How does the model decide which words matter to this one? Multiplication. The same multiplication you've now met twice. Third time makes the pattern official. Episode twenty-four: the dot product measured whether two words point the same way. Episode twenty-six: it measured whether an input matches a neuron's stencil. Now: it measures whether a request slip matches a spine label. Multiply the slip's numbers against the label's numbers, position by position, add it all up. One number out: the raw match score. It is the same instrument, every time. So one visit produces a row of numbers: this token's slip, scored against every label in the room.
How it works — the demo
One slip swept against all labels, a raw score materializing per pair; the third-appearance reveal of the instrument.
"Bank" versus "river": big. Versus "the": small. Versus a stray comma: nearly nothing. Every pair gets its number, computed identically, no exceptions and no judgment — just arithmetic agreement between patterns. One engineering wrinkle, and it's on every architecture diagram, so let's own it. Longer vectors produce bigger sums automatically — more terms in the addition — which would blow the scores up as models widen. The fix: divide every score by the square root of the vector length. That's the entire meaning of "scaled" in scaled dot-product attention. A safety valve, keeping the race fair at any size. And remember what's being compared: not the words themselves — the slips and labels manufactured by learned lenses. Which means relevance itself is learned.
The trap to avoid
Raw scores aren't attention yet — unbounded, messy, not comparable until normalized (next episode).
Why it matters — and what’s next
The scores need to become a budget.
The same two words can score high in one layer and low in another, because each floor's lenses were tuned, by training, to seek different relationships. Grammar on one floor, story logic on another. Now the trap, which is really a cliffhanger: these raw scores are not yet attention. They're unbounded — they can be huge, or negative. They don't sum to anything sensible. You can't divide a word's reading budget according to a mess. The scores need to become something spendable: a clean set of weights that sum to exactly one. The machine that does that conversion is called softmax, and it has personality: it sharpens winners, it never fully silences anyone, and you'll meet it twice more before the series ends. Next episode: turning scores into focus.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
How does the model decide which words matter to this one? Multiplication. The same multiplication you've now met twice. Third time makes the pattern official.
Episode twenty-four: the dot product measured whether two words point the same way. Episode twenty-six: it measured whether an input matches a neuron's stencil. Now: it measures whether a request slip matches a spine label. Multiply the slip's numbers against the label's numbers, position by position, add it all up. One number out: the raw match score. It is the same instrument, every time.
So one visit produces a row of numbers: this token's slip, scored against every label in the room. "Bank" versus "river": big. Versus "the": small. Versus a stray comma: nearly nothing. Every pair gets its number, computed identically, no exceptions and no judgment — just arithmetic agreement between patterns.
One engineering wrinkle, and it's on every architecture diagram, so let's own it. Longer vectors produce bigger sums automatically — more terms in the addition — which would blow the scores up as models widen. The fix: divide every score by the square root of the vector length. That's the entire meaning of "scaled" in scaled dot-product attention. A safety valve, keeping the race fair at any size.
And remember what's being compared: not the words themselves — the slips and labels manufactured by learned lenses. Which means relevance itself is learned. The same two words can score high in one layer and low in another, because each floor's lenses were tuned, by training, to seek different relationships. Grammar on one floor, story logic on another.
Now the trap, which is really a cliffhanger: these raw scores are not yet attention. They're unbounded — they can be huge, or negative. They don't sum to anything sensible. You can't divide a word's reading budget according to a mess. The scores need to become something spendable: a clean set of weights that sum to exactly one.
The machine that does that conversion is called softmax, and it has personality: it sharpens winners, it never fully silences anyone, and you'll meet it twice more before the series ends. Next episode: turning scores into focus.