hawkes.aiKey ideas
- The one idea — Attention is orderless — RoPE weaves position into the vectors themselves by rotating them position-dependent angles, making relative distance geometric.
- How it is shown — Identical sentences with shuffled words scoring identically until rotation is applied; clock hands at many speeds encoding position; relative distance emerging from angle differences.
- The trap to avoid — Imagining position as a metadata tag — it's woven into the geometry, which is why it can be stretched (and why stretching frays).
- What it sets up — If position is angles, angles can be RESCALED.
Inside essentially every modern open AI model, word order isn't stored as numbers or tags — it's stored as angles. Literal rotations, like clock hands. Here's why.
The one idea
Attention is orderless — RoPE weaves position into the vectors themselves by rotating them position-dependent angles, making relative distance geometric.
A blindness this series has stepped around for a hundred episodes: attention reads every word at once — orderless, episode thirty-six. Unaided, "dog bites man" and "man bites dog" look identical. Order is meaning. Where does the machine keep it? The answer is a rotation — and it's beautiful. The early fix glued on address tags — a learned "you are position forty-two" added to each token; GPT-2's era ran on it. Workable but rigid: addresses memorized, not understood, useless past the last seen in training. What attention needs isn't addresses. It's relationships: not "where are you" — "how far apart are you two?" Enter RoPE — rotary position embeddings. Rotate each token's vectors by an angle proportional to position: token ten turns a little, one hundred more. The magic: when two rotated vectors meet in the dot product, what survives is the difference of their angles — their distance.
How it works — the demo
Identical sentences with shuffled words scoring identically until rotation is applied; clock hands at many speeds encoding position; relative distance emerging from angle differences.
Absolute positions cancel. Order emerges from pure geometry, nothing glued on. And it's not one clock — it's a chronometer. Different vector-slices rotate at different speeds: fast hands distinguishing neighbors, slow hands placing tokens across documents. Position becomes a chord of frequencies, fine and coarse at once — the flavor of elegance this series keeps finding at the machine's best joints. RoPE won the era — the position system inside essentially every modern open model. Nothing glued on. Relative structure native. No address book to outgrow. One property matters beyond the rest, so hold it: angles are continuous. And continuous things — unlike address tags — can be rescaled.
The trap to avoid
Imagining position as a metadata tag — it's woven into the geometry, which is why it can be stretched (and why stretching frays).
Why it matters — and what’s next
If position is angles, angles can be RESCALED.
The trap is the metadata fantasy — position as a tag riding beside each word, detachable. It's not. Position is woven into the vectors' geometry, inseparable from meaning. The weaving is why it works — and it cuts both ways: woven things stretch, and stretching distorts fabric. Exactly the next episode. The door rotation opened: a model trained on eight thousand positions meets a hundred-twenty-eight-thousand-token document. Address tags would run out. Angles rescale — squeeze more positions into the dials' trained range. It works and ships everywhere. It frays in ways the banners don't mention. Next: advertised versus usable context.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
A blindness this series has stepped around for a hundred episodes: attention reads every word at once — orderless, episode thirty-six. Unaided, "dog bites man" and "man bites dog" look identical. Order is meaning. Where does the machine keep it? The answer is a rotation — and it's beautiful.
The early fix glued on address tags — a learned "you are position forty-two" added to each token; GPT-2's era ran on it. Workable but rigid: addresses memorized, not understood, useless past the last seen in training. What attention needs isn't addresses. It's relationships: not "where are you" — "how far apart are you two?"
Enter RoPE — rotary position embeddings. Rotate each token's vectors by an angle proportional to position: token ten turns a little, one hundred more. The magic: when two rotated vectors meet in the dot product, what survives is the difference of their angles — their distance. Absolute positions cancel. Order emerges from pure geometry, nothing glued on.
And it's not one clock — it's a chronometer. Different vector-slices rotate at different speeds: fast hands distinguishing neighbors, slow hands placing tokens across documents. Position becomes a chord of frequencies, fine and coarse at once — the flavor of elegance this series keeps finding at the machine's best joints.
RoPE won the era — the position system inside essentially every modern open model. Nothing glued on. Relative structure native. No address book to outgrow. One property matters beyond the rest, so hold it: angles are continuous. And continuous things — unlike address tags — can be rescaled.
The trap is the metadata fantasy — position as a tag riding beside each word, detachable. It's not. Position is woven into the vectors' geometry, inseparable from meaning. The weaving is why it works — and it cuts both ways: woven things stretch, and stretching distorts fabric. Exactly the next episode.
The door rotation opened: a model trained on eight thousand positions meets a hundred-twenty-eight-thousand-token document. Address tags would run out. Angles rescale — squeeze more positions into the dials' trained range. It works and ships everywhere. It frays in ways the banners don't mention. Next: advertised versus usable context.