hawkes.aiKey ideas
- The one idea — Linear/delta attention replaces the growing cache with a fixed-size state — constant cost at any length, bought with lossy recall; hybrids hedge with a few true-attention layers.
- How it is shown — The suitcase: a fixed case repacked every token vs the endless shelf; blazing speed at long context; a needle-recall test exposing the loss; the hybrid layout as the hedge.
- The trap to avoid — Treating "1M context" on a linear hybrid as equal to true-attention context — know which layers remember exactly.
- What it sets up — Order and position have their own machinery.
Some new models keep a fixed-size memory no matter how long the chat gets — constant speed forever. The price: recall goes lossy. Ask for one buried detail and see.
The one idea
Linear/delta attention replaces the growing cache with a fixed-size state — constant cost at any length, bought with lossy recall; hybrids hedge with a few true-attention layers.
The radical branch of the cache wars carries a suitcase. True attention drags an endless shelf — every token remembered exactly, forever heavier. Linear attention packs one fixed case, repacked at every step: nothing grows, ever, at any length. Blazing speed, constant memory — and a real cost. The trade the newest models are making: Instead of banking every token and scanning all per query — act two's all-pairs ritual — linear attention folds each arriving token into a fixed-size state, a standing summary. Queries consult the summary. Cost per token: constant, at a page or a million tokens. A recurrent heartbeat wearing attention's clothes — history compressed into a living present. Now the honest cost. A fixed case holding an infinite journey must blur: early details overwritten by later foldings — and precise needle-recall, the one account number from four hundred pages back, is exactly where measurements show it failing. Lossless recall was attention's founding superpower.
How it works — the demo
The suitcase: a fixed case repacked every token vs the endless shelf; blazing speed at long context; a needle-recall test exposing the loss; the hybrid layout as the hedge.
The suitcase trades it for speed, deliberately. The question: how much do you actually need? Then, smarter packing. Delta-rule variants replace instead of accumulate: a new association overwrites the outdated one it supersedes, the case curated rather than sedimented. Learned gates decide what fades and what persists. Recall improves measurably. The suitcase acquired a librarian — the lineage the newest hybrids build on. And production found its settlement: hybrids. Most layers carry the suitcase — constant cost, blazing long-context speed — while every few floors, one true attention layer keeps the endless shelf as an exact-recall safety net. That mostly-linear recipe is how the newest million-token models are built — Kimi's delta-attention line among them. Not attention replaced; attention rationed.
The trap to avoid
Treating "1M context" on a linear hybrid as equal to true-attention context — know which layers remember exactly.
Why it matters — and what’s next
Order and position have their own machinery.
The trap: reading every million-token banner as the same promise. Beneath identical numbers can sit different memory physics — a true-attention million versus a mostly-suitcase million with periodic safety nets. For your contracts, your codebases, your needles: the recall guarantees differ. The test is cheap: hide real needles in real haystacks and measure. Banners advertise capacity. Only physics delivers recall. Three generations, one bent bill: share the shelves, compress the rows, cap the case. But every design leaned on something this series has never explained: attention scans all words at once, orderless by nature — so how does the machine know which token came first, which fifth, which four-thousandth? The answer is a rotation, and it's beautiful. Next group: how models remember position.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
The radical branch of the cache wars carries a suitcase. True attention drags an endless shelf — every token remembered exactly, forever heavier. Linear attention packs one fixed case, repacked at every step: nothing grows, ever, at any length. Blazing speed, constant memory — and a real cost. The trade the newest models are making:
Instead of banking every token and scanning all per query — act two's all-pairs ritual — linear attention folds each arriving token into a fixed-size state, a standing summary. Queries consult the summary. Cost per token: constant, at a page or a million tokens. A recurrent heartbeat wearing attention's clothes — history compressed into a living present.
Now the honest cost. A fixed case holding an infinite journey must blur: early details overwritten by later foldings — and precise needle-recall, the one account number from four hundred pages back, is exactly where measurements show it failing. Lossless recall was attention's founding superpower. The suitcase trades it for speed, deliberately. The question: how much do you actually need?
Then, smarter packing. Delta-rule variants replace instead of accumulate: a new association overwrites the outdated one it supersedes, the case curated rather than sedimented. Learned gates decide what fades and what persists. Recall improves measurably. The suitcase acquired a librarian — the lineage the newest hybrids build on.
And production found its settlement: hybrids. Most layers carry the suitcase — constant cost, blazing long-context speed — while every few floors, one true attention layer keeps the endless shelf as an exact-recall safety net. That mostly-linear recipe is how the newest million-token models are built — Kimi's delta-attention line among them. Not attention replaced; attention rationed.
The trap: reading every million-token banner as the same promise. Beneath identical numbers can sit different memory physics — a true-attention million versus a mostly-suitcase million with periodic safety nets. For your contracts, your codebases, your needles: the recall guarantees differ. The test is cheap: hide real needles in real haystacks and measure. Banners advertise capacity. Only physics delivers recall.
Three generations, one bent bill: share the shelves, compress the rows, cap the case. But every design leaned on something this series has never explained: attention scans all words at once, orderless by nature — so how does the machine know which token came first, which fifth, which four-thousandth? The answer is a rotation, and it's beautiful. Next group: how models remember position.