hawkes.aiKey ideas
- The one idea — Store the KV cache in fixed-size pages with a lookup table — a 1960s OS idea that ended fragmentation and unlocked bigger batches.
- How it is shown — Ragged contiguous shelves → uniform pages + a per-request page-table; waste collapsing; the freed space becoming batch capacity.
- The trap to avoid — Waiting for "AI breakthroughs" to look like new math — twice now (FlashAttention, PagedAttention) the win was bookkeeping.
- What it sets up — Shared prefix pages.
Early AI servers wasted most of their cache memory — reservations sitting empty, gaps too ragged to reuse. The fix: cut memory into uniform pages, keep a lookup table. That's it.
The one idea
Store the KV cache in fixed-size pages with a lookup table — a 1960s OS idea that ended fragmentation and unlocked bigger batches.
One of modern AI's most important serving breakthroughs is an idea your operating system has used since the sixties. Cut memory into uniform pages. Keep a table of who owns which. That's PagedAttention. What it fixed — and why the fix printed money — is today's story. First, the disease. Each conversation got one contiguous slab, reserved for how long it might get. Reservations sat mostly empty; departed chats left ragged gaps; new requests were refused while free memory sat shredded into slivers. Measured honestly, systems wasted most of their cache memory — full by appearance, hollow in fact. Now the cure. Cut the cache into small fixed-size pages. A conversation's memory becomes a chain of pages living anywhere, stitched by a lookup table.
How it works — the demo
Ragged contiguous shelves → uniform pages + a per-request page-table; waste collapsing; the freed space becoming batch capacity.
Growing? Grab any free page. Leaving? Pages return to the pool instantly, individually reusable. No slabs, no slivers, no worst-case reservations. Waste collapses to nearly nothing. Why did bookkeeping print money? Because memory was the batch ceiling. Every reclaimed gigabyte seats more passengers, and episode one-ten told you what bigger batches do: throughput up, cost down. It shipped in vLLM — the engine it made famous — and paged caches became how the industry serves, period. Paging pays a bonus. Requests often begin identically — same instructions, same preamble.
The trap to avoid
Waiting for "AI breakthroughs" to look like new math — twice now (FlashAttention, PagedAttention) the win was bookkeeping.
Why it matters — and what’s next
Shared prefix pages.
Page chains can simply point at the same physical pages: one copy, hundreds of readers, private pages only where conversations diverge. File it away — it resurfaces in two episodes as why "cached input" costs a fraction of fresh on your bill. The trap — and this act has now taught it twice: expecting breakthroughs to look like new math. FlashAttention reordered a loop. PagedAttention borrowed a sixties ledger. Both moved the industry more than most new architectures. Scouting what's next in AI, watch the systems door as closely as the math door. History keeps walking through it. One serving trick remains, and it's the strangest of the three: hire a second, smaller model. The little one sprints ahead drafting; the giant approves in single glances — provably changing nothing about the answer, dramatically changing the clock. Next: a small model drafts, a big one approves.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
One of modern AI's most important serving breakthroughs is an idea your operating system has used since the sixties. Cut memory into uniform pages. Keep a table of who owns which. That's PagedAttention. What it fixed — and why the fix printed money — is today's story.
First, the disease. Each conversation got one contiguous slab, reserved for how long it might get. Reservations sat mostly empty; departed chats left ragged gaps; new requests were refused while free memory sat shredded into slivers. Measured honestly, systems wasted most of their cache memory — full by appearance, hollow in fact.
Now the cure. Cut the cache into small fixed-size pages. A conversation's memory becomes a chain of pages living anywhere, stitched by a lookup table. Growing? Grab any free page. Leaving? Pages return to the pool instantly, individually reusable. No slabs, no slivers, no worst-case reservations. Waste collapses to nearly nothing.
Why did bookkeeping print money? Because memory was the batch ceiling. Every reclaimed gigabyte seats more passengers, and episode one-ten told you what bigger batches do: throughput up, cost down. It shipped in vLLM — the engine it made famous — and paged caches became how the industry serves, period.
Paging pays a bonus. Requests often begin identically — same instructions, same preamble. Page chains can simply point at the same physical pages: one copy, hundreds of readers, private pages only where conversations diverge. File it away — it resurfaces in two episodes as why "cached input" costs a fraction of fresh on your bill.
The trap — and this act has now taught it twice: expecting breakthroughs to look like new math. FlashAttention reordered a loop. PagedAttention borrowed a sixties ledger. Both moved the industry more than most new architectures. Scouting what's next in AI, watch the systems door as closely as the math door. History keeps walking through it.
One serving trick remains, and it's the strangest of the three: hire a second, smaller model. The little one sprints ahead drafting; the giant approves in single glances — provably changing nothing about the answer, dramatically changing the clock. Next: a small model drafts, a big one approves.