hawkes.aiKey ideas
- The one idea — Past tokens' keys and values never change — compute them once, shelve them, and each new token only pays for itself.
- How it is shown — The naive loop re-reading everything per word vs. the shelf: one new column per token.
- The trap to avoid — "Stateless model" meets stateful serving — the cache is per-conversation state on the server, and breaking it (editing early text) re-bills the whole prefill.
Why doesn't every chat message cost a fortune? One trick: past tokens' work never changes, so the machine saves it — and each new word only pays for itself.
The one idea
Past tokens' keys and values never change — compute them once, shelve them, and each new token only pays for itself.
Watch the naive version of generation. For every new word, it re-reads the entire conversation from scratch. Every word. Without one optimization, chat as you know it would cost a fortune. This episode is that optimization. Find the waste precisely. The new token's request slip — genuinely new, must be computed. But every old token rebuilding its label and its book? Identical to last time. And here's the beautiful reason why: the causal mask. The past cannot see the future — so a token's label and book can never be changed by anything that comes after it. Token forty's key is finished forever. Rebuilding it is paying twice for settled work.
How it works — the demo
The naive loop re-reading everything per word vs. the shelf: one new column per token.
So: shelve them. The first time each token's key and value are computed, store them. That shelf is the KV cache. Each new token computes only its own artifacts, reads everyone else's off the shelf, and adds its key and value to the end. One new column of work per word. The magnitude: a thousand-token desk, five-hundred-word reply — the naive machine re-processes the whole thousand for every word: hundreds of thousands of token-passes per answer. The cached machine pays once per token. That orders-of-magnitude gap is the difference between chat as a product and chat as a demo. A conversation has two phases. Prefill: your whole prompt processes in one parallel flash — episode thirty-six's grid — building the shelf in bulk. That's the pause before the first word appears. Then decode: token by token, each one extending the shelf by a single slot. Flood, then drip — the star of episode forty-eight.
The trap to avoid
"Stateless model" meets stateful serving — the cache is per-conversation state on the server, and breaking it (editing early text) re-bills the whole prefill.
Why it matters — and what’s next
The trap explains real invoices. The shelf is valid only while the text it was built from stays untouched. Edit one word near the top and every slot after it is wrong — those tokens attended to what you changed. The tail re-prefills from the edit point. That's why providers discount repeated identical prefixes — prompt caching. Stable prefixes are money. The model itself is still stateless — frozen weights, no memory, exactly as promised. But the serving system now carries state: one shelf per live conversation, held in hardware, evicted when you go quiet, rebuilt when you return. Your chat has a physical footprint in a rack somewhere. Stateless mind, stateful plumbing — keep both halves. The cache is AI's best bargain: pay once per token. But every slot lives in the scarcest real estate in computing — GPU memory. Next: the bill, with real numbers.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 3:00 of narration
Watch the naive version of generation. For every new word, it re-reads the entire conversation from scratch. Every word. Without one optimization, chat as you know it would cost a fortune. This episode is that optimization.
Find the waste precisely. The new token's request slip — genuinely new, must be computed. But every old token rebuilding its label and its book? Identical to last time. And here's the beautiful reason why: the causal mask. The past cannot see the future — so a token's label and book can never be changed by anything that comes after it. Token forty's key is finished forever. Rebuilding it is paying twice for settled work.
So: shelve them. The first time each token's key and value are computed, store them. That shelf is the KV cache. Each new token computes only its own artifacts, reads everyone else's off the shelf, and adds its key and value to the end. One new column of work per word.
The magnitude: a thousand-token desk, five-hundred-word reply — the naive machine re-processes the whole thousand for every word: hundreds of thousands of token-passes per answer. The cached machine pays once per token. That orders-of-magnitude gap is the difference between chat as a product and chat as a demo.
A conversation has two phases. Prefill: your whole prompt processes in one parallel flash — episode thirty-six's grid — building the shelf in bulk. That's the pause before the first word appears. Then decode: token by token, each one extending the shelf by a single slot. Flood, then drip — the star of episode forty-eight.
The trap explains real invoices. The shelf is valid only while the text it was built from stays untouched. Edit one word near the top and every slot after it is wrong — those tokens attended to what you changed. The tail re-prefills from the edit point. That's why providers discount repeated identical prefixes — prompt caching. Stable prefixes are money.
The model itself is still stateless — frozen weights, no memory, exactly as promised. But the serving system now carries state: one shelf per live conversation, held in hardware, evicted when you go quiet, rebuilt when you return. Your chat has a physical footprint in a rack somewhere. Stateless mind, stateful plumbing — keep both halves.
The cache is AI's best bargain: pay once per token. But every slot lives in the scarcest real estate in computing — GPU memory. Next: the bill, with real numbers.