hawkes.aiKey ideas
- The one idea — The cache lives in VRAM and grows per token held — long context times many users can dwarf the model itself.
- How it is shown — The real math: ~half a megabyte per token on a 7B-class model; a long chat ballooning to gigabytes beside the weights.
- The trap to avoid — "My GPU fits the weights, so I'm fine" — the cache scales with context × concurrent users, and it's the thing that runs out.
- What it sets up — GQA (fewer KV heads).
On a mid-size model, every single word held in context costs about half a megabyte of GPU memory. Do the math on a long chat — it outweighs the model.
The one idea
The cache lives in VRAM and grows per token held — long context times many users can dwarf the model itself.
Long context isn't straining the machine because it's "thinking harder." It's this: a cache that grows with every token held, pouring into the same scarce memory the model lives in. Time for real numbers. Itemize one slot. For every token, the shelf holds its key and its value — at every layer, for every head, in two-byte numbers. Run the multiplication on a seven-B-class model and one token's slot costs about half a megabyte. Half a megabyte — per word held on the desk. The desk from episode forty-four suddenly has a weight problem. Now scale it. A four-thousand-token conversation: about two gigabytes of cache. A hundred-thousand-token context — the long-window sizes on every modern spec sheet: around fifty gigabytes.
How it works — the demo
The real math: ~half a megabyte per token on a 7B-class model; a long chat ballooning to gigabytes beside the weights.
Bigger than the model itself. At long context, the machine's memory of the conversation physically outweighs the machine. And a serving GPU isn't holding one conversation — it's holding dozens, every one with its own shelf. Cache demand is context length times concurrent users, and that product, not the weights, is usually what caps how many people one GPU can serve. When your request queues at a busy hour, this reservoir is what filled. This reservoir bent the industry's architecture. Sharing shelved labels across many readers — that's GQA and MLA, whole episodes in the architecture act. Compressing slots to lower precision. And paging the cache like an operating system pages RAM — PagedAttention, coming in the serving arc. When you meet those episodes, remember they exist because of this picture.
The trap to avoid
"My GPU fits the weights, so I'm fine" — the cache scales with context × concurrent users, and it's the thing that runs out.
Why it matters — and what’s next
GQA (fewer KV heads).
The trap, and every self-hoster hits it: "my GPU fits the model, I'm fine." The weights are the down payment. The cache is the rent — growing with every token of context, multiplied by every simultaneous user, competing for the same VRAM. Plans that only budget the weights die in production. Episode one-oh-three hands you the full arithmetic to do yourself. One more secret hides in the serving room — you glimpsed it last episode. The machine runs in two rhythms: a flood, then a drip. They are physically different workloads, with different bottlenecks and different prices. Next: prefill versus decode — the two machines inside every reply.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Long context isn't straining the machine because it's "thinking harder." It's this: a cache that grows with every token held, pouring into the same scarce memory the model lives in. Time for real numbers.
Itemize one slot. For every token, the shelf holds its key and its value — at every layer, for every head, in two-byte numbers. Run the multiplication on a seven-B-class model and one token's slot costs about half a megabyte. Half a megabyte — per word held on the desk. The desk from episode forty-four suddenly has a weight problem.
Now scale it. A four-thousand-token conversation: about two gigabytes of cache. A hundred-thousand-token context — the long-window sizes on every modern spec sheet: around fifty gigabytes. Bigger than the model itself. At long context, the machine's memory of the conversation physically outweighs the machine.
And a serving GPU isn't holding one conversation — it's holding dozens, every one with its own shelf. Cache demand is context length times concurrent users, and that product, not the weights, is usually what caps how many people one GPU can serve. When your request queues at a busy hour, this reservoir is what filled.
This reservoir bent the industry's architecture. Sharing shelved labels across many readers — that's GQA and MLA, whole episodes in the architecture act. Compressing slots to lower precision. And paging the cache like an operating system pages RAM — PagedAttention, coming in the serving arc. When you meet those episodes, remember they exist because of this picture.
The trap, and every self-hoster hits it: "my GPU fits the model, I'm fine." The weights are the down payment. The cache is the rent — growing with every token of context, multiplied by every simultaneous user, competing for the same VRAM. Plans that only budget the weights die in production. Episode one-oh-three hands you the full arithmetic to do yourself.
One more secret hides in the serving room — you glimpsed it last episode. The machine runs in two rhythms: a flood, then a drip. They are physically different workloads, with different bottlenecks and different prices. Next: prefill versus decode — the two machines inside every reply.