hawkes.aiKey ideas
- The one idea — Grouped-Query Attention lets groups of query heads share key/value heads — shrinking the KV cache several-fold at near-zero quality cost.
- How it is shown — 32 query heads each with private K/V shelves collapsing to 8 shared shelves; the cache bar dropping 4×; answers unchanged.
- The trap to avoid — Cache size follows KV heads, not total heads — the config file, not the marketing page, tells you.
- What it sets up — Share was generation one — compression is generation two.
In classic attention, thirty-two heads each keep a private copy of everything the conversation remembers. Thirty-two copies. The first fix was almost insultingly simple: make them share.
The one idea
Grouped-Query Attention lets groups of query heads share key/value heads — shrinking the KV cache several-fold at near-zero quality cost.
Act two gave chat its memory: the KV cache, banking every token's keys and values. Then context windows grew a hundredfold, and the savior became the glutton — in classic attention, every head keeps its own copy of everything. Today, reform one, starting with a simple question: why so many copies? Audit the chamber. Multi-head attention — episode thirty-four's lenses — gives every head its own keys and values: thirty-two heads, thirty-two copies of the conversation's memory, at every layer. The lenses need different question-perspectives. Their memories? Near-duplicates, stacked thirty-two high. That multiplication is the whole weight problem. The reform: share. Grouped-Query Attention gathers query heads into groups of four, each group reading one shared key-value shelf.
How it works — the demo
32 query heads each with private K/V shelves collapsing to 8 shared shelves; the cache bar dropping 4×; answers unchanged.
Thirty-two questioners, eight memories: the cache drops four-fold in a clean stroke. The insight is surgical: diversity matters in the questions each head asks; the remembered material can be pooled almost for free. Does sharing hurt? Barely — the finding that mattered. Quality holds near-indistinguishable from full private shelves while the cache drops several-fold. The industry noticed: within two years, essentially every major open family shipped with it. It isn't a trick anymore. It's the default. You collect the dividends whether you knew it or not. The napkin's cache term — the living one everyone forgets — grows several times slower, so long chats fit cards they'd have burst. Serving holds seat more passengers per gigabyte.
The trap to avoid
Cache size follows KV heads, not total heads — the config file, not the marketing page, tells you.
Why it matters — and what’s next
Share was generation one — compression is generation two.
Long-context pricing eases. One architecture line, and every downstream memory bill shrank. The trap for practitioners: cache size follows the KV heads, not the headline head-count. Two "thirty-two-head" models can carry very different appetites depending on their shared-shelf counts — and only the config file says. When sizing long-context work, read that line. The marketing page prices the dream; the config prices the memory. But notice what sharing didn't touch: the shelves still grow with every token — fewer copies, same march. Generation one cut the multiplier. Generation two goes after the rows: what if each remembered token were stored as something radically smaller that unfolds on demand? Next: MLA — the compressed cache.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Act two gave chat its memory: the KV cache, banking every token's keys and values. Then context windows grew a hundredfold, and the savior became the glutton — in classic attention, every head keeps its own copy of everything. Today, reform one, starting with a simple question: why so many copies?
Audit the chamber. Multi-head attention — episode thirty-four's lenses — gives every head its own keys and values: thirty-two heads, thirty-two copies of the conversation's memory, at every layer. The lenses need different question-perspectives. Their memories? Near-duplicates, stacked thirty-two high. That multiplication is the whole weight problem.
The reform: share. Grouped-Query Attention gathers query heads into groups of four, each group reading one shared key-value shelf. Thirty-two questioners, eight memories: the cache drops four-fold in a clean stroke. The insight is surgical: diversity matters in the questions each head asks; the remembered material can be pooled almost for free.
Does sharing hurt? Barely — the finding that mattered. Quality holds near-indistinguishable from full private shelves while the cache drops several-fold. The industry noticed: within two years, essentially every major open family shipped with it. It isn't a trick anymore. It's the default.
You collect the dividends whether you knew it or not. The napkin's cache term — the living one everyone forgets — grows several times slower, so long chats fit cards they'd have burst. Serving holds seat more passengers per gigabyte. Long-context pricing eases. One architecture line, and every downstream memory bill shrank.
The trap for practitioners: cache size follows the KV heads, not the headline head-count. Two "thirty-two-head" models can carry very different appetites depending on their shared-shelf counts — and only the config file says. When sizing long-context work, read that line. The marketing page prices the dream; the config prices the memory.
But notice what sharing didn't touch: the shelves still grow with every token — fewer copies, same march. Generation one cut the multiplier. Generation two goes after the rows: what if each remembered token were stored as something radically smaller that unfolds on demand? Next: MLA — the compressed cache.