hawkes.aiKey ideas
- The one idea — Multi-head Latent Attention stores one small learned compression per token instead of per-head keys/values — an order-of-magnitude cache cut, trained in from the start.
- How it is shown — Full K/V rows collapsing into slim latent capsules; heads unfolding views from the capsule on demand; the cache bar dropping ~10×.
- The trap to avoid — Learned compression isn't a bolt-on — it's an architecture choice made at training time, not a switch you flip on an existing model.
- What it sets up — What if memory didn't grow AT ALL?
Sharing memory between heads cut the cache fourfold. DeepSeek went further: compress each token's memory into one slim learned capsule. Tenfold smaller — the model learns the compression itself.
The one idea
Multi-head Latent Attention stores one small learned compression per token instead of per-head keys/values — an order-of-magnitude cache cut, trained in from the start.
Generation two asks a radical question: why store the memory at all? Store the seed of it. Multi-head Latent Attention keeps one slim learned capsule per token — not every head's keys and values — unfolding what's needed on demand. The cache drops an order of magnitude. The trick: Each arriving token is pressed through a learned compression into one small latent vector — that's what the cache keeps. At attention time, small projections bloom each head's keys and values back out of the capsule. Essence stored once; perspectives unfolded as needed. The banquet was never kept — only the recipe. Recognize the shape: low-rank. Episode eighty-six made this bet for fine-tuning — meaningful change lives in few directions — and MLA makes it for memory: a token's attention-relevant content fits a slim latent, if the compression is learned.
How it works — the demo
Full K/V rows collapsing into slim latent capsules; heads unfolding views from the capsule on demand; the cache bar dropping ~10×.
Trained end-to-end, the wager pays. Low-rank keeps collecting rent across this series. The receipts: the latent cache runs an order of magnitude below classic attention — dramatically below even grouped shelves — quality held, because compression was trained in. And it shipped at scale: DeepSeek's line runs on it; their famously cheap long-context serving owes much to this. Same intelligence, a tenth the memory bill. Why doesn't tenfold compression wreck recall? The model chose what to keep: the down-projection trains under next-word pressure, learning which distinctions earn their bytes — compression as skill, not squeeze. The deep difference from act four's presses: co-designed with the mind it serves. Bolted-on guesses. Trained-in knows.
The trap to avoid
Learned compression isn't a bolt-on — it's an architecture choice made at training time, not a switch you flip on an existing model.
Why it matters — and what’s next
What if memory didn't grow AT ALL?
The trap: mistaking the architecture menu for the settings menu. MLA is chosen at training time — a model's heads must learn from birth to bloom from capsules. You don't flip it on an existing tower; conversions are surgery with retraining, not a flag. Picking a model, you inherit its cache architecture. Choose knowingly. But even capsules march: slimmer rows, same growth — a million tokens, a million entries. Which brings the radical branch, where the question flips: what if memory never grew at all? One fixed-size case, repacked forever, at any context length. Fast, strange, and not quite attention anymore. Next: linear and delta attention.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Generation two asks a radical question: why store the memory at all? Store the seed of it. Multi-head Latent Attention keeps one slim learned capsule per token — not every head's keys and values — unfolding what's needed on demand. The cache drops an order of magnitude. The trick:
Each arriving token is pressed through a learned compression into one small latent vector — that's what the cache keeps. At attention time, small projections bloom each head's keys and values back out of the capsule. Essence stored once; perspectives unfolded as needed. The banquet was never kept — only the recipe.
Recognize the shape: low-rank. Episode eighty-six made this bet for fine-tuning — meaningful change lives in few directions — and MLA makes it for memory: a token's attention-relevant content fits a slim latent, if the compression is learned. Trained end-to-end, the wager pays. Low-rank keeps collecting rent across this series.
The receipts: the latent cache runs an order of magnitude below classic attention — dramatically below even grouped shelves — quality held, because compression was trained in. And it shipped at scale: DeepSeek's line runs on it; their famously cheap long-context serving owes much to this. Same intelligence, a tenth the memory bill.
Why doesn't tenfold compression wreck recall? The model chose what to keep: the down-projection trains under next-word pressure, learning which distinctions earn their bytes — compression as skill, not squeeze. The deep difference from act four's presses: co-designed with the mind it serves. Bolted-on guesses. Trained-in knows.
The trap: mistaking the architecture menu for the settings menu. MLA is chosen at training time — a model's heads must learn from birth to bloom from capsules. You don't flip it on an existing tower; conversions are surgery with retraining, not a flag. Picking a model, you inherit its cache architecture. Choose knowingly.
But even capsules march: slimmer rows, same growth — a million tokens, a million entries. Which brings the radical branch, where the question flips: what if memory never grew at all? One fixed-size case, repacked forever, at any context length. Fast, strange, and not quite attention anymore. Next: linear and delta attention.