hawkes.aiKey ideas
- The one idea — Weights + KV cache + overhead must fit in VRAM — and "fitting" via offload to slower rooms silently destroys speed.
- How it is shown — A model-mass tried against a VRAM box: fits / fits-with-cache-crowding / spills across the narrow bridge.
- The trap to avoid — "It loaded, so it fits" — offloaded layers commute over the bridge every single token.
- What it sets up — Shrink the bytes per weight.
The model loaded. It's running. And it's generating one word every few seconds — because half of it lives outside the fast memory, commuting in for every single token.
The one idea
Weights + KV cache + overhead must fit in VRAM — and "fitting" via offload to slower rooms silently destroys speed.
Can your machine run that model? Not "how fast" — can it, at all. One number answers: VRAM. If the model can't live in the working room, everything else is spectators. Here's the arithmetic of yes and no. What must fit? Three tenants. The weights — all of them; every token reads the full set. The KV cache — episode forty-seven's conversation memory, growing with every turn. And working scraps: buffers, activations. Weights plus cache plus scraps, all inside one fixed room.
How it works — the demo
A model-mass tried against a VRAM box: fits / fits-with-cache-crowding / spills across the narrow bridge.
That's the lease. Feel the numbers. Eight billion weights at sixteen bits: sixteen gigabytes — technically inside a twenty-four-gig card, but the cache will crowd it fast. The same brain at four bits: five-ish gigabytes, comfortable. A seventy-billion model even at four bits: around forty — no single consumer card opens that door. Same intelligence, wildly different leases — all bytes per weight. The mercy option: offload — exile some layers across the bridge to system memory. The model runs. But every token visits every layer, and the exiled ones commute over a bridge ten times slower or worse. Offload a little, lose a little. Offload a lot, and "it runs" becomes seconds per token.
The trap to avoid
"It loaded, so it fits" — offloaded layers commute over the bridge every single token.
Why it matters — and what’s next
Shrink the bytes per weight.
Hence the industry's obsession. The room can't cheaply grow — memory is the card's expensive part. But bytes per weight is a dial, and turning it shrinks every model at once: sixteen bits to four quarters the lease. The next four episodes are that dial — and how far it turns before the brain notices. The trap: "it loaded, so it fits." Loading is a technicality — the honest questions: where did the layers go, and what happens as the cache grows? A model that fits at hello can be spilling by message forty, speed sagging. Fit is a budget across your whole conversation, not a checkbox at startup. So everything now hangs on the width of one crate. The reframe that opens the door: a weight was never simply a number — it's a number written at some chosen precision — and the choice was always negotiable. How coarse can the handwriting get? Next: FP32 to FP4 — what precision means.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Can your machine run that model? Not "how fast" — can it, at all. One number answers: VRAM. If the model can't live in the working room, everything else is spectators. Here's the arithmetic of yes and no.
What must fit? Three tenants. The weights — all of them; every token reads the full set. The KV cache — episode forty-seven's conversation memory, growing with every turn. And working scraps: buffers, activations. Weights plus cache plus scraps, all inside one fixed room. That's the lease.
Feel the numbers. Eight billion weights at sixteen bits: sixteen gigabytes — technically inside a twenty-four-gig card, but the cache will crowd it fast. The same brain at four bits: five-ish gigabytes, comfortable. A seventy-billion model even at four bits: around forty — no single consumer card opens that door. Same intelligence, wildly different leases — all bytes per weight.
The mercy option: offload — exile some layers across the bridge to system memory. The model runs. But every token visits every layer, and the exiled ones commute over a bridge ten times slower or worse. Offload a little, lose a little. Offload a lot, and "it runs" becomes seconds per token.
Hence the industry's obsession. The room can't cheaply grow — memory is the card's expensive part. But bytes per weight is a dial, and turning it shrinks every model at once: sixteen bits to four quarters the lease. The next four episodes are that dial — and how far it turns before the brain notices.
The trap: "it loaded, so it fits." Loading is a technicality — the honest questions: where did the layers go, and what happens as the cache grows? A model that fits at hello can be spilling by message forty, speed sagging. Fit is a budget across your whole conversation, not a checkbox at startup.
So everything now hangs on the width of one crate. The reframe that opens the door: a weight was never simply a number — it's a number written at some chosen precision — and the choice was always negotiable. How coarse can the handwriting get? Next: FP32 to FP4 — what precision means.