hawkes.aiKey ideas
- The one idea — VRAM needed ≈ params × bytes-per-weight + KV cache + overhead — a 30-second napkin that beats every "will it run?" forum thread.
- How it is shown — The napkin live: 8B and 70B at different quants against a 24 GB card, with the cache term growing as the chat lengthens.
- The trap to avoid — Sizing at zero tokens — fitting at hello isn't fitting at message forty.
- What it sets up — When the napkin says no single card.
Your model fit perfectly until message forty, when the chat itself outgrew the card. There's a term in the math everyone forgets, and it grows as you talk.
The one idea
VRAM needed ≈ params × bytes-per-weight + KV cache + overhead — a 30-second napkin that beats every "will it run?" forum thread.
Here's a superpower disguised as arithmetic: one napkin, three terms, and you'll know whether any model fits any machine — before downloading forty gigabytes to find out. Everything this act built converges here. Let's do the math you'll actually use. Term one: the weights. Parameters times bytes per weight — and you know the dial: sixteen-bit means two bytes; eight-bit about one; four-bit, call it zero-point-six with its bookkeeping. So an eight-billion model at four-bit: around five gigabytes. Seventy billion at four-bit: around forty. That's the resident, permanent mass. Term two: the cache — and it's alive. Episode forty-seven: every token of context banks its keys and values, so this term grows with your conversation, and long contexts on mid-size models claim real gigabytes. Exact size varies by architecture; your runtime will report it.
How it works — the demo
The napkin live: 8B and 70B at different quants against a 24 GB card, with the cache term growing as the chat lengthens.
What matters on the napkin: it's not zero, and it's not fixed. Term three: overhead — activations, buffers, the runtime's furniture. Budget a gigabyte or two and move on. And that's the napkin: weights, plus cache, plus overhead, must fit your VRAM — with headroom, because a room filled to the brim is a room about to spill. Three terms. Thirty seconds. Every "will it run" thread on the internet, retired. Run it against a twenty-four-gig card. Eight-billion at eight-bit: nine-ish for weights — comfortable, cache welcome, long chats fine. Thirty-two-billion at four-bit: around nineteen — it fits, but the cache is now negotiating for the leftovers, so watch your context. Seventy-billion at four-bit: forty for weights alone.
The trap to avoid
Sizing at zero tokens — fitting at hello isn't fitting at message forty.
Why it matters — and what’s next
When the napkin says no single card.
The napkin says no, and the napkin is right. The trap — and it bites everyone once: sizing at zero tokens. The model loads, the demo sings, and three days later a long working session crawls, because the cache term you ignored has eaten the headroom and started the spill. Size for the conversation you'll actually have, not the hello you'll test with. The napkin has three terms; the second one is a verb. And sometimes the napkin's verdict is bigger than your shopping cart: for frontier-scale models, no single card on Earth fits the weights — the arithmetic fails every room in the building. What then? Then you stop trying to fit the model into a machine, and start cutting the model across many. Next: one GPU isn't enough — now what?
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
Here's a superpower disguised as arithmetic: one napkin, three terms, and you'll know whether any model fits any machine — before downloading forty gigabytes to find out. Everything this act built converges here. Let's do the math you'll actually use.
Term one: the weights. Parameters times bytes per weight — and you know the dial: sixteen-bit means two bytes; eight-bit about one; four-bit, call it zero-point-six with its bookkeeping. So an eight-billion model at four-bit: around five gigabytes. Seventy billion at four-bit: around forty. That's the resident, permanent mass.
Term two: the cache — and it's alive. Episode forty-seven: every token of context banks its keys and values, so this term grows with your conversation, and long contexts on mid-size models claim real gigabytes. Exact size varies by architecture; your runtime will report it. What matters on the napkin: it's not zero, and it's not fixed.
Term three: overhead — activations, buffers, the runtime's furniture. Budget a gigabyte or two and move on. And that's the napkin: weights, plus cache, plus overhead, must fit your VRAM — with headroom, because a room filled to the brim is a room about to spill. Three terms. Thirty seconds. Every "will it run" thread on the internet, retired.
Run it against a twenty-four-gig card. Eight-billion at eight-bit: nine-ish for weights — comfortable, cache welcome, long chats fine. Thirty-two-billion at four-bit: around nineteen — it fits, but the cache is now negotiating for the leftovers, so watch your context. Seventy-billion at four-bit: forty for weights alone. The napkin says no, and the napkin is right.
The trap — and it bites everyone once: sizing at zero tokens. The model loads, the demo sings, and three days later a long working session crawls, because the cache term you ignored has eaten the headroom and started the spill. Size for the conversation you'll actually have, not the hello you'll test with. The napkin has three terms; the second one is a verb.
And sometimes the napkin's verdict is bigger than your shopping cart: for frontier-scale models, no single card on Earth fits the weights — the arithmetic fails every room in the building. What then? Then you stop trying to fit the model into a machine, and start cutting the model across many. Next: one GPU isn't enough — now what?