hawkes.aiKey ideas
- The one idea — Generating a token means hauling every active weight from memory to cores — so bandwidth, not compute, usually sets chat speed.
- How it is shown — The napkin: 14 GB of weights on a 1 TB/s conveyor → ~70 full deliveries → ~70 tokens/sec ceiling, cores be damned.
- The trap to avoid — Upgrading TFLOPS to speed up local chat — same conveyor, same tokens per second.
- What it sets up — Rooms, each bigger and slower.
To generate one word, a GPU hauls every single weight of the model out of memory. Billions of numbers, moved for one token. Then it does it again.
The one idea
Generating a token means hauling every active weight from memory to cores — so bandwidth, not compute, usually sets chat speed.
Here's the number no spec sheet prints: during chat, your GPU's cores can sit mostly idle — not because the work is done, but because the work hasn't arrived. The machine you bought for its math spends its life waiting for deliveries. Why? Because of what episode ninety showed: every token's journey touches essentially every active weight — billions of numbers hauled from warehouse to cores, multiplied once, discarded. Then the next token needs them all hauled again. The multiplying takes almost no time. The hauling is the job. Chat is a freight operation wearing a math costume. And that gives you a superpower: divide. A fourteen-gigabyte model on a one-terabyte-per-second road: the road can deliver the full cargo about seventy times a second — so about seventy tokens per second, ceiling, roughly what you'll see solo.
How it works — the demo
The napkin: 14 GB of weights on a 1 TB/s conveyor → ~70 full deliveries → ~70 tokens/sec ceiling, cores be damned.
Notice what's not in the formula: the cores. For generation, the conveyor is the speedometer. One refinement, because it explains real behavior: chat has two phases. Reading your prompt processes many words at once — big grids, cores blazing, genuinely compute-bound. Then writing the reply goes one token at a time, and the starving begins. That's why long prompts pause before the first word, then the reply streams at a steady, road-limited rhythm. Once you see the freight problem, the industry's fixes decode themselves. Shrink the crates: fewer bytes per weight — that's quantization, later this act. Share the delivery: make one warehouse-haul serve many requests at once — that's batching, the soul of the serving story later this act. Widen the road: each chip generation's bandwidth leap.
The trap to avoid
Upgrading TFLOPS to speed up local chat — same conveyor, same tokens per second.
Why it matters — and what’s next
Rooms, each bigger and slower.
Every fix is a freight fix. So the trap, and it costs people real money: upgrading compute to speed up local chat. Double the TFLOPS on the same road and your tokens per second barely move — you've hired more geniuses to stand in the same breadline. For running models, shop the road and the warehouse. The arithmetic was never the queue. And the conveyor is just one hallway. Behind the warehouse there are more rooms — bigger, slower, farther — and above it, smaller, faster, closer ones, all the way down to the cores' own hands. Six rooms, and every performance mystery in AI lives in the commute between them. Next: the memory hierarchy.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
Here's the number no spec sheet prints: during chat, your GPU's cores can sit mostly idle — not because the work is done, but because the work hasn't arrived. The machine you bought for its math spends its life waiting for deliveries.
Why? Because of what episode ninety showed: every token's journey touches essentially every active weight — billions of numbers hauled from warehouse to cores, multiplied once, discarded. Then the next token needs them all hauled again. The multiplying takes almost no time. The hauling is the job. Chat is a freight operation wearing a math costume.
And that gives you a superpower: divide. A fourteen-gigabyte model on a one-terabyte-per-second road: the road can deliver the full cargo about seventy times a second — so about seventy tokens per second, ceiling, roughly what you'll see solo. Notice what's not in the formula: the cores. For generation, the conveyor is the speedometer.
One refinement, because it explains real behavior: chat has two phases. Reading your prompt processes many words at once — big grids, cores blazing, genuinely compute-bound. Then writing the reply goes one token at a time, and the starving begins. That's why long prompts pause before the first word, then the reply streams at a steady, road-limited rhythm.
Once you see the freight problem, the industry's fixes decode themselves. Shrink the crates: fewer bytes per weight — that's quantization, later this act. Share the delivery: make one warehouse-haul serve many requests at once — that's batching, the soul of the serving story later this act. Widen the road: each chip generation's bandwidth leap. Every fix is a freight fix.
So the trap, and it costs people real money: upgrading compute to speed up local chat. Double the TFLOPS on the same road and your tokens per second barely move — you've hired more geniuses to stand in the same breadline. For running models, shop the road and the warehouse. The arithmetic was never the queue.
And the conveyor is just one hallway. Behind the warehouse there are more rooms — bigger, slower, farther — and above it, smaller, faster, closer ones, all the way down to the cores' own hands. Six rooms, and every performance mystery in AI lives in the commute between them. Next: the memory hierarchy.