hawkes.aiKey ideas
- The one idea — Reading your prompt and writing the reply are physically different workloads — a parallel compute-bound flood, then a serial memory-bound drip.
- How it is shown — Split screen: the prompt igniting the whole grid at once vs. reply tokens dripping while the entire weight-cube streams past for each one.
- The trap to avoid — Judging "speed" as one number — time-to-first-token and tokens-per-second measure different machines with different bottlenecks.
- What it sets up — Decode's idle math units.
There's a reason output tokens cost more than input tokens on every AI price sheet. Reading your prompt and writing the reply are physically different workloads.
The one idea
Reading your prompt and writing the reply are physically different workloads — a parallel compute-bound flood, then a serial memory-bound drip.
Your prompt and its reply run on two different kinds of physics. Same model, same hardware — two machines. See both, and every latency mystery dissolves. Machine one: prefill. Your whole prompt processes at once — the parallel grid from episode thirty-six — building the cache shelf in bulk. Every math unit on the chip is busy; the hardware runs saturated, the way it was designed to. The limit here is raw arithmetic: prefill is compute-bound. Longer prompt, more math, more time. Machine two: decode. One token at a time — and for each one, essentially the entire model must stream from memory through the processor. Billions of weights, moved, for one word. The math units sit mostly idle, waiting on the stream.
How it works — the demo
Split screen: the prompt igniting the whole grid at once vs. reply tokens dripping while the entire weight-cube streams past for each one.
Decode isn't limited by arithmetic at all — it's limited by memory bandwidth. The chip isn't thinking hard. It's waiting on a conveyor belt. You already feel both machines daily. The pause before the first word: that's prefill — the flood, settling your prompt. The pace of the reply after: that's decode — the drip. Time-to-first-token and tokens-per-second. Two numbers, two bottlenecks, and no single "speed" figure can honestly summarize both. When a benchmark quotes one number, ask which machine it measured. This asymmetry runs the serving economy. It's why input tokens cost less than output tokens on every price sheet — the flood is efficient, the drip is wasteful. It's why servers batch many users together — if the freight train is running anyway, fill it with passengers.
The trap to avoid
Judging "speed" as one number — time-to-first-token and tokens-per-second measure different machines with different bottlenecks.
Why it matters — and what’s next
Decode's idle math units.
And those idle math units during decode? They're an open invitation — a trick called speculative decoding will exploit them brilliantly in the hardware act. The trap: buying or benchmarking on one number. A chip with heroic math throughput can lose a chat workload to a chip with wider memory channels, because chat lives in decode and decode is bandwidth. A summarization pipeline crunching huge documents leans prefill, and compute matters more. Know which machine your workload actually runs, or the spec sheet will happily mislead you. Flood, then drip — and with that, the serving room is complete: the desk, the shelf, the two machines. All that's left of this act is the summit. The place where a final vector becomes an actual word — the scoreboard the whole tower exists to feed. Next group: the head, and the model's raw opinion. Logits, at last.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
Your prompt and its reply run on two different kinds of physics. Same model, same hardware — two machines. See both, and every latency mystery dissolves.
Machine one: prefill. Your whole prompt processes at once — the parallel grid from episode thirty-six — building the cache shelf in bulk. Every math unit on the chip is busy; the hardware runs saturated, the way it was designed to. The limit here is raw arithmetic: prefill is compute-bound. Longer prompt, more math, more time.
Machine two: decode. One token at a time — and for each one, essentially the entire model must stream from memory through the processor. Billions of weights, moved, for one word. The math units sit mostly idle, waiting on the stream. Decode isn't limited by arithmetic at all — it's limited by memory bandwidth. The chip isn't thinking hard. It's waiting on a conveyor belt.
You already feel both machines daily. The pause before the first word: that's prefill — the flood, settling your prompt. The pace of the reply after: that's decode — the drip. Time-to-first-token and tokens-per-second. Two numbers, two bottlenecks, and no single "speed" figure can honestly summarize both. When a benchmark quotes one number, ask which machine it measured.
This asymmetry runs the serving economy. It's why input tokens cost less than output tokens on every price sheet — the flood is efficient, the drip is wasteful. It's why servers batch many users together — if the freight train is running anyway, fill it with passengers. And those idle math units during decode? They're an open invitation — a trick called speculative decoding will exploit them brilliantly in the hardware act.
The trap: buying or benchmarking on one number. A chip with heroic math throughput can lose a chat workload to a chip with wider memory channels, because chat lives in decode and decode is bandwidth. A summarization pipeline crunching huge documents leans prefill, and compute matters more. Know which machine your workload actually runs, or the spec sheet will happily mislead you.
Flood, then drip — and with that, the serving room is complete: the desk, the shelf, the two machines. All that's left of this act is the summit. The place where a final vector becomes an actual word — the scoreboard the whole tower exists to feed. Next group: the head, and the model's raw opinion. Logits, at last.