hawkes.aiKey ideas
- The one idea — Fast-for-one and cheap-for-a-million are opposite optimization targets — one machine can't maximize both at once.
- How it is shown — A private fast lane (batch 1, cores starving, user delighted) vs a full highway (big batches, cores fed, each car slower).
- The trap to avoid — Judging a serving system on one metric — a latency king can be a throughput pauper, and vice versa.
- What it sets up — How the highway actually gets loaded.
Every AI service quietly picks between two masters: you, who wants the answer now — and the million others who want it cheap. One machine can't fully serve both.
The one idea
Fast-for-one and cheap-for-a-million are opposite optimization targets — one machine can't maximize both at once.
Serving one user fast and a million cheap are opposite problems — and every AI system you touch has picked a side or a compromise. One machine, two definitions of fast. The fight between them explains your bill, your wait, and half the industry's engineering. Name the speeds. Latency is one user's clock: time to first token, then how fast their tokens flow. Throughput is the fleet meter: total tokens per second across everyone, indifferent to any individual. Different instruments, different gods — and the machine can't fully worship both. Why do they fight? Episode ninety-two. Alone, the machine hauls every weight across the conveyor for one token — cores starving, single-digit efficiency. Serve a crowd, and the same haul feeds dozens of tokens at once.
How it works — the demo
A private fast lane (batch 1, cores starving, user delighted) vs a full highway (big batches, cores fed, each car slower).
Company makes the machine efficient; solitude wastes it. Your private fast lane is physics-level luxury. So every product picks a point on the dial. Premium tiers buy you toward the private lane. Batch tiers — the discounted overnight kind — live at the packed end, where efficiency peaks. Mainstream chat sits in the tense middle. Cheap and instant together is the combination physics resists — someone always pays for empty road. Now you can read any system by its god. Your local rig is a latency temple: batch of one, waste embraced, instant response. A big API is a throughput temple: your stream shares the road with thousands, tuned for total flow. Neither is wrong — each worships correctly for its congregation.
The trap to avoid
Judging a serving system on one metric — a latency king can be a throughput pauper, and vice versa.
Why it matters — and what’s next
How the highway actually gets loaded.
Judge each by its own scripture. The trap: one-number verdicts. A stopwatch king may be a fleet-meter pauper — magnificent in demos, ruinous at scale. Its mocked rival may move ten times the traffic at a tenth the cost. Always ask both: how fast for one, how cheap for a million. Any benchmark that answers only one is selling something. One mechanism makes the whole throughput temple run: the gate. Requests arrive scattered; the machine holds its breath a beat, gathers a formation, launches it together. You've felt that beat a thousand times without knowing its name. Next: batching — why servers make you wait, and why you should thank them.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Serving one user fast and a million cheap are opposite problems — and every AI system you touch has picked a side or a compromise. One machine, two definitions of fast. The fight between them explains your bill, your wait, and half the industry's engineering.
Name the speeds. Latency is one user's clock: time to first token, then how fast their tokens flow. Throughput is the fleet meter: total tokens per second across everyone, indifferent to any individual. Different instruments, different gods — and the machine can't fully worship both.
Why do they fight? Episode ninety-two. Alone, the machine hauls every weight across the conveyor for one token — cores starving, single-digit efficiency. Serve a crowd, and the same haul feeds dozens of tokens at once. Company makes the machine efficient; solitude wastes it. Your private fast lane is physics-level luxury.
So every product picks a point on the dial. Premium tiers buy you toward the private lane. Batch tiers — the discounted overnight kind — live at the packed end, where efficiency peaks. Mainstream chat sits in the tense middle. Cheap and instant together is the combination physics resists — someone always pays for empty road.
Now you can read any system by its god. Your local rig is a latency temple: batch of one, waste embraced, instant response. A big API is a throughput temple: your stream shares the road with thousands, tuned for total flow. Neither is wrong — each worships correctly for its congregation. Judge each by its own scripture.
The trap: one-number verdicts. A stopwatch king may be a fleet-meter pauper — magnificent in demos, ruinous at scale. Its mocked rival may move ten times the traffic at a tenth the cost. Always ask both: how fast for one, how cheap for a million. Any benchmark that answers only one is selling something.
One mechanism makes the whole throughput temple run: the gate. Requests arrive scattered; the machine holds its breath a beat, gathers a formation, launches it together. You've felt that beat a thousand times without knowing its name. Next: batching — why servers make you wait, and why you should thank them.