hawkes.aiKey ideas
- The one idea — Schedule at the token step, not the request: finished sequences exit mid-flight, new ones board instantly — no empty seats, ever.
- How it is shown — The perpetual plane: seats refilling the moment they empty, the queue outside draining continuously, throughput multiplying.
- The trap to avoid — Ragged mid-flight turnover shreds tidy memory layouts — the fix creates a new problem (seed for EP 112).
- What it sets up — The memory mess the perpetual plane creates.
Old AI servers had a dumb rule: the whole batch waits until the slowest request finishes. Short requests, trapped behind essays. The fix changed the economics of every AI service.
The one idea
Schedule at the token step, not the request: finished sequences exit mid-flight, new ones board instantly — no empty seats, ever.
The plane takes off full, stays full, and never lands — because passengers board mid-air. Continuous batching: the scheduling change inside every modern AI service, and a quiet reason your tokens cost what they cost. The perpetual plane's trick: The insight is about rhythm. Generation — episode fifty-six — is a drumbeat: one pass through the tower per beat, each passenger advancing one token. And the thing nobody exploited for years: the manifest can change between beats. Nothing in the math requires the same passengers twice. Stop scheduling flights. Schedule beats. The mechanics: each beat, whoever struck their goodbye token — episode fifty-eight — steps off, seat released. The same beat, queued requests board: prompt read, first token joins the formation. The batch stops being a cohort and becomes a population — churning, breathing, never draining.
How it works — the demo
The perpetual plane: seats refilling the moment they empty, the queue outside draining continuously, throughput multiplying.
Your two-word answer leaves when done. The essay drones on, holding no one hostage. What it bought: the trapped-passenger problem, gone. The empty-seat waste, gone. The same hardware moving multiples of the traffic — from scheduling alone, no new chips, no new math. The idea arrived in twenty-twenty-two — a research system called Orca — and within two years it wasn't a technique. It was simply how serving works. And the feel changed. Static batching made crowds mean waiting for the next flight. Now you board at the next beat — milliseconds — even at rush hour. Neighbors arrive and depart around your streaming reply, invisibly.
The trap to avoid
Ragged mid-flight turnover shreds tidy memory layouts — the fix creates a new problem (seed for EP 112).
Why it matters — and what’s next
The memory mess the perpetual plane creates.
That effortless quality under load? This churn, running perfectly. The trap is really a bill coming due: churn shreds memory. Each passenger's KV cache — episode forty-seven — once sat in tidy contiguous rows for a fixed cohort. With passengers tearing out and wedging in every beat, the hold turns ragged: fragmentation. Solve scheduling, inherit a memory slum — and the slum caps your batch size, which caps everything. The fix came from an unexpected shelf: one of the oldest ideas in computing — the trick operating systems have used since the sixties to manage exactly this disease. Applied to attention, it ended the fragmentation and named a famous serving engine. Next: PagedAttention.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
The plane takes off full, stays full, and never lands — because passengers board mid-air. Continuous batching: the scheduling change inside every modern AI service, and a quiet reason your tokens cost what they cost. The perpetual plane's trick:
The insight is about rhythm. Generation — episode fifty-six — is a drumbeat: one pass through the tower per beat, each passenger advancing one token. And the thing nobody exploited for years: the manifest can change between beats. Nothing in the math requires the same passengers twice. Stop scheduling flights. Schedule beats.
The mechanics: each beat, whoever struck their goodbye token — episode fifty-eight — steps off, seat released. The same beat, queued requests board: prompt read, first token joins the formation. The batch stops being a cohort and becomes a population — churning, breathing, never draining. Your two-word answer leaves when done. The essay drones on, holding no one hostage.
What it bought: the trapped-passenger problem, gone. The empty-seat waste, gone. The same hardware moving multiples of the traffic — from scheduling alone, no new chips, no new math. The idea arrived in twenty-twenty-two — a research system called Orca — and within two years it wasn't a technique. It was simply how serving works.
And the feel changed. Static batching made crowds mean waiting for the next flight. Now you board at the next beat — milliseconds — even at rush hour. Neighbors arrive and depart around your streaming reply, invisibly. That effortless quality under load? This churn, running perfectly.
The trap is really a bill coming due: churn shreds memory. Each passenger's KV cache — episode forty-seven — once sat in tidy contiguous rows for a fixed cohort. With passengers tearing out and wedging in every beat, the hold turns ragged: fragmentation. Solve scheduling, inherit a memory slum — and the slum caps your batch size, which caps everything.
The fix came from an unexpected shelf: one of the oldest ideas in computing — the trick operating systems have used since the sixties to manage exactly this disease. Applied to attention, it ended the fragmentation and named a famous serving engine. Next: PagedAttention.