hawkes.aiKey ideas
- The one idea — Give each GPU different floors of the tower — light chatter that tolerates ordinary links, at the price of idle bubbles.
- How it is shown — Layers 1-20 on card A, 21-40 on card B; the relay; bubbles forming; microbatches filling them.
- The trap to avoid — A pipeline fed one request at a time is mostly idle — pipelines need queues to earn their keep.
- What it sets up — Copies, experts, and sharded bookkeeping.
Split a model across GPUs the obvious way: first half here, second half there. The catch: while one card works, the other one just sits there, waiting.
The one idea
Give each GPU different floors of the tower — light chatter that tolerates ordinary links, at the price of idle bubbles.
The gentler surgery: cut between the floors. Layers one through twenty on this card; twenty-one through forty on that. A token climbs the first card's floors, then rides one slim cable next door to finish. Pipeline parallelism: the assembly line. What it fixes — and what it breaks. First the win: silence. Between floors, only a small activation bundle crosses — the token's working state, tiny beside the weights, which never travel. No stitch per layer, no metronome. So this cut tolerates ordinary links, survives between machines, and powers two-card home rigs. Now the flaw. While the second card climbs, the first waits — then they swap.
How it works — the demo
Layers 1-20 on card A, 21-40 on card B; the relay; bubbles forming; microbatches filling them.
One token in flight idles half your machine; more stages, worse fractions. These holes have an industry name: bubbles. An assembly line with one item on the belt is mostly an expensive queue of silence. The remedy is traffic. Feed the line many small batches, nose to tail: while card two finishes one, card one climbs the next. In steady state every station stays busy, bubbles squeezed to the run's ends — fill and drain. Pipelines earn their keep when a queue keeps them fed. So the kingdoms divide by chatter tolerance. Inside a server: the saw, feeding on fast links. Between servers: the pipeline, sipping through the network. And they compose — a frontier model wears both at once, sawn within each chassis, floored across them.
The trap to avoid
A pipeline fed one request at a time is mostly idle — pipelines need queues to earn their keep.
Why it matters — and what’s next
Copies, experts, and sharded bookkeeping.
Your two-card rig runs the humble end of the same idea. The trap: expecting a pipeline to speed up you. One request leaves the belt nearly empty — solo chat gets one card's pace with extra steps. Pipelines are throughput machines: they shine with queues, batches, traffic. For one user they buy episode one-oh-four's gift: existence — the big model runs at all. Speed needs crowds. Two cuts down; the zoo has more: whole copies training on different data. Towers of specialists where only some floors wake per token. Bookkeeping sharded across fleets. And through it all, AI's least glamorous truth: at frontier scale something is always broken — and the run must not care. Next: the other splits.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
The gentler surgery: cut between the floors. Layers one through twenty on this card; twenty-one through forty on that. A token climbs the first card's floors, then rides one slim cable next door to finish. Pipeline parallelism: the assembly line. What it fixes — and what it breaks.
First the win: silence. Between floors, only a small activation bundle crosses — the token's working state, tiny beside the weights, which never travel. No stitch per layer, no metronome. So this cut tolerates ordinary links, survives between machines, and powers two-card home rigs.
Now the flaw. While the second card climbs, the first waits — then they swap. One token in flight idles half your machine; more stages, worse fractions. These holes have an industry name: bubbles. An assembly line with one item on the belt is mostly an expensive queue of silence.
The remedy is traffic. Feed the line many small batches, nose to tail: while card two finishes one, card one climbs the next. In steady state every station stays busy, bubbles squeezed to the run's ends — fill and drain. Pipelines earn their keep when a queue keeps them fed.
So the kingdoms divide by chatter tolerance. Inside a server: the saw, feeding on fast links. Between servers: the pipeline, sipping through the network. And they compose — a frontier model wears both at once, sawn within each chassis, floored across them. Your two-card rig runs the humble end of the same idea.
The trap: expecting a pipeline to speed up you. One request leaves the belt nearly empty — solo chat gets one card's pace with extra steps. Pipelines are throughput machines: they shine with queues, batches, traffic. For one user they buy episode one-oh-four's gift: existence — the big model runs at all. Speed needs crowds.
Two cuts down; the zoo has more: whole copies training on different data. Towers of specialists where only some floors wake per token. Bookkeeping sharded across fleets. And through it all, AI's least glamorous truth: at frontier scale something is always broken — and the run must not care. Next: the other splits.