hawkes.aiKey ideas
- The one idea — Frontier models are cut across many GPUs — by layers or through every matrix — and the interconnect becomes the new bandwidth wall.
- How it is shown — A model sliced two ways across a rack: floor-by-floor (pipeline) vs through-every-grid (tensor), with the chatter between cards made visible.
- The trap to avoid — Assuming two GPUs = twice the speed — slicing adds chatter; parallelism is for fitting first, throughput second.
- What it sets up — The full parallelism taxonomy, done properly.
The biggest GPU you can buy holds under two hundred gigabytes. Frontier models spill way past that. So the industry does something barbaric-sounding: it cuts the brain into pieces.
The one idea
Frontier models are cut across many GPUs — by layers or through every matrix — and the interconnect becomes the new bandwidth wall.
A frontier model doesn't fit on one GPU — serving-width weights run to hundreds of gigabytes; the workhorse card holds about eighty, the biggest under two hundred. So the industry cuts the brain into pieces housed across a rack. Two classic styles of cutting — and the difference runs everything. Cut one: by floor. Pipeline parallelism — the first twenty layers here, the next twenty there, and the token rides the relay, handed across at each boundary. Simple, clean — with a tax: while one room works, the others wait. Bubbles of idleness drift through the relay. Tolerable, never free. Cut two: through every grid. Tensor parallelism — every giant matrix split across the rooms, all computing their shards simultaneously, then stitching partial results over the cables.
How it works — the demo
A model sliced two ways across a rack: floor-by-floor (pipeline) vs through-every-grid (tensor), with the chatter between cards made visible.
No bubbles; real speed. The price: stitching happens constantly — every layer, every token. The cables become the machine's most important organ. Notice who's back. Inside a server, cards talk over specialized links — potent, but slimmer than a card's own memory road. Between servers: a network, slimmer still. Episode ninety-two's bandwidth wall didn't die; it climbed a level. The rooms gain a seventh — the other computer — and the commute law rules there too. In practice: serving a big model typically means one chassis, eight cards, tensor-sliced — a single tight organism. Training frontier models means thousands of cards and every cutting style at once — a zoo of parallelisms with its own vocabulary.
The trap to avoid
Assuming two GPUs = twice the speed — slicing adds chatter; parallelism is for fitting first, throughput second.
Why it matters — and what’s next
The full parallelism taxonomy, done properly.
That zoo is where this act goes next. The trap: assuming two cards mean twice the speed. For a model that already fits, slicing adds chatter that eats much of the win — solo chat barely quickens. The honest accounting: parallelism buys fitting first — models that couldn't exist now run — and throughput second, for servers juggling many users. Buy existence, not miracles. Step back: you've walked from a single multiply, through gauges, rooms, presses, and crates, to a brain living as a neighborhood of machines. The act ahead turns the neighborhood into a city — each cut in detail, and the serving tricks that let one rack answer the world. Next: tensor parallelism, properly.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
A frontier model doesn't fit on one GPU — serving-width weights run to hundreds of gigabytes; the workhorse card holds about eighty, the biggest under two hundred. So the industry cuts the brain into pieces housed across a rack. Two classic styles of cutting — and the difference runs everything.
Cut one: by floor. Pipeline parallelism — the first twenty layers here, the next twenty there, and the token rides the relay, handed across at each boundary. Simple, clean — with a tax: while one room works, the others wait. Bubbles of idleness drift through the relay. Tolerable, never free.
Cut two: through every grid. Tensor parallelism — every giant matrix split across the rooms, all computing their shards simultaneously, then stitching partial results over the cables. No bubbles; real speed. The price: stitching happens constantly — every layer, every token. The cables become the machine's most important organ.
Notice who's back. Inside a server, cards talk over specialized links — potent, but slimmer than a card's own memory road. Between servers: a network, slimmer still. Episode ninety-two's bandwidth wall didn't die; it climbed a level. The rooms gain a seventh — the other computer — and the commute law rules there too.
In practice: serving a big model typically means one chassis, eight cards, tensor-sliced — a single tight organism. Training frontier models means thousands of cards and every cutting style at once — a zoo of parallelisms with its own vocabulary. That zoo is where this act goes next.
The trap: assuming two cards mean twice the speed. For a model that already fits, slicing adds chatter that eats much of the win — solo chat barely quickens. The honest accounting: parallelism buys fitting first — models that couldn't exist now run — and throughput second, for servers juggling many users. Buy existence, not miracles.
Step back: you've walked from a single multiply, through gauges, rooms, presses, and crates, to a brain living as a neighborhood of machines. The act ahead turns the neighborhood into a city — each cut in detail, and the serving tricks that let one rack answer the world. Next: tensor parallelism, properly.