hawkes.aiKey ideas
- The one idea — Data parallelism, expert parallelism, and ZeRO's sharded bookkeeping — composed so 10,000-GPU runs survive constant hardware failure.
- How it is shown — A flotilla of model copies averaging gradients; training bookkeeping sharded; a failure mid-run absorbed by checkpoint-resume without stopping the fleet.
- The trap to avoid — Thinking frontier training is "buy GPUs, press go" — the real engineering is fault-tolerance choreography.
- What it sets up — Experts get their own act.
You think frontier AI training is buy ten thousand GPUs and press go? The real engineering is surviving the crash that happens every few hours, without stopping.
The one idea
Data parallelism, expert parallelism, and ZeRO's sharded bookkeeping — composed so 10,000-GPU runs survive constant hardware failure.
On a ten-thousand-GPU training run, hardware fails constantly — a card dies, a link drops, memory quits. One documented frontier run logged hundreds of unplanned interruptions in under two months. And the run barely flinches. Tonight: the last three splits, and the choreography that makes failure boring. Split three doesn't cut the model at all — it copies it. Data parallelism: a flotilla of identical towers, each digesting a different slice of the river, each computing its private nudges. The nudges average; every copy takes the same step together. The oldest scaling trick, still the backbone: more data per hour, bought with more copies. But copies carry baggage. Training keeps gradients and optimizer state — momentum records, master weights — several times the model's own size, and naive copying duplicates it all on every ship.
How it works — the demo
A flotilla of model copies averaging gradients; training bookkeeping sharded; a failure mid-run absorbed by checkpoint-resume without stopping the fleet.
ZeRO is the reform: shard the baggage across the fleet, each copy keeping one slice, borrowing the rest when needed. Same math, fraction of the memory — and suddenly bigger models fit the same fleet. Split four gets a glimpse, because it deserves an act: expert parallelism. Some towers keep committees of specialist sub-networks — and each token wakes just one or two, so specialists can live on different machines, tokens routed to them. Sparse, strange traffic — the architecture behind several frontier models. The full story — mixture of experts — opens the next act. The masterpiece that makes scale survivable: checkpoints. On a rhythm, the entire run — weights, optimizer, data position — freezes into a snapshot. When hardware dies, the fleet swaps the part, rewinds to the last snapshot, resumes. Automated, constant, boring by design.
The trap to avoid
Thinking frontier training is "buy GPUs, press go" — the real engineering is fault-tolerance choreography.
Why it matters — and what’s next
Experts get their own act.
Episode sixty-two's loss curve descends through hundreds of failures without visibly noticing one. The trap: believing frontier training is "buy GPUs, press go." The GPUs are table stakes. The craft is composition — tensor inside servers, pipeline across them, data across the fleet, baggage sharded, experts routed — while the failure ballet runs underneath. That orchestration layer is a real moat — why compute-rich newcomers still stumble. The machines are for sale. The choreography isn't. So: you've seen how giant brains get built across machines. The act's remaining stretch flips to the other building — serving, where a finished model faces millions of small, impatient requests, and the physics you've learned becomes pricing, queues, and product feel. First door: the hidden tax of talking between chips. Next.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
On a ten-thousand-GPU training run, hardware fails constantly — a card dies, a link drops, memory quits. One documented frontier run logged hundreds of unplanned interruptions in under two months. And the run barely flinches. Tonight: the last three splits, and the choreography that makes failure boring.
Split three doesn't cut the model at all — it copies it. Data parallelism: a flotilla of identical towers, each digesting a different slice of the river, each computing its private nudges. The nudges average; every copy takes the same step together. The oldest scaling trick, still the backbone: more data per hour, bought with more copies.
But copies carry baggage. Training keeps gradients and optimizer state — momentum records, master weights — several times the model's own size, and naive copying duplicates it all on every ship. ZeRO is the reform: shard the baggage across the fleet, each copy keeping one slice, borrowing the rest when needed. Same math, fraction of the memory — and suddenly bigger models fit the same fleet.
Split four gets a glimpse, because it deserves an act: expert parallelism. Some towers keep committees of specialist sub-networks — and each token wakes just one or two, so specialists can live on different machines, tokens routed to them. Sparse, strange traffic — the architecture behind several frontier models. The full story — mixture of experts — opens the next act.
The masterpiece that makes scale survivable: checkpoints. On a rhythm, the entire run — weights, optimizer, data position — freezes into a snapshot. When hardware dies, the fleet swaps the part, rewinds to the last snapshot, resumes. Automated, constant, boring by design. Episode sixty-two's loss curve descends through hundreds of failures without visibly noticing one.
The trap: believing frontier training is "buy GPUs, press go." The GPUs are table stakes. The craft is composition — tensor inside servers, pipeline across them, data across the fleet, baggage sharded, experts routed — while the failure ballet runs underneath. That orchestration layer is a real moat — why compute-rich newcomers still stumble. The machines are for sale. The choreography isn't.
So: you've seen how giant brains get built across machines. The act's remaining stretch flips to the other building — serving, where a finished model faces millions of small, impatient requests, and the physics you've learned becomes pricing, queues, and product feel. First door: the hidden tax of talking between chips. Next.