hawkes.aiKey ideas
- The one idea — Adding experts grows capacity without growing per-token compute — the decoupling that made trillion-parameter models affordable.
- How it is shown — A tower widening — committees swelling from 8 to 64 to 256 experts — while the per-token lit path stays the same width; the cost line flat as the mass line soars.
- The trap to avoid — "Cheap" means compute-cheap: memory follows total, and routing traffic taxes the interconnect.
- What it sets up — The imbalance crack.
In a dense model, every new parameter makes every word cost more. Experts broke that link — capacity soars while per-word compute stays flat. That decoupling built the trillion era.
The one idea
Adding experts grows capacity without growing per-token compute — the decoupling that made trillion-parameter models affordable.
The trick that built the trillion-parameter era: growth that doesn't touch the bill. Watch the tower widen — eight experts per floor, sixty-four, two-fifty-six — while every token's path keeps the same slender width. Mass soars; per-word cost stays flat. Why does that line refuse to rise? The dense law: grow the knowledge and every token pays more — capacity and cost, handcuffed. The committee breaks the cuffs. New experts join the shelves, not the path: a token still wakes its top few, whatever the committee's size. Capacity climbs a free axis. That decoupling is the whole economic story of modern giants. What did it buy?
How it works — the demo
A tower widening — committees swelling from 8 to 64 to 256 experts — while the per-token lit path stays the same width; the cost line flat as the mass line soars.
First the proofs: research models hit a trillion parameters years ago at mid-size compute. Then the era: today's open six-hundred-billion and trillion-class giants — streaming with thirty-billion gaits — could not exist as dense models at survivable cost. One choice repopulated the frontier. The dividend pays during training too: at identical compute budgets, committee models repeatedly train better than dense ones — more capacity per flop burned, act three's scaling economics bent kindly. When compute is Earth's scarcest resource — act four showed it is — an architecture buying more capability per flop doesn't stay optional. It becomes the default. Now the fine print — "cheap" has a precise meaning: compute-cheap. Memory still follows the total — every shelf rented, episode one-twenty-three. And distribution adds a toll: experts spread across machines, tokens commuting to them and back — all-to-all traffic on episode one-oh-eight's interconnect. The committee saved arithmetic and spent plumbing.
The trap to avoid
"Cheap" means compute-cheap: memory follows total, and routing traffic taxes the interconnect.
Why it matters — and what’s next
The imbalance crack.
The trap is the one-word summary. "Cheap" means cheap per token, at scale, for operators with the racks and fabric. It doesn't mean small or laptop-bound — the giants remain datacenter organisms. For everyone else, the gift arrives through the APIs those operators run, priced by the flat little line. But the entire bargain rests on one assumption: that the committee shares the work. And routers, left alone, are rich-get-richer machines — favored experts train more, improve more, get favored more, until a few cells hoard everything and paid-for capacity sits atrophied. The democracy needs enforcement. Next: MoE's headache — keeping experts balanced.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
The trick that built the trillion-parameter era: growth that doesn't touch the bill. Watch the tower widen — eight experts per floor, sixty-four, two-fifty-six — while every token's path keeps the same slender width. Mass soars; per-word cost stays flat. Why does that line refuse to rise?
The dense law: grow the knowledge and every token pays more — capacity and cost, handcuffed. The committee breaks the cuffs. New experts join the shelves, not the path: a token still wakes its top few, whatever the committee's size. Capacity climbs a free axis. That decoupling is the whole economic story of modern giants.
What did it buy? First the proofs: research models hit a trillion parameters years ago at mid-size compute. Then the era: today's open six-hundred-billion and trillion-class giants — streaming with thirty-billion gaits — could not exist as dense models at survivable cost. One choice repopulated the frontier.
The dividend pays during training too: at identical compute budgets, committee models repeatedly train better than dense ones — more capacity per flop burned, act three's scaling economics bent kindly. When compute is Earth's scarcest resource — act four showed it is — an architecture buying more capability per flop doesn't stay optional. It becomes the default.
Now the fine print — "cheap" has a precise meaning: compute-cheap. Memory still follows the total — every shelf rented, episode one-twenty-three. And distribution adds a toll: experts spread across machines, tokens commuting to them and back — all-to-all traffic on episode one-oh-eight's interconnect. The committee saved arithmetic and spent plumbing.
The trap is the one-word summary. "Cheap" means cheap per token, at scale, for operators with the racks and fabric. It doesn't mean small or laptop-bound — the giants remain datacenter organisms. For everyone else, the gift arrives through the APIs those operators run, priced by the flat little line.
But the entire bargain rests on one assumption: that the committee shares the work. And routers, left alone, are rich-get-richer machines — favored experts train more, improve more, get favored more, until a few cells hoard everything and paid-for capacity sits atrophied. The democracy needs enforcement. Next: MoE's headache — keeping experts balanced.