hawkes.aiKey ideas
- The one idea — MoE replaces each feed-forward layer with a committee of experts and wakes only a few per token — huge capacity, modest per-token compute.
- How it is shown — A dense tower where everything lights per token vs a committee tower where 2 of 100s of chambers wake — same answer quality, a fraction of the work.
- The trap to avoid — Reading "trillion parameters" as "every word touches a trillion weights" — per-token, most of an MoE is asleep.
- What it sets up — So which number is real — total or active?
Trillion-parameter model doesn't mean what you think. Only a couple of specialist chambers wake per floor for each token — the other hundreds sleep right through your question.
The one idea
MoE replaces each feed-forward layer with a committee of experts and wakes only a few per token — huge capacity, modest per-token compute.
The biggest models on Earth carry a secret: per word, most of them is asleep. A trillion parameters on the label — three percent awake for any given token. That's Mixture-of-Experts, the architecture of the trillion-parameter era, and this act opens by taking it apart. First, the baseline you already know. In a dense model — every tower this series has climbed — all weights work on every token. Episode twenty-nine's layers, at total burn: the entire mass fires for a comma. It works beautifully. But it welds capability to cost: grow the knowledge, and every single token pays the bigger bill. The Mixture-of-Experts move renovates the vaults — the feed-forward floors where episode forty parked the model's knowledge. Each floor becomes a committee: dozens or hundreds of separate expert networks, plus a small gate at the door.
How it works — the demo
A dense tower where everything lights per token vs a committee tower where 2 of 100s of chambers wake — same answer quality, a fraction of the work.
The floor's capacity multiplies. And crucially, the gate means no token has to meet the whole committee. The rule: per token, the gate scores the committee and wakes only the top few — often two, sometimes eight — whose outputs blend into the token's stream. The rest sleep, unread, uncomputed, costing nothing. Next token, different experts. The model is enormous everywhere and small anywhere — capacity without the everything-always bill. And who runs on this? At the top end — increasingly everyone. The open trillion-scale giants are Mixture-of-Experts. The famous efficiency champions are.
The trap to avoid
Reading "trillion parameters" as "every word touches a trillion weights" — per-token, most of an MoE is asleep.
Why it matters — and what’s next
So which number is real — total or active?
Several closed frontier models are widely believed to be. The idea itself dates to nineteen-ninety-one; what changed is that scale made its economics irresistible. This is simply how the biggest minds are built now. The trap: reading the label as the experience. "One trillion parameters" does not mean every word drinks from a trillion weights — per token, the overwhelming mass is asleep. Dense intuitions break here: parameter count stopped being one number the day the committee moved in. Which raises the obvious question — if not the total, then what? Meet the two bars that now define every big model: total — everything on the shelves — and active — what one token actually touches. Both real, both mattering in different ways, and confusing them is the era's favorite spec-sheet trick. Next: active versus total parameters — and which number your speed, your bill, and your memory actually follow.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
The biggest models on Earth carry a secret: per word, most of them is asleep. A trillion parameters on the label — three percent awake for any given token. That's Mixture-of-Experts, the architecture of the trillion-parameter era, and this act opens by taking it apart.
First, the baseline you already know. In a dense model — every tower this series has climbed — all weights work on every token. Episode twenty-nine's layers, at total burn: the entire mass fires for a comma. It works beautifully. But it welds capability to cost: grow the knowledge, and every single token pays the bigger bill.
The Mixture-of-Experts move renovates the vaults — the feed-forward floors where episode forty parked the model's knowledge. Each floor becomes a committee: dozens or hundreds of separate expert networks, plus a small gate at the door. The floor's capacity multiplies. And crucially, the gate means no token has to meet the whole committee.
The rule: per token, the gate scores the committee and wakes only the top few — often two, sometimes eight — whose outputs blend into the token's stream. The rest sleep, unread, uncomputed, costing nothing. Next token, different experts. The model is enormous everywhere and small anywhere — capacity without the everything-always bill.
And who runs on this? At the top end — increasingly everyone. The open trillion-scale giants are Mixture-of-Experts. The famous efficiency champions are. Several closed frontier models are widely believed to be. The idea itself dates to nineteen-ninety-one; what changed is that scale made its economics irresistible. This is simply how the biggest minds are built now.
The trap: reading the label as the experience. "One trillion parameters" does not mean every word drinks from a trillion weights — per token, the overwhelming mass is asleep. Dense intuitions break here: parameter count stopped being one number the day the committee moved in. Which raises the obvious question — if not the total, then what?
Meet the two bars that now define every big model: total — everything on the shelves — and active — what one token actually touches. Both real, both mattering in different ways, and confusing them is the era's favorite spec-sheet trick. Next: active versus total parameters — and which number your speed, your bill, and your memory actually follow.