hawkes.aiKey ideas
- The one idea — A tiny learned gate scores experts per token and sends it to the top few — trained jointly, with emergent specialties that are surprisingly low-level.
- How it is shown — The router scoring a token against all experts, top-2 lighting; the reveal of real specialties: punctuation, numbers, syntax — not "the math expert."
- The trap to avoid — Imagining human-legible specialists — experts mostly aren't topics, and nobody assigned them.
- What it sets up — What if the committee goes undemocratic?
Inside expert models, the experts mostly aren't what you'd guess — no tidy math department. Real specialties: punctuation, numbers, syntax. And nobody assigned them. They emerged.
The one idea
A tiny learned gate scores experts per token and sends it to the top few — trained jointly, with emergent specialties that are surprisingly low-level.
Every token's path through a trillion parameters is decided by the tower's smallest component: the router. A whisper — scores, a pick, done — repeated at every floor, billions of times a day. How the cop works, who trained it, and the strange truth of what experts specialize in. The mechanism is embarrassingly small: one little matrix multiply — the token's vector scored against each expert's learned direction, episode twenty-six's dot product running the biggest models on Earth. Top scores win; the same scores set the blend. No rules, no lookup table. One multiply, one pick. What most people miss: nobody assigned anything. Router and experts train together, from scratch, under the same next-word pressure as everything else — episode sixty's ritual. Early routing is near-random. Then a division of labor crystallizes: experts drift toward niches, the router learns their map, each shaping the other. The org chart is emergent.
How it works — the demo
The router scoring a token against all experts, top-2 lighting; the reveal of real specialties: punctuation, numbers, syntax — not "the math expert."
So what did they specialize in? Audits expected tidy departments — the math expert, the French expert — and found something stranger: specialties organized by token-texture. An expert drawing punctuation. One favoring numbers. One for code's rhythms, one for connective words. Useful divisions — just not human ones. The committee organized itself in a language we don't speak. Modern designs refined the chamber. A shared expert stays always-on — a generalist absorbing common work, freeing specialists to specialize harder. Committees got finer-grained: many small experts over few large, a higher-resolution canvas. Shared plus fine-grained — the recipe inside the current open giants. The trap: the nameplate fantasy.
The trap to avoid
Imagining human-legible specialists — experts mostly aren't topics, and nobody assigned them.
Why it matters — and what’s next
What if the committee goes undemocratic?
When a pitch says "we route your question to our legal expert," that's not MoE — real specialties are textural, emergent, illegible. Effective? Provably. Explainable in department terms? No. Routing between whole models is a product pattern; routing inside an MoE speaks its own language. The committee has its cop, its org chart, its refinements. Two questions remain. The triumphant: how far does the trick scale? And the uncomfortable, already flickering: what happens when the democracy fails and a few experts hoard the work? Next group.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Every token's path through a trillion parameters is decided by the tower's smallest component: the router. A whisper — scores, a pick, done — repeated at every floor, billions of times a day. How the cop works, who trained it, and the strange truth of what experts specialize in.
The mechanism is embarrassingly small: one little matrix multiply — the token's vector scored against each expert's learned direction, episode twenty-six's dot product running the biggest models on Earth. Top scores win; the same scores set the blend. No rules, no lookup table. One multiply, one pick.
What most people miss: nobody assigned anything. Router and experts train together, from scratch, under the same next-word pressure as everything else — episode sixty's ritual. Early routing is near-random. Then a division of labor crystallizes: experts drift toward niches, the router learns their map, each shaping the other. The org chart is emergent.
So what did they specialize in? Audits expected tidy departments — the math expert, the French expert — and found something stranger: specialties organized by token-texture. An expert drawing punctuation. One favoring numbers. One for code's rhythms, one for connective words. Useful divisions — just not human ones. The committee organized itself in a language we don't speak.
Modern designs refined the chamber. A shared expert stays always-on — a generalist absorbing common work, freeing specialists to specialize harder. Committees got finer-grained: many small experts over few large, a higher-resolution canvas. Shared plus fine-grained — the recipe inside the current open giants.
The trap: the nameplate fantasy. When a pitch says "we route your question to our legal expert," that's not MoE — real specialties are textural, emergent, illegible. Effective? Provably. Explainable in department terms? No. Routing between whole models is a product pattern; routing inside an MoE speaks its own language.
The committee has its cop, its org chart, its refinements. Two questions remain. The triumphant: how far does the trick scale? And the uncomfortable, already flickering: what happens when the democracy fails and a few experts hoard the work? Next group.