hawkes.aiKey ideas
- The one idea — For any compute budget there's a best balance of model size and data — roughly 20 tokens per parameter — and the 2020-era giants were dramatically undertrained; the modern twist over-trains small models because inference pays the bills.
- How it is shown — Same budget, two allocations: the 280B giant vs the balanced 70B on more data — the smaller one winning; then Llama-3-8B at ~1,900 tokens/param.
- The trap to avoid — Treating "compute-optimal" as the final word — optimal for training cost isn't optimal for serving cost; the frontier deliberately "overtrains" now.
- What it sets up — Cheap-to-serve small models.
In twenty-twenty-two, a model four times smaller beat its giant stablemate across the board — same compute, four times the data. The famous giants weren't dumb. They were starving.
The one idea
For any compute budget there's a best balance of model size and data — roughly 20 tokens per parameter — and the 2020-era giants were dramatically undertrained; the modern twist over-trains small models because inference pays the bills.
Same compute budget, split two ways: a giant model on thin data, or a modest model on a torrent. One experiment settled it — and embarrassed nearly every famous model alive. Bigger isn't optimal. Balanced is. The era it corrected: the race read scaling laws as a height contest — one-seventy-five billion parameters, two-eighty, five-thirty — each giant fed conspicuously thin data. The baked-in assumption: given more compute, grow the model. The experiment — twenty-twenty-two, DeepMind: hundreds of runs sweeping the split at matched budgets. The result: a valley with a clean floor — loss minimizes when size and tokens grow together, roughly twenty tokens per parameter. The reigning giants sat far up the wall: too large, underfed. Compute-optimal had a formula nobody had followed.
How it works — the demo
Same budget, two allocations: the 280B giant vs the balanced 70B on more data — the smaller one winning; then Llama-3-8B at ~1,900 tokens/param.
The demonstration had a name: Chinchilla. Seventy billion parameters — four times smaller than its stablemate Gopher — on four times the tokens, same compute. It won across the board. And episode twenty-eight's thread resolves: this is why a modern eight-B beats the old giants. They weren't dumb. They were starving. The modern twist: labs now train far past the floor — Llama-3's eight-B drank fifteen trillion tokens, near a hundred times the ratio. Wasteful by training math; brilliant by business math: training is paid once, serving forever, and a small model made unusually strong is cheap every day of its life. The optimum moved when the bill did. The trap: canonizing the ratio.
The trap to avoid
Treating "compute-optimal" as the final word — optimal for training cost isn't optimal for serving cost; the frontier deliberately "overtrains" now.
Why it matters — and what’s next
Cheap-to-serve small models.
Chinchilla answered one question — best loss for a training budget — and the industry faced others: serving budgets, latency targets. Twenty-per-one is a landmark, not a law. What survives every revision: allocation beats accumulation. The split outranks the spend. With balance, the furnace fundamentals are complete: game, number, blame, walk, fuel, price, laws. But what comes out isn't the assistant you know. A freshly-trained model is a weirdo — it completes text; it doesn't answer. The finishing school is the back half of this act. Next.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Same compute budget, split two ways: a giant model on thin data, or a modest model on a torrent. One experiment settled it — and embarrassed nearly every famous model alive. Bigger isn't optimal. Balanced is.
The era it corrected: the race read scaling laws as a height contest — one-seventy-five billion parameters, two-eighty, five-thirty — each giant fed conspicuously thin data. The baked-in assumption: given more compute, grow the model.
The experiment — twenty-twenty-two, DeepMind: hundreds of runs sweeping the split at matched budgets. The result: a valley with a clean floor — loss minimizes when size and tokens grow together, roughly twenty tokens per parameter. The reigning giants sat far up the wall: too large, underfed. Compute-optimal had a formula nobody had followed.
The demonstration had a name: Chinchilla. Seventy billion parameters — four times smaller than its stablemate Gopher — on four times the tokens, same compute. It won across the board. And episode twenty-eight's thread resolves: this is why a modern eight-B beats the old giants. They weren't dumb. They were starving.
The modern twist: labs now train far past the floor — Llama-3's eight-B drank fifteen trillion tokens, near a hundred times the ratio. Wasteful by training math; brilliant by business math: training is paid once, serving forever, and a small model made unusually strong is cheap every day of its life. The optimum moved when the bill did.
The trap: canonizing the ratio. Chinchilla answered one question — best loss for a training budget — and the industry faced others: serving budgets, latency targets. Twenty-per-one is a landmark, not a law. What survives every revision: allocation beats accumulation. The split outranks the spend.
With balance, the furnace fundamentals are complete: game, number, blame, walk, fuel, price, laws. But what comes out isn't the assistant you know. A freshly-trained model is a weirdo — it completes text; it doesn't answer. The finishing school is the back half of this act. Next.