hawkes.aiKey ideas
- The one idea — Parameter count is capacity, not smarts — data, training compute, architecture, and post-training decide what fills the capacity.
- How it is shown — The counterexample: a modern 8-billion-parameter model beating 2020's 175-billion giant on standard benchmarks.
- The trap to avoid — Ranking models by the B-number — it predicts memory footprint and serving cost, not capability.
- What it sets up — What fills the capacity.
In 2020, GPT-3 shocked the world with a hundred seventy-five billion parameters. Today, models twenty-two times smaller beat it on standard benchmarks. The count measures shelf space, not smarts.
The one idea
Parameter count is capacity, not smarts — data, training compute, architecture, and post-training decide what fills the capacity.
Bigger isn't always smarter. Here's the counterexample that should recalibrate how you read every model announcement. In 2020, GPT-3 shocked the world with a hundred seventy-five billion parameters. Today, well-trained eight-billion-parameter models beat it across standard benchmarks — twenty-two times smaller, and better. Same test, opposite headline. So what did the count actually measure? Parameter count is capacity — shelf space. What fills the shelves is training: how much data, how good, how long, how well-tuned the run. A giant shelf filled with a thin muddy pour learns less than a small shelf saturated with a rich one. The 2020 giant was, by modern standards, dramatically undertrained. Everyone's giants were. And size is only one lever on a panel.
How it works — the demo
The counterexample: a modern 8-billion-parameter model beating 2020's 175-billion giant on standard benchmarks.
Data quality — the deciding one, again and again. Training compute. Architecture — how the same count of numbers is arranged. And post-training: the finishing work that turns a raw predictor into a useful assistant. Every one of these gets its own episodes ahead. The count is one dial among five. None of this makes the number useless — it just tells you different things than the marketing implies. Parameter count predicts memory footprint, serving cost, and what hardware class you'll need. It's a logistics number. Treat it like a shipping weight, not an IQ score. And a modern wrinkle blurs the number further: some architectures use only a fraction of their weights on any given token — a trillion stored, a fraction active. "Total versus active parameters" gets its own episode in the model-zoo act.
The trap to avoid
Ranking models by the B-number — it predicts memory footprint and serving cost, not capability.
Why it matters — and what’s next
What fills the capacity.
For now: the single headline number has never told you less. So the trap, plainly: ranking models by the B-number. When a vendor leads with size, the questions that matter are the ones not on the slide: trained on what? For how long? Evaluated how — and by whom? You'll be able to ask all three sharply by the end of this series. Capacity isn't content. Hold that, and model announcements become legible. Now — back into the machine. One neuron asks one question. Next: a thousand of them, asking at once.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Bigger isn't always smarter. Here's the counterexample that should recalibrate how you read every model announcement.
In 2020, GPT-3 shocked the world with a hundred seventy-five billion parameters. Today, well-trained eight-billion-parameter models beat it across standard benchmarks — twenty-two times smaller, and better. Same test, opposite headline. So what did the count actually measure?
Parameter count is capacity — shelf space. What fills the shelves is training: how much data, how good, how long, how well-tuned the run. A giant shelf filled with a thin muddy pour learns less than a small shelf saturated with a rich one. The 2020 giant was, by modern standards, dramatically undertrained. Everyone's giants were.
And size is only one lever on a panel. Data quality — the deciding one, again and again. Training compute. Architecture — how the same count of numbers is arranged. And post-training: the finishing work that turns a raw predictor into a useful assistant. Every one of these gets its own episodes ahead. The count is one dial among five.
None of this makes the number useless — it just tells you different things than the marketing implies. Parameter count predicts memory footprint, serving cost, and what hardware class you'll need. It's a logistics number. Treat it like a shipping weight, not an IQ score.
And a modern wrinkle blurs the number further: some architectures use only a fraction of their weights on any given token — a trillion stored, a fraction active. "Total versus active parameters" gets its own episode in the model-zoo act. For now: the single headline number has never told you less.
So the trap, plainly: ranking models by the B-number. When a vendor leads with size, the questions that matter are the ones not on the slide: trained on what? For how long? Evaluated how — and by whom? You'll be able to ask all three sharply by the end of this series.
Capacity isn't content. Hold that, and model announcements become legible. Now — back into the machine. One neuron asks one question. Next: a thousand of them, asking at once.