hawkes.aiKey ideas
- The one idea — Loss falls as a smooth, predictable power law in parameters, data, and compute — and that predictability, more than any single result, is the industry's business case.
- How it is shown — The straight line on the log-log chart; small cheap runs forecasting a giant run's loss before the check is signed.
- The trap to avoid — The law predicts loss, not abilities — capabilities arrive unevenly along the smooth curve (the emergence debate, deferred honestly).
- What it sets up — The law says spend more — but spend it HOW?
GPT-4's final loss was predicted in advance — read off a line fitted to far smaller, cheaper runs. That prophecy is why the billions get spent.
The one idea
Loss falls as a smooth, predictable power law in parameters, data, and compute — and that predictability, more than any single result, is the industry's business case.
One graph explains why everyone is spending billions. A straight line on strange axes, and a decade of dots that keep landing on it. The scaling laws. The finding, published in twenty-twenty and confirmed relentlessly since: grow model, data, and compute together, and loss falls along a power law — a straight line when both axes count in tenfold leaps. No cliffs, no walls, no plateau in sight at any scale yet reached. Smooth, boring, relentless — and the boringness is the value. Because here's what a smooth law buys: prophecy. Run small, cheap experiments, fit the line, extend it — and read off what a ten-thousand-times-larger run will achieve before signing the check. GPT-4's final loss was publicly reported as predicted, accurately, from far smaller runs. Nobody bets a jet on a mystery — they bet it on a line that hasn't yet lied about loss.
How it works — the demo
The straight line on the log-log chart; small cheap runs forecasting a giant run's loss before the check is signed.
The law has three dials — parameters, data, compute — and each obeys its own power law. Turn one alone and gains starve: a giant model on thin data, an ocean through a tiny model. Together, they compound. Which plants next episode's question: for a fixed budget, what's the right ratio? The industry's first assumption was expensively wrong. And understand what this line did to the world. It converted research into capital allocation: spend ten times more, receive a predictable improvement — repeatably. That sentence is why "just scale it" graduated from hunch to strategy to industry — why the price tags from last episode keep being paid. The line is the business case. The trap — and it's where the confident slides go quiet: the law predicts loss, not abilities.
The trap to avoid
The law predicts loss, not abilities — capabilities arrive unevenly along the smooth curve (the emergence debate, deferred honestly).
Why it matters — and what’s next
The law says spend more — but spend it HOW?
The curve is smooth; capabilities arrive along it unevenly — some appearing to switch on abruptly between one scale and the next. Whether those jumps are real emergence or artifacts of how we measure is a live scientific fight, and episode eighty-eight referees it properly. What's safe to say: the line forecasts the score, not the skills. Bet on it accordingly. So the law says spend more, and the industry obeyed. But the law never said how to split the spend — how much on size, how much on data — and for two years, everyone split it wrong. Next: Chinchilla. The correction that embarrassed an era of giants, and finally pays off a promise from episode twenty-eight.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
One graph explains why everyone is spending billions. A straight line on strange axes, and a decade of dots that keep landing on it. The scaling laws.
The finding, published in twenty-twenty and confirmed relentlessly since: grow model, data, and compute together, and loss falls along a power law — a straight line when both axes count in tenfold leaps. No cliffs, no walls, no plateau in sight at any scale yet reached. Smooth, boring, relentless — and the boringness is the value.
Because here's what a smooth law buys: prophecy. Run small, cheap experiments, fit the line, extend it — and read off what a ten-thousand-times-larger run will achieve before signing the check. GPT-4's final loss was publicly reported as predicted, accurately, from far smaller runs. Nobody bets a jet on a mystery — they bet it on a line that hasn't yet lied about loss.
The law has three dials — parameters, data, compute — and each obeys its own power law. Turn one alone and gains starve: a giant model on thin data, an ocean through a tiny model. Together, they compound. Which plants next episode's question: for a fixed budget, what's the right ratio? The industry's first assumption was expensively wrong.
And understand what this line did to the world. It converted research into capital allocation: spend ten times more, receive a predictable improvement — repeatably. That sentence is why "just scale it" graduated from hunch to strategy to industry — why the price tags from last episode keep being paid. The line is the business case.
The trap — and it's where the confident slides go quiet: the law predicts loss, not abilities. The curve is smooth; capabilities arrive along it unevenly — some appearing to switch on abruptly between one scale and the next. Whether those jumps are real emergence or artifacts of how we measure is a live scientific fight, and episode eighty-eight referees it properly. What's safe to say: the line forecasts the score, not the skills. Bet on it accordingly.
So the law says spend more, and the industry obeyed. But the law never said how to split the spend — how much on size, how much on data — and for two years, everyone split it wrong. Next: Chinchilla. The correction that embarrassed an era of giants, and finally pays off a promise from episode twenty-eight.