hawkes.aiKey ideas
- The one idea — Pretraining = predict the next token across the whole internet — self-supervised, so every position is a free exam question with the answer attached.
- How it is shown — The sliding-panel game over streaming text; one page yielding hundreds of free exercises; the label-factory problem dissolving.
- The trap to avoid — "It's just a word game" — getting better at the game at scale requires absorbing grammar, facts, and structure; the byproducts are the product.
- What it sets up — How wrong was the guess?" needs a number.
Nobody taught AI grammar, geography, or reasoning. Training chased exactly one number, guess quality, and everything else showed up as residue. The byproducts are the product.
The one idea
Pretraining = predict the next token across the whole internet — self-supervised, so every position is a free exam question with the answer attached.
Its only childhood lesson: guess the next word. Not read comprehension, not logic drills, not curated facts. One game, trillions of rounds. Tonight: why that game — and the quiet genius inside it. To see the genius, see the problem it dissolved. Machine learning traditionally ran on labeled examples — humans marking each one: this is a cat, this sentence is positive. Useful, and hopeless at scale: teaching a machine everything needs trillions of examples, and no workforce on Earth could label them. For decades, that wall stood. The trick that broke the wall: text labels itself. At every position in every sentence ever written, there's a question — the words so far — and an answer, sitting right next to it: the word that actually came next. One page yields hundreds of exercises. Nobody writes them; reality already did.
How it works — the demo
The sliding-panel game over streaming text; one page yielding hundreds of free exercises; the label-factory problem dissolving.
The field calls it self-supervision, and it converted the entire written internet into free curriculum overnight. And here's why the humble game teaches everything. Guessing the next word of real text, well, demands whatever that text demands. Sometimes the next word is grammar. Sometimes it's a fact. Sometimes it's the consequence of a story, the resolution of an argument, the next line of working code. The internet's next words, taken together, quietly require nearly every competence humans put into writing. Master the game, and the masteries follow. Sit with the strangeness: nobody optimized for grammar. No objective said "learn geography" or "acquire reasoning." One number was chased — guess quality — and everything else condensed as residue. The byproducts are the product. It's the strangest fact in modern AI — and this act's remaining episodes are its consequences.
The trap to avoid
"It's just a word game" — getting better at the game at scale requires absorbing grammar, facts, and structure; the byproducts are the product.
Why it matters — and what’s next
How wrong was the guess?" needs a number.
The trap: hearing "next-word prediction" and filing the whole enterprise as trivial. The game is humble; the exam is everything humanity ever wrote. Layer-one truth, layer-three blindness — episode six's lesson. What you can honestly say: it learned everything sideways, in service of a guess. What you can't say: that this made what it learned small. One piece is missing from the ritual. "Compare the guess to the truth" — compare how? The furnace can't act on a vibe; it needs a number. One number that says exactly how wrong this guess was, so the nudges know how hard to push. That number has a name, a shape, and a personality. Next: loss — the number the machine lives to shrink.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
Its only childhood lesson: guess the next word. Not read comprehension, not logic drills, not curated facts. One game, trillions of rounds. Tonight: why that game — and the quiet genius inside it.
To see the genius, see the problem it dissolved. Machine learning traditionally ran on labeled examples — humans marking each one: this is a cat, this sentence is positive. Useful, and hopeless at scale: teaching a machine everything needs trillions of examples, and no workforce on Earth could label them. For decades, that wall stood.
The trick that broke the wall: text labels itself. At every position in every sentence ever written, there's a question — the words so far — and an answer, sitting right next to it: the word that actually came next. One page yields hundreds of exercises. Nobody writes them; reality already did. The field calls it self-supervision, and it converted the entire written internet into free curriculum overnight.
And here's why the humble game teaches everything. Guessing the next word of real text, well, demands whatever that text demands. Sometimes the next word is grammar. Sometimes it's a fact. Sometimes it's the consequence of a story, the resolution of an argument, the next line of working code. The internet's next words, taken together, quietly require nearly every competence humans put into writing. Master the game, and the masteries follow.
Sit with the strangeness: nobody optimized for grammar. No objective said "learn geography" or "acquire reasoning." One number was chased — guess quality — and everything else condensed as residue. The byproducts are the product. It's the strangest fact in modern AI — and this act's remaining episodes are its consequences.
The trap: hearing "next-word prediction" and filing the whole enterprise as trivial. The game is humble; the exam is everything humanity ever wrote. Layer-one truth, layer-three blindness — episode six's lesson. What you can honestly say: it learned everything sideways, in service of a guess. What you can't say: that this made what it learned small.
One piece is missing from the ritual. "Compare the guess to the truth" — compare how? The furnace can't act on a vibe; it needs a number. One number that says exactly how wrong this guess was, so the nudges know how hard to push. That number has a name, a shape, and a personality. Next: loss — the number the machine lives to shrink.