hawkes.aiKey ideas
- The one idea — The embedding table: each token ID looks up a long learned vector — a list of thousands of numbers that IS what enters the model.
- How it is shown — ID → row → the list unrolling → a point materializing in space; the glitch-token callback resolving.
- The trap to avoid — Asking which single number means what — meaning lives in the pattern of all coordinates, not in any cell.
- What it sets up — Similar words should be nearby… should they?
Inside the model, a table of up to two hundred thousand rows holds the meaning of every token. Nobody wrote a single entry. Training nudged them into place.
The one idea
The embedding table: each token ID looks up a long learned vector — a list of thousands of numbers that IS what enters the model.
One ID becomes a list of four thousand numbers. That list is "meaning." Let me show you the table where it happens. Everything so far ended at a wall: token forty-two is just a parking spot. Here's the other side of the wall. The model holds a giant table — one row per menu item, up to two hundred thousand rows. Row forty-two is a long list of numbers. Reading out that list — the embedding — is what actually enters the machine. How long a list? GPT-2 made do with seven hundred sixty-eight numbers per token. Llama-class models: around four thousand. Frontier scale: over twelve thousand. Every token you send gets this entire treatment.
How it works — the demo
ID → row → the list unrolling → a point materializing in space; the glitch-token callback resolving.
Where do the values come from? Training. At birth, the table is random noise. Every time the model errs, blame flows backward and nudges the responsible rows — being wrong gets rarer, nudge by nudge. Billions of nudges later, the table has become a map of meaning. Nobody wrote a single entry. Because a list of numbers is a location. Two numbers: a point on a map. Four thousand numbers: a point in four-thousand-dimensional space. You can't picture it — nobody can — but the math doesn't care. Every token now lives somewhere. And now episode nineteen fully resolves: a glitch token is a row that training never touched.
The trap to avoid
Asking which single number means what — meaning lives in the pattern of all coordinates, not in any cell.
Why it matters — and what’s next
Similar words should be nearby… should they?
Its address exists; its coordinates are still birth-noise. Feed it in and you inject a point from nowhere into a space where location is meaning. Haunted — mechanically. The trap: asking which number means what. Cell two thousand ninety is not "the royalty cell." Meaning isn't in any single coordinate — it's in the pattern across all of them together. Directions, not cells. Hold that phrase; it becomes a whole act later. Which sets up the perfect question. If every word is a point, then similar words should be… nearby. Should they? Are "cat" and "kitten" neighbors in there? Next episode: meaning as direction — and the map hidden inside the machine.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
One ID becomes a list of four thousand numbers. That list is "meaning." Let me show you the table where it happens.
Everything so far ended at a wall: token forty-two is just a parking spot. Here's the other side of the wall. The model holds a giant table — one row per menu item, up to two hundred thousand rows. Row forty-two is a long list of numbers. Reading out that list — the embedding — is what actually enters the machine.
How long a list? GPT-2 made do with seven hundred sixty-eight numbers per token. Llama-class models: around four thousand. Frontier scale: over twelve thousand. Every token you send gets this entire treatment.
Where do the values come from? Training. At birth, the table is random noise. Every time the model errs, blame flows backward and nudges the responsible rows — being wrong gets rarer, nudge by nudge. Billions of nudges later, the table has become a map of meaning. Nobody wrote a single entry.
Because a list of numbers is a location. Two numbers: a point on a map. Four thousand numbers: a point in four-thousand-dimensional space. You can't picture it — nobody can — but the math doesn't care. Every token now lives somewhere.
And now episode nineteen fully resolves: a glitch token is a row that training never touched. Its address exists; its coordinates are still birth-noise. Feed it in and you inject a point from nowhere into a space where location is meaning. Haunted — mechanically.
The trap: asking which number means what. Cell two thousand ninety is not "the royalty cell." Meaning isn't in any single coordinate — it's in the pattern across all of them together. Directions, not cells. Hold that phrase; it becomes a whole act later.
Which sets up the perfect question. If every word is a point, then similar words should be… nearby. Should they? Are "cat" and "kitten" neighbors in there? Next episode: meaning as direction — and the map hidden inside the machine.