hawkes.aiKey ideas
- The one idea — The output head projects the final stream vector onto the whole vocabulary — one learned matrix, one score per token, computed fresh every step.
- How it is shown — The summit unwrapped: the last conveyor vector fanning out through the head into two hundred thousand simultaneous scores.
- The trap to avoid — The head predicts a distribution, not a word — choosing is separate machinery, and conflating them hides where control lives.
- What it sets up — Intermediate floors already "lean" toward answers (logit lens).
Every single word your AI writes, it first scores all two hundred thousand tokens it knows — the entire vocabulary, rescored from scratch, every word, every reply.
The one idea
The output head projects the final stream vector onto the whole vocabulary — one learned matrix, one score per token, computed fresh every step.
This is the summit — the apparatus the whole tower exists to feed. One final step: a single vector becomes a score for every possible token in the vocabulary. What arrives is the last token's stream — the conveyor from episode thirty, now carrying a hundred floors of accumulated annotation. Everything the machine gathered, retrieved, and resolved about your text has been distilled, floor by floor, into this one vector. By the top, its job is singular: encode what comes next. The mechanism is one matrix — the head — one row per vocabulary entry. Two hundred thousand of them. Each row is pressed against the arriving vector — and yes, that's the dot product, making its fourth appearance: word similarity, the neuron, attention scoring, and now this. Each row asks "how much does the situation resemble me being next?" Two hundred thousand questions, one flash, and the scoreboard lights. And recognize the shape: the head is the embedding table's mirror.
How it works — the demo
The summit unwrapped: the last conveyor vector fanning out through the head into two hundred thousand simultaneous scores.
The entrance turned an ID into a vector; the exit turns a vector into a score for every ID. Same species of table, pointed opposite ways — and in many models, famously, they're literally the same weights shared, entrance and exit, one table working both doors. And hold the tempo: this is not a one-time event. Every single generated word triggers the full ceremony — a fresh final vector, a fresh press against all two hundred thousand rows, a fresh skyline. The scoreboard you're looking at is rebuilt for every token of every reply you have ever received. One gorgeous trick before we leave: press a middle floor's vector against the head, out of turn — and a ghost of the final skyline is already there. Hazy at floor forty, sharper at eighty, crisp at the summit. The prediction isn't born at the top; it crystallizes on the way up. The interpretability act makes an instrument of this. Now the trap, and it reorganizes everything downstream: the head does not pick a word.
The trap to avoid
The head predicts a distribution, not a word — choosing is separate machinery, and conflating them hides where control lives.
Why it matters — and what’s next
Intermediate floors already "lean" toward answers (logit lens).
It produces an opinion — the full scored field, nothing more. Choosing is separate machinery, with its own dials, its own randomness, its own failure modes. Conflate the two and you'll blame the model for the chooser's behavior. Keep them apart and you'll know exactly where control lives. That scored skyline has a name — logits — and it is the model's real output. Not text. This. Next episode we walk the skyline and learn to read it, because everything the rest of this series does to models — steer them, constrain them, measure their honesty — is done to what you're looking at right now.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 3:00 of narration
This is the summit — the apparatus the whole tower exists to feed. One final step: a single vector becomes a score for every possible token in the vocabulary.
What arrives is the last token's stream — the conveyor from episode thirty, now carrying a hundred floors of accumulated annotation. Everything the machine gathered, retrieved, and resolved about your text has been distilled, floor by floor, into this one vector. By the top, its job is singular: encode what comes next.
The mechanism is one matrix — the head — one row per vocabulary entry. Two hundred thousand of them. Each row is pressed against the arriving vector — and yes, that's the dot product, making its fourth appearance: word similarity, the neuron, attention scoring, and now this. Each row asks "how much does the situation resemble me being next?" Two hundred thousand questions, one flash, and the scoreboard lights.
And recognize the shape: the head is the embedding table's mirror. The entrance turned an ID into a vector; the exit turns a vector into a score for every ID. Same species of table, pointed opposite ways — and in many models, famously, they're literally the same weights shared, entrance and exit, one table working both doors.
And hold the tempo: this is not a one-time event. Every single generated word triggers the full ceremony — a fresh final vector, a fresh press against all two hundred thousand rows, a fresh skyline. The scoreboard you're looking at is rebuilt for every token of every reply you have ever received.
One gorgeous trick before we leave: press a middle floor's vector against the head, out of turn — and a ghost of the final skyline is already there. Hazy at floor forty, sharper at eighty, crisp at the summit. The prediction isn't born at the top; it crystallizes on the way up. The interpretability act makes an instrument of this.
Now the trap, and it reorganizes everything downstream: the head does not pick a word. It produces an opinion — the full scored field, nothing more. Choosing is separate machinery, with its own dials, its own randomness, its own failure modes. Conflate the two and you'll blame the model for the chooser's behavior. Keep them apart and you'll know exactly where control lives.
That scored skyline has a name — logits — and it is the model's real output. Not text. This. Next episode we walk the skyline and learn to read it, because everything the rest of this series does to models — steer them, constrain them, measure their honesty — is done to what you're looking at right now.