hawkes.aiKey ideas
- The one idea — Embedding dimension is capacity — more coordinates, more room for distinctions — priced in compute, with no human-readable meaning per dimension.
- How it is shown — The same word at 2D → 50D → 4,000D, distinctions unfolding at each step.
- The trap to avoid — Dimension-worship — and asking "what does dimension 8 mean?" (features live in directions, smeared across coordinates).
- What it sets up — Superposition — "more concepts than dimensions.
Describe every person you know using only two numbers, height and age. Distinct people collapse onto the same spot. That's what a too-small meaning space does to words.
The one idea
Embedding dimension is capacity — more coordinates, more room for distinctions — priced in compute, with no human-readable meaning per dimension.
More numbers per word means more shades of meaning — at a price. Last stop of the foundations: what dimensions actually buy. Describe every person you know with two numbers — height and age. Distinct people collapse onto the same spot. Give yourself three hundred numbers, and humor, taste, and temperament all fit. Dimensions are room to differ. Words need the room. "Bank" by the river, "bank" with your money, "banking on it." A small space forces conflations — distinct senses crushed onto shared coordinates. A big space lets distinctions breathe. Real sizes: GPT-2 made do with seven hundred sixty-eight.
How it works — the demo
The same word at 2D → 50D → 4,000D, distinctions unfolding at each step.
Llama-class models run about four thousand. GPT-3: over twelve thousand. And the embedding services behind search products sell vectors in the fifteen-hundred to three-thousand range. Bigger buys nuance. Bigger also burns. The price: every dimension is more multiplication in every layer, more memory for every remembered token. Width is one of the biggest levers on a model's cost — you'll feel this again, painfully, in the hardware act. Two closing cautions. The wrinkle: models cram more concepts than dimensions into the space, by overlapping them cleverly — superposition, a whole episode in the trust act, because it's why reading a model's mind is hard. And the trap: no individual dimension is a human concept.
The trap to avoid
Dimension-worship — and asking "what does dimension 8 mean?" (features live in directions, smeared across coordinates).
Why it matters — and what’s next
Superposition — "more concepts than dimensions.
Features live in directions, smeared across coordinates. "What does dimension eight mean?" The honest answer: mu. And that's the intake, complete. Bytes to tokens, tokens to IDs, IDs to points in a space where meaning has geography. Act One is done — you now understand the input to every AI on Earth. Next act, the machine itself. It starts embarrassingly small: one neuron. Multiply, add — and one small squash.
Ask an AI researcher what dimension eight of a word vector means, and the honest answer is: nothing you can name. Features live in directions, smeared across coordinates.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:15 of narration
More numbers per word means more shades of meaning — at a price. Last stop of the foundations: what dimensions actually buy.
Describe every person you know with two numbers — height and age. Distinct people collapse onto the same spot. Give yourself three hundred numbers, and humor, taste, and temperament all fit. Dimensions are room to differ.
Words need the room. "Bank" by the river, "bank" with your money, "banking on it." A small space forces conflations — distinct senses crushed onto shared coordinates. A big space lets distinctions breathe.
Real sizes: GPT-2 made do with seven hundred sixty-eight. Llama-class models run about four thousand. GPT-3: over twelve thousand. And the embedding services behind search products sell vectors in the fifteen-hundred to three-thousand range. Bigger buys nuance. Bigger also burns.
The price: every dimension is more multiplication in every layer, more memory for every remembered token. Width is one of the biggest levers on a model's cost — you'll feel this again, painfully, in the hardware act.
Two closing cautions. The wrinkle: models cram more concepts than dimensions into the space, by overlapping them cleverly — superposition, a whole episode in the trust act, because it's why reading a model's mind is hard. And the trap: no individual dimension is a human concept. Features live in directions, smeared across coordinates. "What does dimension eight mean?" The honest answer: mu.
And that's the intake, complete. Bytes to tokens, tokens to IDs, IDs to points in a space where meaning has geography. Act One is done — you now understand the input to every AI on Earth. Next act, the machine itself. It starts embarrassingly small: one neuron. Multiply, add — and one small squash.