hawkes.aiKey ideas
- The one idea — Byte-Pair Encoding: start with characters, merge the most common adjacent pair, repeat — that's how every token is born.
- How it is shown — Live BPE on the corpus "banana banana band": watch "banana" collapse from six characters to two chunks, then "bandana" get assembled from learned parts.
- The trap to avoid — Thinking someone designed the token list — frequency chose everything; every quirk of the last five episodes is this loop at planet scale.
- What it sets up — The loop leaves behind a frozen ranked list.
The recipe that builds GPT's token menu is three words long: count, merge, repeat. Watch it run on 'banana banana band' and you'll miss nothing.
The one idea
Byte-Pair Encoding: start with characters, merge the most common adjacent pair, repeat — that's how every token is born.
Give me a tiny alphabet, and I'll show you exactly how tokens are born. The recipe is called Byte-Pair Encoding, and it's almost insultingly simple. Count. Merge. Repeat. That's the entire algorithm behind every tokenizer you've met in this series. Our whole training world is three words: banana, banana, band. Chop everything to characters — the smallest menu possible. Now the loop begins. Watch every step — there are only two. Count every adjacent pair. A-N appears five times. N-A: four. B-A: three. N-D: once.
How it works — the demo
Live BPE on the corpus "banana banana band": watch "banana" collapse from six characters to two chunks, then "bandana" get assembled from learned parts.
The winner, by frequency: A-N. Merge the winner into one new symbol — AN — add it to the menu, and rewrite the corpus with it. Banana just went from six pieces to four. That's one merge. That's the whole move. Count again — the pairs have changed, because AN is now a unit. B-AN wins. Merge. Then a tie — the rulebook breaks it: BAN-AN. Merge. Watch banana collapse: six characters, four pieces, three… two. The word is condensing into itself. Now scale the toy. Run a thousand merges on real text: a small menu of short chunks. Run a hundred thousand, two hundred thousand: common words become single tokens.
The trap to avoid
Thinking someone designed the token list — frequency chose everything; every quirk of the last five episodes is this loop at planet scale.
Why it matters — and what’s next
The loop leaves behind a frozen ranked list.
That mysterious "vocab size" in every model card is just — how many merges they ran. Fifty thousand for GPT-2's chopper; roughly a hundred thousand for GPT-4's. The dial is a business decision: bigger menus compress text better, but cost memory at the meaning-table. Now feed it a word it never saw: bandana. The learned merges apply where they fit: BAN — D — AN — A. No panic, no error, no "unknown." Words it's never met get assembled from parts it has. Remember strawberry? You're looking at why it arrives in pieces. The trap: thinking someone designed the token list. Nobody chose "straw" plus "berry." Nobody decided Thai pays triple. Frequency chose — this exact loop, run on a mountain of text. Every quirk from the last five episodes is just this counting, at planet scale. And when the counting stops, what's left behind is strangely humble: a ranked list of merges, frozen in the order they were learned. Next episode: why that list — that list alone — IS the tokenizer.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 3:00 of narration
Give me a tiny alphabet, and I'll show you exactly how tokens are born.
The recipe is called Byte-Pair Encoding, and it's almost insultingly simple. Count. Merge. Repeat. That's the entire algorithm behind every tokenizer you've met in this series.
Our whole training world is three words: banana, banana, band. Chop everything to characters — the smallest menu possible. Now the loop begins. Watch every step — there are only two.
Count every adjacent pair. A-N appears five times. N-A: four. B-A: three. N-D: once. The winner, by frequency: A-N.
Merge the winner into one new symbol — AN — add it to the menu, and rewrite the corpus with it. Banana just went from six pieces to four. That's one merge. That's the whole move.
Count again — the pairs have changed, because AN is now a unit. B-AN wins. Merge. Then a tie — the rulebook breaks it: BAN-AN. Merge. Watch banana collapse: six characters, four pieces, three… two. The word is condensing into itself.
Now scale the toy. Run a thousand merges on real text: a small menu of short chunks. Run a hundred thousand, two hundred thousand: common words become single tokens. That mysterious "vocab size" in every model card is just — how many merges they ran. Fifty thousand for GPT-2's chopper; roughly a hundred thousand for GPT-4's. The dial is a business decision: bigger menus compress text better, but cost memory at the meaning-table.
Now feed it a word it never saw: bandana. The learned merges apply where they fit: BAN — D — AN — A. No panic, no error, no "unknown." Words it's never met get assembled from parts it has. Remember strawberry? You're looking at why it arrives in pieces.
The trap: thinking someone designed the token list. Nobody chose "straw" plus "berry." Nobody decided Thai pays triple. Frequency chose — this exact loop, run on a mountain of text. Every quirk from the last five episodes is just this counting, at planet scale.
And when the counting stops, what's left behind is strangely humble: a ranked list of merges, frozen in the order they were learned. Next episode: why that list — that list alone — IS the tokenizer.