hawkes.aiKey ideas
- The one idea — Glitch tokens are menu slots whose table rows never got trained — feeding one in injects noise from nowhere, and the model breaks.
- How it is shown — The SolidGoldMagikarp story: a Reddit username that earned a menu slot, then vanished from the model's training data.
- The trap to avoid — Assuming "it's in the vocabulary" means "it was learned" — the tokenizer/model seam is where the ghosts live.
- What it sets up — Some tokens aren't words at all — they're commands.
In 2023, researchers found a Reddit username the AI could not repeat back. Ask it to say 'SolidGoldMagikarp' and it rambled, dodged, sometimes turned hostile.
The one idea
Glitch tokens are menu slots whose table rows never got trained — feeding one in injects noise from nowhere, and the model breaks.
There are words that break the AI. Nobody put them there on purpose. In 2023, researchers cataloguing GPT's vocabulary found tokens that behaved like cursed objects. Ask the model to simply repeat " SolidGoldMagikarp," and it… couldn't. It answered about something else, rambled, sometimes turned hostile. A word the machine could not say. The cause is the seam you already know: the tokenizer and the model are separate artifacts, built from different data piles. The tokenizer counted one internet. The model trained on a filtered, different internet. Anything frequent in the first pile but absent from the second gets a menu slot — and no education.
How it works — the demo
The SolidGoldMagikarp story: a Reddit username that earned a menu slot, then vanished from the model's training data.
" SolidGoldMagikarp" was a Reddit username from an obsessive counting forum — so frequent in the tokenizer's pile that it earned its own menu slot. Then the model's training pile filtered that forum out. The room was built. Nobody ever moved in. Mechanically: every token's meaning lives in a table row the model learns by seeing the token used. A token never seen keeps its birth values — random static. Feed it in, and you inject a coordinate from nowhere: a point in meaning-space no training ever touched. The model must still respond — it always responds — so the static ricochets through every layer, and out comes confabulation, evasion, sometimes insults. Not a hack. Not a secret message.
The trap to avoid
Assuming "it's in the vocabulary" means "it was learned" — the tokenizer/model seam is where the ghosts live.
Why it matters — and what’s next
Some tokens aren't words at all — they're commands.
An unfurnished room with the lights off. The trap: assuming "it's in the vocabulary" means "it was learned." The vocabulary is a contract written by one system and honored by another — and the gaps between them are where the ghosts live. Modern pipelines hunt these now, aligning the two data piles. But new vocabularies keep shipping with quiet, undertrained corners. People still go ghost-hunting. They still find ghosts. So the vocabulary has empty rooms — accidents of the seam. But it also has rooms you were never meant to enter on purpose: tokens that aren't words at all. They're commands. Next: the secret control characters running every chat you've ever had.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
There are words that break the AI. Nobody put them there on purpose.
In 2023, researchers cataloguing GPT's vocabulary found tokens that behaved like cursed objects. Ask the model to simply repeat " SolidGoldMagikarp," and it… couldn't. It answered about something else, rambled, sometimes turned hostile. A word the machine could not say.
The cause is the seam you already know: the tokenizer and the model are separate artifacts, built from different data piles. The tokenizer counted one internet. The model trained on a filtered, different internet. Anything frequent in the first pile but absent from the second gets a menu slot — and no education.
" SolidGoldMagikarp" was a Reddit username from an obsessive counting forum — so frequent in the tokenizer's pile that it earned its own menu slot. Then the model's training pile filtered that forum out. The room was built. Nobody ever moved in.
Mechanically: every token's meaning lives in a table row the model learns by seeing the token used. A token never seen keeps its birth values — random static. Feed it in, and you inject a coordinate from nowhere: a point in meaning-space no training ever touched.
The model must still respond — it always responds — so the static ricochets through every layer, and out comes confabulation, evasion, sometimes insults. Not a hack. Not a secret message. An unfurnished room with the lights off.
The trap: assuming "it's in the vocabulary" means "it was learned." The vocabulary is a contract written by one system and honored by another — and the gaps between them are where the ghosts live. Modern pipelines hunt these now, aligning the two data piles. But new vocabularies keep shipping with quiet, undertrained corners. People still go ghost-hunting. They still find ghosts.
So the vocabulary has empty rooms — accidents of the seam. But it also has rooms you were never meant to enter on purpose: tokens that aren't words at all. They're commands. Next: the secret control characters running every chat you've ever had.