hawkes.aiKey ideas
- The one idea — Unicode gives é two legal spellings — identical glyphs, different bytes — so "looks identical" means nothing below the glyph layer; normalize before comparing.
- How it is shown — café vs café: two byte sequences, one appearance; the dedup/search/password failures that follow.
- The trap to avoid — Trusting string length or byte equality on un-normalized text.
- What it sets up — Invisible characters are real model inputs.
Your password works on one phone and fails on another. You typed it perfectly both times. Different keyboards can spell the same accent with different bytes.
The one idea
Unicode gives é two legal spellings — identical glyphs, different bytes — so "looks identical" means nothing below the glyph layer; normalize before comparing.
These two words are identical. Pixel for pixel. And they don't match. Café… and café. Same glyphs on screen; different numbers underneath. Unicode gives é two legal spellings: one single fused character — or an ordinary e followed by an invisible "add an accent" character that stacks on top. In bytes: spelling one is two bytes. Spelling two is three — the plain e, then a two-byte combining mark. Same look, different data.
How it works — the demo
café vs café: two byte sequences, one appearance; the dedup/search/password failures that follow.
Any system comparing raw bytes now swears these twins are strangers. Where it bites: search that misses obvious matches. Duplicate "unique" usernames. A password that works on one phone and fails on another — different keyboards emit different spellings. Filenames a Mac and a server can't agree on. The civilized fix is normalization: run every string through a standard rewrite before comparing — compose the twins into one spelling, or decompose them. Two standard forms, NFC and NFD. Boring, one line of code, saves careers. Emoji push it further: one visible thing that's secretly several characters chained by invisible joiners — a skin tone stuck to a thumbs-up, a family assembled from separate people.
The trap to avoid
Trusting string length or byte equality on un-normalized text.
Why it matters — and what’s next
Invisible characters are real model inputs.
"How long is this string?" now has multiple true answers, depending on which layer you ask. Now the AI layer: the tokenizer chops bytes. Two spellings, different bytes — possibly different tokens. The model can treat identical-looking words as different things, or spend extra tokens on the decomposed one. Invisible characters are real inputs. Attackers know it too — that story waits in the security act. The trap, in one line: "looks identical" means nothing below the glyph layer. Normalize at every boundary; never trust a length. Next: the chopper itself — the token, the single most load-bearing idea in this entire series.
Unicode gives the letter é two legal spellings: one fused character, or an e plus an invisible accent. Same pixels on screen. Different data. Real bugs.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
These two words are identical. Pixel for pixel. And they don't match.
Café… and café. Same glyphs on screen; different numbers underneath. Unicode gives é two legal spellings: one single fused character — or an ordinary e followed by an invisible "add an accent" character that stacks on top.
In bytes: spelling one is two bytes. Spelling two is three — the plain e, then a two-byte combining mark. Same look, different data. Any system comparing raw bytes now swears these twins are strangers.
Where it bites: search that misses obvious matches. Duplicate "unique" usernames. A password that works on one phone and fails on another — different keyboards emit different spellings. Filenames a Mac and a server can't agree on.
The civilized fix is normalization: run every string through a standard rewrite before comparing — compose the twins into one spelling, or decompose them. Two standard forms, NFC and NFD. Boring, one line of code, saves careers.
Emoji push it further: one visible thing that's secretly several characters chained by invisible joiners — a skin tone stuck to a thumbs-up, a family assembled from separate people. "How long is this string?" now has multiple true answers, depending on which layer you ask.
Now the AI layer: the tokenizer chops bytes. Two spellings, different bytes — possibly different tokens. The model can treat identical-looking words as different things, or spend extra tokens on the decomposed one. Invisible characters are real inputs. Attackers know it too — that story waits in the security act.
The trap, in one line: "looks identical" means nothing below the glyph layer. Normalize at every boundary; never trust a length. Next: the chopper itself — the token, the single most load-bearing idea in this entire series.