hawkes.aiKey ideas
- The one idea — Tokenizers are trained mostly on English, so other languages pay more tokens per thought — in money, latency, and context space.
- How it is shown — One meaning, three languages, three token meters racing apart.
- The trap to avoid — Reading multilingual benchmark gaps as pure ability — part of the gap is arithmetic set on the tokenizer's training day.
- What it sets up — Frequency decided the menu.
Who decided Thai speakers pay triple for AI? Nobody, and that's the problem. A frequency count on mostly-English text set the prices before the model ever trained.
The one idea
Tokenizers are trained mostly on English, so other languages pay more tokens per thought — in money, latency, and context space.
Same sentence. Same meaning. Triple the price. This is the token tax. You know the menu: tens of thousands of chunks, learned by counting what's frequent. Here's the detail that decides who pays: the pile it counted was mostly English. So English words earned big, efficient chunks — whole words riding as single tokens. Now watch Thai, or Burmese, or Amharic hit the same chopper. The menu barely knows them. Words shatter into syllables, characters, sometimes raw bytes. Where English pays three tokens, another language can pay nine for the very same thought.
How it works — the demo
One meaning, three languages, three token meters racing apart.
Why? The tokenizer isn't malicious — it's a frequency machine. Chunks get earned by repetition, and languages underrepresented in the pile never earn their merges. The unfairness was baked in before the model trained a single weight. And tokens are the meter for everything. So the same thought in Thai costs more money, streams slower, and fills the model's working memory up to three times faster. A document "too long for the context window" might be a short document, badly chopped. It reaches quality too. A word delivered as one familiar chunk arrives pre-digested. A word delivered as seven shards makes the model reassemble meaning from fragments. Some of the gap between a model's English and its Thai isn't intelligence.
The trap to avoid
Reading multilingual benchmark gaps as pure ability — part of the gap is arithmetic set on the tokenizer's training day.
Why it matters — and what’s next
Frequency decided the menu.
It's the tax. The trap: reading multilingual benchmarks as pure ability. Part of what you're measuring is arithmetic — decided years earlier, the day someone chose the tokenizer's training mix. The fix is real and underway: bigger vocabularies, deliberately multilingual mixes, smarter byte handling. The tax is shrinking. But any fixed menu plays favorites with someone — the only question is who, and by how much. Next: you build one of these choppers yourself — from scratch, in three minutes. Count, merge, repeat. Once you've built one, nothing about tokens will ever be mysterious again.
When a model looks worse in Thai than English, you assume it's less intelligent there. Part of that gap is arithmetic: the same thought costs up to triple the tokens.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Same sentence. Same meaning. Triple the price. This is the token tax.
You know the menu: tens of thousands of chunks, learned by counting what's frequent. Here's the detail that decides who pays: the pile it counted was mostly English.
So English words earned big, efficient chunks — whole words riding as single tokens. Now watch Thai, or Burmese, or Amharic hit the same chopper. The menu barely knows them. Words shatter into syllables, characters, sometimes raw bytes. Where English pays three tokens, another language can pay nine for the very same thought.
Why? The tokenizer isn't malicious — it's a frequency machine. Chunks get earned by repetition, and languages underrepresented in the pile never earn their merges. The unfairness was baked in before the model trained a single weight.
And tokens are the meter for everything. So the same thought in Thai costs more money, streams slower, and fills the model's working memory up to three times faster. A document "too long for the context window" might be a short document, badly chopped.
It reaches quality too. A word delivered as one familiar chunk arrives pre-digested. A word delivered as seven shards makes the model reassemble meaning from fragments. Some of the gap between a model's English and its Thai isn't intelligence. It's the tax.
The trap: reading multilingual benchmarks as pure ability. Part of what you're measuring is arithmetic — decided years earlier, the day someone chose the tokenizer's training mix.
The fix is real and underway: bigger vocabularies, deliberately multilingual mixes, smarter byte handling. The tax is shrinking. But any fixed menu plays favorites with someone — the only question is who, and by how much.
Next: you build one of these choppers yourself — from scratch, in three minutes. Count, merge, repeat. Once you've built one, nothing about tokens will ever be mysterious again.