hawkes.aiKey ideas
- The one idea — An image is cut into fixed patches, each embedded as a vector — so a transformer reads a picture as a sequence of tokens, exactly like text.
- How it is shown — A photo gridded into patches; each patch flattened and projected into a vector; the same tower from act two reading the patch-sequence.
- The trap to avoid — Imagining the model "sees" a picture holistically — it reads tiles, and patch size caps what fine detail survives.
- What it sets up — But do patches and words even speak the same language?
When you upload a photo, the AI reads it as a grid of tile-tokens. Anything smaller than a tile, like tiny text, can simply vanish.
The one idea
An image is cut into fixed patches, each embedded as a vector — so a transformer reads a picture as a sequence of tokens, exactly like text.
For a hundred and fifty episodes, this machine has only read text. Now it opens its eyes — and the surprise is that seeing is the trick you already know. The model doesn't gaze at your photo; it slices it into tiles, turns each into a token, and reads the picture as a sentence. Act six: how an image becomes tokens. Step one: the grid. The image is cut into fixed patches — sixteen pixels a side is common — turning one photo into hundreds of tiles. The grid is dumb on purpose: it doesn't find faces or edges, it just chops. A patch of sky and a patch of eyeball are treated identically — squares of pixels, waiting to become tokens. Step two: each patch becomes a vector. Unroll its pixels into a list of numbers, then project that into the same meaning-space arrow episode twenty-one built for words.
How it works — the demo
A photo gridded into patches; each patch flattened and projected into a vector; the same tower from act two reading the patch-sequence.
The patch is now a token — no different in kind from the ones carrying 'cat' or 'the.' The tower doesn't care they came from pixels. Input is input. Step three is nothing new. The patch-tokens get position information — which tile sat where — and flow into the same transformer from act two. Attention relates them: the patch holding an eye attends to the patch holding a mouth; a shadow attends to what cast it. Seeing, mechanically, is attention between tiles. But patches cost, and it shapes everything downstream. Whatever lives inside one patch collapses into a single token — so tiny text, a distant face, fine texture can blur below the grid. Use more, smaller patches and you see sharper — but every patch is a token, and tokens are money, episode one-fifteen. The trap: imagining the model sees your photo the way you do, in one holistic glance.
The trap to avoid
Imagining the model "sees" a picture holistically — it reads tiles, and patch size caps what fine detail survives.
Why it matters — and what’s next
But do patches and words even speak the same language?
It doesn't. It reads tiles, in sequence, and anything finer than a patch is gone before understanding begins. This is why, later this act, multimodal models miscount objects and misread small text. They're reading a low-resolution transcript. So a picture is now a sequence of tokens, readable by machinery you already understand. But a question opens underneath: the token for a patch of dog and the word 'dog' — do they mean the same thing, or just share a tower? For seeing and reading to truly connect, pictures and words need one shared language. Next: CLIP.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
For a hundred and fifty episodes, this machine has only read text. Now it opens its eyes — and the surprise is that seeing is the trick you already know. The model doesn't gaze at your photo; it slices it into tiles, turns each into a token, and reads the picture as a sentence. Act six: how an image becomes tokens.
Step one: the grid. The image is cut into fixed patches — sixteen pixels a side is common — turning one photo into hundreds of tiles. The grid is dumb on purpose: it doesn't find faces or edges, it just chops. A patch of sky and a patch of eyeball are treated identically — squares of pixels, waiting to become tokens.
Step two: each patch becomes a vector. Unroll its pixels into a list of numbers, then project that into the same meaning-space arrow episode twenty-one built for words. The patch is now a token — no different in kind from the ones carrying 'cat' or 'the.' The tower doesn't care they came from pixels. Input is input.
Step three is nothing new. The patch-tokens get position information — which tile sat where — and flow into the same transformer from act two. Attention relates them: the patch holding an eye attends to the patch holding a mouth; a shadow attends to what cast it. Seeing, mechanically, is attention between tiles.
But patches cost, and it shapes everything downstream. Whatever lives inside one patch collapses into a single token — so tiny text, a distant face, fine texture can blur below the grid. Use more, smaller patches and you see sharper — but every patch is a token, and tokens are money, episode one-fifteen.
The trap: imagining the model sees your photo the way you do, in one holistic glance. It doesn't. It reads tiles, in sequence, and anything finer than a patch is gone before understanding begins. This is why, later this act, multimodal models miscount objects and misread small text. They're reading a low-resolution transcript.
So a picture is now a sequence of tokens, readable by machinery you already understand. But a question opens underneath: the token for a patch of dog and the word 'dog' — do they mean the same thing, or just share a tower? For seeing and reading to truly connect, pictures and words need one shared language. Next: CLIP.