hawkes.aiKey ideas
- The one idea — A video-input model doesn't watch continuously — it samples a handful of frames, turns each into patch-tokens, and reasons over them as a sequence of stills in one shared stream.
- How it is shown — A ten-minute clip reduced to a few sampled snapshots fed to a VLM; it nails the gist and misses fast events that fell between samples.
- The trap to avoid — Assuming it watched every frame — it flips through snapshots, so it misses things between them and is weak on precise timing and counts over time.
- What it sets up — Vision and video done — now the last sense in this act: SOUND.
Ask AI about your ten-minute video and it answers from a handful of snapshots — confidently, even about the parts that fell between them.
The one idea
A video-input model doesn't watch continuously — it samples a handful of frames, turns each into patch-tokens, and reasons over them as a sequence of stills in one shared stream.
Hand an AI a video, ask about it, and it feels like it watched the whole thing. It didn't. It samples a handful of frames — snapshots — and reasons over just those. Most of your clip, it never sees. That one fact tells you what video AI is good at and where it'll miss. A video is just many frames. The model samples some — say one a second — and turns each into patch-tokens, as in episode one-fifty. Those tokens flow into the shared residual stream from episode one-fifty-two. Understanding video is reading a sequence of stills. What does it do well? The gist. From a handful of snapshots it reads the setting, objects, people, broad action, the arc.
How it works — the demo
A ten-minute clip reduced to a few sampled snapshots fed to a VLM; it nails the gist and misses fast events that fell between samples.
Ask 'what happens in this video?' and it answers cleanly — most videos' story is legible from a few frames. Summarizing, identifying the scene: sparse sampling is plenty. What does it miss? Anything between the samples. A fast event that starts and ends between two snapshots is invisible — a quick gesture, a ball thrown and caught. Precise timing, exact counts over time, fleeting moments: all fall through the gaps. Longer video, sparser sampling, more slips past. Why not feed every frame? Cost. Each frame is a pile of tokens, from episode one-fifty, and a video has thousands. Feed them all and you blow up context and compute — tokens are money, episode one-fifteen, at scale. Sampling is the necessary compression.
The trap to avoid
Assuming it watched every frame — it flips through snapshots, so it misses things between them and is weak on precise timing and counts over time.
Why it matters — and what’s next
Vision and video done — now the last sense in this act: SOUND.
More frames, more detail, more cost. The trap: assuming it watched every second. It saw a slideshow — and answers confidently about moments it never saw, filling gaps with a guess. That confidence over the gaps is the danger zone. For anything precise: sample denser, clip the segment, or tell it where to look. Trust the gist; verify the detail. Step back: this act — still images made and read, video generated and understood — all built on tokens in one shared stream. One sense is still dark: sound. And it yields to the same trick: slice sound into tokens, pour them in, and a model can listen and speak. Next group: speech in, speech out.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Hand an AI a video, ask about it, and it feels like it watched the whole thing. It didn't. It samples a handful of frames — snapshots — and reasons over just those. Most of your clip, it never sees. That one fact tells you what video AI is good at and where it'll miss.
A video is just many frames. The model samples some — say one a second — and turns each into patch-tokens, as in episode one-fifty. Those tokens flow into the shared residual stream from episode one-fifty-two. Understanding video is reading a sequence of stills.
What does it do well? The gist. From a handful of snapshots it reads the setting, objects, people, broad action, the arc. Ask 'what happens in this video?' and it answers cleanly — most videos' story is legible from a few frames. Summarizing, identifying the scene: sparse sampling is plenty.
What does it miss? Anything between the samples. A fast event that starts and ends between two snapshots is invisible — a quick gesture, a ball thrown and caught. Precise timing, exact counts over time, fleeting moments: all fall through the gaps. Longer video, sparser sampling, more slips past.
Why not feed every frame? Cost. Each frame is a pile of tokens, from episode one-fifty, and a video has thousands. Feed them all and you blow up context and compute — tokens are money, episode one-fifteen, at scale. Sampling is the necessary compression. More frames, more detail, more cost.
The trap: assuming it watched every second. It saw a slideshow — and answers confidently about moments it never saw, filling gaps with a guess. That confidence over the gaps is the danger zone. For anything precise: sample denser, clip the segment, or tell it where to look. Trust the gist; verify the detail.
Step back: this act — still images made and read, video generated and understood — all built on tokens in one shared stream. One sense is still dark: sound. And it yields to the same trick: slice sound into tokens, pour them in, and a model can listen and speak. Next group: speech in, speech out.