hawkes.aiKey ideas
- The one idea — Tokens/sec is the speed metric — but "fast" depends entirely on who's consuming the tokens: a reader, a skimmer, or a machine.
- How it is shown — The same stream at 8, 40, and 300 tokens/sec against three consumers: a reader (saturated early), a code-skimmer, an agent pipeline (starving until the top).
- The trap to avoid — Speed claims without workload context — and forgetting time-to-first-token is a separate number.
- What it sets up — The meter doesn't just measure — it BILLS.
At forty tokens a second, a human reader is fully satisfied. An AI agent chewing that same stream is starving — it wanted three hundred. Fast for whom?
The one idea
Tokens/sec is the speed metric — but "fast" depends entirely on who's consuming the tokens: a reader, a skimmer, or a machine.
Every spec sheet in this act ends at one consumer number: tokens per second — the reply's streaming rate. But "fast" isn't a number; it's a fit between the stream and whoever's drinking from it. Same speed, three verdicts. Let's calibrate yours. Consumer one: a human, reading. People read a few words a second — five to ten tokens per second already outruns you; twenty feels instant. Read-along chat saturates early: past reading pace, velocity just piles up unread text. The meter can brag; your eyes can't cash it. Consumer two: a skimmer generating code or documents — waiting for the block to land.
How it works — the demo
The same stream at 8, 40, and 300 tokens/sec against three consumers: a reader (saturated early), a code-skimmer, an agent pipeline (starving until the top).
Fifty to a hundred tokens per second is felt time. Consumer three changes the game: pipelines — chains where one call's output feeds the next. No human bottleneck; every second compounds across steps. Machines are never satisfied — and they own a later act. And there's a second clock: time to first token — the silence before anything appears. Different quantity, different causes: queueing, episode one-ten's batch beat, and prompt-reading — episode ninety-two's prefill, why long documents stretch the silence. Feel matters twice: how soon it starts, then how fast it flows. Judge both. So read claims like an engineer.
The trap to avoid
Speed claims without workload context — and forgetting time-to-first-token is a separate number.
Why it matters — and what’s next
The meter doesn't just measure — it BILLS.
A bare tokens-per-second number is marketing until it answers: what batch size, what prompt length, what first-token wait? A batch-of-one hero run — episode one-oh-nine's private lane — tells you little about rush hour. The habit is one question long: fast for whom, under what load? The trap: speed judged without naming consumer and load is a number in a vacuum — fast for reading is slow for pipelines; a hero benchmark isn't your rush hour. And that meter doesn't just measure the stream — it BILLS it, and bills the two directions very differently. Why does the reply cost several times the prompt? Next — and you already know the physics.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:15 of narration
Every spec sheet in this act ends at one consumer number: tokens per second — the reply's streaming rate. But "fast" isn't a number; it's a fit between the stream and whoever's drinking from it. Same speed, three verdicts. Let's calibrate yours.
Consumer one: a human, reading. People read a few words a second — five to ten tokens per second already outruns you; twenty feels instant. Read-along chat saturates early: past reading pace, velocity just piles up unread text. The meter can brag; your eyes can't cash it.
Consumer two: a skimmer generating code or documents — waiting for the block to land. Fifty to a hundred tokens per second is felt time. Consumer three changes the game: pipelines — chains where one call's output feeds the next. No human bottleneck; every second compounds across steps. Machines are never satisfied — and they own a later act.
And there's a second clock: time to first token — the silence before anything appears. Different quantity, different causes: queueing, episode one-ten's batch beat, and prompt-reading — episode ninety-two's prefill, why long documents stretch the silence. Feel matters twice: how soon it starts, then how fast it flows. Judge both.
So read claims like an engineer. A bare tokens-per-second number is marketing until it answers: what batch size, what prompt length, what first-token wait? A batch-of-one hero run — episode one-oh-nine's private lane — tells you little about rush hour. The habit is one question long: fast for whom, under what load?
The trap: speed judged without naming consumer and load is a number in a vacuum — fast for reading is slow for pipelines; a hero benchmark isn't your rush hour. And that meter doesn't just measure the stream — it BILLS it, and bills the two directions very differently. Why does the reply cost several times the prompt? Next — and you already know the physics.