hawkes.aiKey ideas
- The one idea — The memory hierarchy — six levels, each bigger and slower — and moving data, not doing math, is the real cost.
- How it is shown — A tensor commuting through nested rooms, travel time ballooning at each threshold.
- The trap to avoid — Judging hardware or code by operation counts — identical math can differ 10× by traffic pattern alone.
- What it sets up — Never let the big grid leave the small room.
Here's the dirty secret of computing: the math is basically free. Moving the numbers to where the math happens — that's what you're actually paying for.
The one idea
The memory hierarchy — six levels, each bigger and slower — and moving data, not doing math, is the real cost.
Inside your machine there are six rooms. The smallest holds a handful of numbers and delivers them instantly. The largest holds everything you own — at postal speed. All of AI performance is a commute between these rooms. The inner three live on the chip itself. Registers: numbers in the cores' hands, distance zero. The on-chip scratchpad: a workbench of fast memory, kilobytes, reachable in a step. The shared shelf ringing the die: the last stop before leaving home. Tiny jewelry-sized rooms — everything in them moves at the speed of the math. The outer three hold the bulk. The warehouse — VRAM — where the model lives, terabytes per second but already a commute.
How it works — the demo
A tensor commuting through nested rooms, travel time ballooning at each threshold.
Then across a narrow bridge: system memory, roomier and painfully farther. Then disk: another city entirely. Each threshold outward: roughly ten to a hundred times slower. Same numbers, same math waiting — all that changed is where they stood. Which reveals the law under this whole act: using a number is nearly free; moving it is what costs. A single fetch from the warehouse burns energy worth hundreds of multiplications — time tells the same story. Modern chips math far faster than they can eat. The bottleneck of this era isn't arithmetic. It's logistics. So the craft of fast AI is commute-planning. Haul in bulk.
The trap to avoid
Judging hardware or code by operation counts — identical math can differ 10× by traffic pattern alone.
Why it matters — and what’s next
Never let the big grid leave the small room.
Keep the hot data on the workbench. Finish every calculation that touches a delivery before releasing it. The best engineers here think like logistics managers — and two programs with identical math can differ ten-fold on traffic pattern alone. The trap: judging by operation counts. Two implementations, identical math, and one runs ten times faster — because the ledger that matters is bytes moved, not operations performed. When you hear that some optimization made AI dramatically faster, ask the professional's question first: what did it stop moving? Which sets up a legend. The biggest thing the tower ever builds is attention's giant grid — every word against every word, so large it lived in the warehouse, hauled out and back at brutal cost. In twenty-twenty-two, someone asked the logistics question: what if it never left the small room? Next: FlashAttention.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Inside your machine there are six rooms. The smallest holds a handful of numbers and delivers them instantly. The largest holds everything you own — at postal speed. All of AI performance is a commute between these rooms.
The inner three live on the chip itself. Registers: numbers in the cores' hands, distance zero. The on-chip scratchpad: a workbench of fast memory, kilobytes, reachable in a step. The shared shelf ringing the die: the last stop before leaving home. Tiny jewelry-sized rooms — everything in them moves at the speed of the math.
The outer three hold the bulk. The warehouse — VRAM — where the model lives, terabytes per second but already a commute. Then across a narrow bridge: system memory, roomier and painfully farther. Then disk: another city entirely. Each threshold outward: roughly ten to a hundred times slower. Same numbers, same math waiting — all that changed is where they stood.
Which reveals the law under this whole act: using a number is nearly free; moving it is what costs. A single fetch from the warehouse burns energy worth hundreds of multiplications — time tells the same story. Modern chips math far faster than they can eat. The bottleneck of this era isn't arithmetic. It's logistics.
So the craft of fast AI is commute-planning. Haul in bulk. Keep the hot data on the workbench. Finish every calculation that touches a delivery before releasing it. The best engineers here think like logistics managers — and two programs with identical math can differ ten-fold on traffic pattern alone.
The trap: judging by operation counts. Two implementations, identical math, and one runs ten times faster — because the ledger that matters is bytes moved, not operations performed. When you hear that some optimization made AI dramatically faster, ask the professional's question first: what did it stop moving?
Which sets up a legend. The biggest thing the tower ever builds is attention's giant grid — every word against every word, so large it lived in the warehouse, hauled out and back at brutal cost. In twenty-twenty-two, someone asked the logistics question: what if it never left the small room? Next: FlashAttention.