hawkes.aiKey ideas
- The one idea — Each token carries one running vector through the whole machine; every component reads from it and ADDS its contribution back — annotation, never erasure.
- How it is shown — The conveyor: modules stamping additions onto a passing vector; early findings surviving to the top.
- The trap to avoid — Imagining layers transforming data destructively hand-to-hand — the stream accumulates (x + f(x)); nothing is overwritten.
- What it sets up — Attention = how OTHER tokens' conveyors get read.
Inside the machine, nothing ever gets erased. Every layer adds its notes on top of everything before — and one humble plus sign made deep networks possible.
The one idea
Each token carries one running vector through the whole machine; every component reads from it and ADDS its contribution back — annotation, never erasure.
There's a single conveyor belt running through the entire machine, and every layer edits it. Meet the residual stream. Every token gets its own stream. It starts life as the token's embedding — the point from the map act — and then it flows upward through every floor of the model, one continuous vector per token, bottom to top. Now the crucial mechanic. Each component doesn't take the vector, transform it, and hand off something new. It reads the stream, computes its contribution, and adds it on top — the original, plus the new. Read, compute, add. In symbols: x plus f of x. Every floor annotates. No floor erases. Why adding?
How it works — the demo
The conveyor: modules stamping additions onto a passing vector; early findings surviving to the top.
Two reasons. Preservation: a finding written at floor two survives, untouched, beneath everything stacked later — the top of the model can still use what the bottom noticed. And trainability: that unbroken additive path is the highway that lets learning signals reach the deep floors at all. It's the design choice that made very deep networks possible. One humble plus sign, carrying the era. The friendliest reading: it's a shared whiteboard. Every module gets to read all the notes so far, add its own, and erase nothing. That's why "working memory" fits — with one caution: this memory lives for one token's journey through one forward pass. It isn't memory across your conversation. That distinction returns, loudly, in the agent act. And here's where episode twenty-five pays off: the stream's width is those thousands of dimensions — and it's finite. Every module in the model communicates with every later module through this one shared channel.
The trap to avoid
Imagining layers transforming data destructively hand-to-hand — the stream accumulates (x + f(x)); nothing is overwritten.
Why it matters — and what’s next
Attention = how OTHER tokens' conveyors get read.
Interpretability researchers treat it literally as a communication bus with limited bandwidth, and later, when we read a model's mind, the stream is exactly where we'll point the instruments. The trap: picturing floors as a relay where each output replaces the last. Nothing is replaced. The vector at the top is the original embedding plus every floor's contribution, summed. When a later episode says a component "writes to the stream" — this is the machinery. Accumulation, all the way up. One thing is conspicuously missing. Each token's conveyor is alone — "bank" is riding its stream with no idea whether "river" is riding nearby. The mechanism that lets streams read each other is the most famous idea in modern AI, and it's next. The problem first — then the machinery.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
There's a single conveyor belt running through the entire machine, and every layer edits it. Meet the residual stream.
Every token gets its own stream. It starts life as the token's embedding — the point from the map act — and then it flows upward through every floor of the model, one continuous vector per token, bottom to top.
Now the crucial mechanic. Each component doesn't take the vector, transform it, and hand off something new. It reads the stream, computes its contribution, and adds it on top — the original, plus the new. Read, compute, add. In symbols: x plus f of x. Every floor annotates. No floor erases.
Why adding? Two reasons. Preservation: a finding written at floor two survives, untouched, beneath everything stacked later — the top of the model can still use what the bottom noticed. And trainability: that unbroken additive path is the highway that lets learning signals reach the deep floors at all. It's the design choice that made very deep networks possible. One humble plus sign, carrying the era.
The friendliest reading: it's a shared whiteboard. Every module gets to read all the notes so far, add its own, and erase nothing. That's why "working memory" fits — with one caution: this memory lives for one token's journey through one forward pass. It isn't memory across your conversation. That distinction returns, loudly, in the agent act.
And here's where episode twenty-five pays off: the stream's width is those thousands of dimensions — and it's finite. Every module in the model communicates with every later module through this one shared channel. Interpretability researchers treat it literally as a communication bus with limited bandwidth, and later, when we read a model's mind, the stream is exactly where we'll point the instruments.
The trap: picturing floors as a relay where each output replaces the last. Nothing is replaced. The vector at the top is the original embedding plus every floor's contribution, summed. When a later episode says a component "writes to the stream" — this is the machinery. Accumulation, all the way up.
One thing is conspicuously missing. Each token's conveyor is alone — "bank" is riding its stream with no idea whether "river" is riding nearby. The mechanism that lets streams read each other is the most famous idea in modern AI, and it's next. The problem first — then the machinery.