hawkes.aiKey ideas
- The one idea — Huge context windows tempt you to "just paste everything," but bigger context isn't free (costs more, slower), isn't perfect (models attend worse to the middle — "lost in the middle"), and is still finite — so it complements retrieval, not replaces it.
- How it is shown — A giant window swallowing a whole corpus, but the middle fading (a buried fact missed), the cost meter spiking, and a still-finite edge — versus retrieval placing the right slice up front.
- The trap to avoid — "The window is huge, so just paste everything and skip retrieval" — costly, slower, and less accurate than giving it the right slice.
- What it sets up — So when do you use RAG vs long context vs something else?
Paste a million tokens into a model and it reads the middle worst of all. Lost in the middle is real — a bigger window isn't a perfect one.
The one idea
Huge context windows tempt you to "just paste everything," but bigger context isn't free (costs more, slower), isn't perfect (models attend worse to the middle — "lost in the middle"), and is still finite — so it complements retrieval, not replaces it.
Context windows have exploded — millions of tokens now. Which tempts a seductive shortcut: skip retrieval entirely, just paste your whole knowledge base in and let the model sort it out. It feels like RAG is obsolete. It isn't. Infinite context is an illusion, and it breaks in three ways. First, it's not free. Every token is paid for, and long contexts are expensive per call — recall, tokens are money. Worse, the compute of attention grows faster than linearly with length, so a giant context is costly and slow on every single request. Pasting everything means paying for everything, every time. Second, quality drops.
How it works — the demo
A giant window swallowing a whole corpus, but the middle fading (a buried fact missed), the cost meter spiking, and a still-finite edge — versus retrieval placing the right slice up front.
Models reliably use the beginning and the end of a long input, but attend worse to the middle. A fact buried halfway through a million tokens can be missed entirely. It's called lost in the middle, and it's a real, measured effect — not a bug you can prompt away. So pasting everything is worse than it looks. You pay more, wait longer, and the model may overlook the one passage that mattered — the very passage retrieval would have placed right up front, small and sharp, where attention is strongest. Less, well-chosen, often beats more, dumped in. Third, it's still finite. However huge the window, there's always a bigger corpus — more documents than fit, more history than the context can hold. Infinite is a figure of speech. The window is large now, but it is never actually unlimited, and real knowledge bases keep outgrowing it.
The trap to avoid
"The window is huge, so just paste everything and skip retrieval" — costly, slower, and less accurate than giving it the right slice.
Why it matters — and what’s next
So when do you use RAG vs long context vs something else?
The trap: the window is huge, so just paste everything and skip retrieval. It's costly, slower, and less accurate than handing the model the right slice. Big context genuinely helps — it complements retrieval, letting you pass more when you need to. But it doesn't replace it. Choosing the right knowledge still wins. So a bigger window isn't free or perfect: it costs more, slows down, and loses things in the middle. Retrieval still earns its keep. Which raises the real question — when should you reach for RAG, when for long context, and when for something else entirely? Next: RAG versus long context versus fine-tune.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Context windows have exploded — millions of tokens now. Which tempts a seductive shortcut: skip retrieval entirely, just paste your whole knowledge base in and let the model sort it out. It feels like RAG is obsolete. It isn't. Infinite context is an illusion, and it breaks in three ways.
First, it's not free. Every token is paid for, and long contexts are expensive per call — recall, tokens are money. Worse, the compute of attention grows faster than linearly with length, so a giant context is costly and slow on every single request. Pasting everything means paying for everything, every time.
Second, quality drops. Models reliably use the beginning and the end of a long input, but attend worse to the middle. A fact buried halfway through a million tokens can be missed entirely. It's called lost in the middle, and it's a real, measured effect — not a bug you can prompt away.
So pasting everything is worse than it looks. You pay more, wait longer, and the model may overlook the one passage that mattered — the very passage retrieval would have placed right up front, small and sharp, where attention is strongest. Less, well-chosen, often beats more, dumped in.
Third, it's still finite. However huge the window, there's always a bigger corpus — more documents than fit, more history than the context can hold. Infinite is a figure of speech. The window is large now, but it is never actually unlimited, and real knowledge bases keep outgrowing it.
The trap: the window is huge, so just paste everything and skip retrieval. It's costly, slower, and less accurate than handing the model the right slice. Big context genuinely helps — it complements retrieval, letting you pass more when you need to. But it doesn't replace it. Choosing the right knowledge still wins.
So a bigger window isn't free or perfect: it costs more, slows down, and loses things in the middle. Retrieval still earns its keep. Which raises the real question — when should you reach for RAG, when for long context, and when for something else entirely? Next: RAG versus long context versus fine-tune.