hawkes.aiKey ideas
- The one idea — You can't embed a whole document as one vector — you split it into chunks, and chunk size, boundaries, and overlap largely determine RAG quality (too big buries the answer; too small scatters it).
- How it is shown — A document cut cleanly on section boundaries with slight overlap (good retrieval) versus dumped through naive fixed-size splitting that cuts mid-sentence and mid-table (broken retrieval).
- The trap to avoid — Naive fixed-size splitting, then blaming the model when RAG is bad — the retrieval/chunking layer is usually the culprit.
- What it sets up — Even with great retrieval and chunking, RAG can still hallucinate.
When RAG gives bad answers, everyone blames the model. Usually the real culprit is more boring: how you chopped the documents before the AI ever saw them.
The one idea
You can't embed a whole document as one vector — you split it into chunks, and chunk size, boundaries, and overlap largely determine RAG quality (too big buries the answer; too small scatters it).
Here's the least glamorous, most important secret in RAG. You can't embed a whole document as one vector — you split it into chunks first. And how you split it quietly decides whether retrieval works or falls apart. Why split? Two reasons. Embeddings capture a passage, not a book — cram a whole document into one vector and its many ideas smear into a meaningless average. And you retrieve chunks, not whole files. So every document gets cut before it's embedded. Now the choices. Chunk size: a paragraph, a section, a page?
How it works — the demo
A document cut cleanly on section boundaries with slight overlap (good retrieval) versus dumped through naive fixed-size splitting that cuts mid-sentence and mid-table (broken retrieval).
Boundaries: split on sentences and sections, not in the middle of a thought. Overlap: repeat a little text between chunks, so an idea straddling the cut isn't lost. Size, boundaries, overlap. Get it wrong and RAG breaks quietly. Too big, and the one relevant sentence drowns in noise, muddying its vector. Too small, and the answer scatters across pieces that don't all get retrieved. Split a table down the middle, and both halves become nonsense. Good chunking respects the document's structure. It cuts where meaning naturally breaks, keeps related things together — a whole table, a whole list — and overlaps just enough for continuity. Unglamorous, and where most RAG quality is won or lost.
The trap to avoid
Naive fixed-size splitting, then blaming the model when RAG is bad — the retrieval/chunking layer is usually the culprit.
Why it matters — and what’s next
Even with great retrieval and chunking, RAG can still hallucinate.
The trap: naive fixed-size splitting — every five hundred characters, mid-sentence, mid-table — then blaming the model when answers are bad. The model is usually fine. The chunking and retrieval layer is the culprit. Fix the cuts before you swap the model. So chunking is the unsexy key to good RAG: split with care, respect structure, overlap a little. Fix this before touching the model. But even with perfect chunks, RAG can still confidently make things up. Why? Next: why RAG still hallucinates.
Cut your documents too big and the answer gets buried in noise. Too small, and it's scattered across pieces. This unsexy splitting decision makes or breaks RAG.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Here's the least glamorous, most important secret in RAG. You can't embed a whole document as one vector — you split it into chunks first. And how you split it quietly decides whether retrieval works or falls apart.
Why split? Two reasons. Embeddings capture a passage, not a book — cram a whole document into one vector and its many ideas smear into a meaningless average. And you retrieve chunks, not whole files. So every document gets cut before it's embedded.
Now the choices. Chunk size: a paragraph, a section, a page? Boundaries: split on sentences and sections, not in the middle of a thought. Overlap: repeat a little text between chunks, so an idea straddling the cut isn't lost. Size, boundaries, overlap.
Get it wrong and RAG breaks quietly. Too big, and the one relevant sentence drowns in noise, muddying its vector. Too small, and the answer scatters across pieces that don't all get retrieved. Split a table down the middle, and both halves become nonsense.
Good chunking respects the document's structure. It cuts where meaning naturally breaks, keeps related things together — a whole table, a whole list — and overlaps just enough for continuity. Unglamorous, and where most RAG quality is won or lost.
The trap: naive fixed-size splitting — every five hundred characters, mid-sentence, mid-table — then blaming the model when answers are bad. The model is usually fine. The chunking and retrieval layer is the culprit. Fix the cuts before you swap the model.
So chunking is the unsexy key to good RAG: split with care, respect structure, overlap a little. Fix this before touching the model. But even with perfect chunks, RAG can still confidently make things up. Why? Next: why RAG still hallucinates.