hawkes.aiKey ideas
- The one idea — Three ways to give a model knowledge or behavior: RAG (retrieve relevant text per query — for large/changing/private knowledge), long context (paste it all in — for small self-contained tasks), fine-tuning (adjust weights — for behavior/style, NOT facts).
- How it is shown — The same need routed three ways — a changing knowledge base → RAG; one self-contained contract → long context; a consistent house style → fine-tuning; with fine-tuning shown failing at facts.
- The trap to avoid — Fine-tuning to "add knowledge/facts" — it teaches behavior/patterns, not reliable up-to-date facts; use RAG for facts.
- What it sets up — RAG and context are per-query — how does an agent remember across a session?
Fine-tuning won't reliably teach your model new facts. It teaches behavior and style. Teams burn real money mixing this up — here's which tool fits which job.
The one idea
Three ways to give a model knowledge or behavior: RAG (retrieve relevant text per query — for large/changing/private knowledge), long context (paste it all in — for small self-contained tasks), fine-tuning (adjust weights — for behavior/style, NOT facts).
There are three ways to give a model knowledge or new behavior, and mixing them up is one of the most expensive mistakes teams make. RAG, long context, and fine-tuning. They look interchangeable and they are not — they solve different problems, and reaching for the wrong one wastes real money and time. Let's separate them. RAG keeps the knowledge external and retrieves the relevant slice per query. It's best for large, changing, or private collections — your documents, a growing knowledge base. It cites its sources, and updating is trivial: change the documents, and the next answer reflects it, no model change at all. Its one weakness is that retrieval can miss the right passage. Long context pastes the material straight into the prompt for this one call. It's best when the material is small and self-contained — a single contract, one file, this meeting's notes.
How it works — the demo
The same need routed three ways — a changing knowledge base → RAG; one self-contained contract → long context; a consistent house style → fine-tuning; with fine-tuning shown failing at facts.
Simple, no pipeline, no storage. But it's costly, slower, prone to lost-in-the-middle, and it persists nothing: when the call ends, the knowledge is gone. Fine-tuning actually adjusts the model's weights on your data. It's the only one of the three that changes the model itself. And it's best for teaching behavior — a consistent style, a tone, an output format, a narrow skill. It bakes in how to respond, making that style intrinsic to the model instead of something you supply every call. And here's the rule that saves the most pain: fine-tuning is bad for facts. It doesn't reliably memorize specific facts, and you can't update it without retraining, so its knowledge goes stale. Use RAG or long context for knowledge. Use fine-tuning for behavior.
The trap to avoid
Fine-tuning to "add knowledge/facts" — it teaches behavior/patterns, not reliable up-to-date facts; use RAG for facts.
Why it matters — and what’s next
RAG and context are per-query — how does an agent remember across a session?
Keep those two straight and you avoid the most costly mistake in the field. The trap: fine-tuning to add facts or fresh knowledge. That's the classic, costly error — you spend the time and money to retrain, and it still gets specific details wrong, and it can't be updated when things change. Facts belong in retrieval; fine-tuning shapes style and skill. Match the tool to the problem and the whole system gets simpler. So: knowledge that's big, private, or changing goes to RAG; small and self-contained goes in the context; behavior and style go to fine-tuning — and you often combine them. But all three are per-query. Nothing carries across a whole session. So how does an agent remember? Next: types of agent memory.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
There are three ways to give a model knowledge or new behavior, and mixing them up is one of the most expensive mistakes teams make. RAG, long context, and fine-tuning. They look interchangeable and they are not — they solve different problems, and reaching for the wrong one wastes real money and time. Let's separate them.
RAG keeps the knowledge external and retrieves the relevant slice per query. It's best for large, changing, or private collections — your documents, a growing knowledge base. It cites its sources, and updating is trivial: change the documents, and the next answer reflects it, no model change at all. Its one weakness is that retrieval can miss the right passage.
Long context pastes the material straight into the prompt for this one call. It's best when the material is small and self-contained — a single contract, one file, this meeting's notes. Simple, no pipeline, no storage. But it's costly, slower, prone to lost-in-the-middle, and it persists nothing: when the call ends, the knowledge is gone.
Fine-tuning actually adjusts the model's weights on your data. It's the only one of the three that changes the model itself. And it's best for teaching behavior — a consistent style, a tone, an output format, a narrow skill. It bakes in how to respond, making that style intrinsic to the model instead of something you supply every call.
And here's the rule that saves the most pain: fine-tuning is bad for facts. It doesn't reliably memorize specific facts, and you can't update it without retraining, so its knowledge goes stale. Use RAG or long context for knowledge. Use fine-tuning for behavior. Keep those two straight and you avoid the most costly mistake in the field.
The trap: fine-tuning to add facts or fresh knowledge. That's the classic, costly error — you spend the time and money to retrain, and it still gets specific details wrong, and it can't be updated when things change. Facts belong in retrieval; fine-tuning shapes style and skill. Match the tool to the problem and the whole system gets simpler.
So: knowledge that's big, private, or changing goes to RAG; small and self-contained goes in the context; behavior and style go to fine-tuning — and you often combine them. But all three are per-query. Nothing carries across a whole session. So how does an agent remember? Next: types of agent memory.