hawkes.aiKey ideas
- The one idea — Semantic entropy catches confabulation by sampling several answers and measuring whether they agree in meaning: if the model knows, they converge; if it's making things up, they scatter into different fabrications. High semantic entropy flags a likely hallucination.
- How it is shown — The same question asked five times — a known fact yields five answers that mean the same thing; an unknown one yields five different made-up answers — the scatter itself is the signal.
- The trap to avoid — Thinking one confident answer is trustworthy — a single fluent reply hides the scatter; you have to sample and compare meanings to see the model contradicting itself.
- What it sets up — Semantic entropy flags likely errors — but flagging isn't a GUARANTEE on the error rate. can we get one?
Want to catch an AI making things up? Ask it the same question five times. If it truly knows, the answers agree. If it's inventing, they scatter.
The one idea
Semantic entropy catches confabulation by sampling several answers and measuring whether they agree in meaning: if the model knows, they converge; if it's making things up, they scatter into different fabrications. High semantic entropy flags a likely hallucination.
Token entropy missed confident lies. The fix for that is beautifully simple: don't ask once — ask several times, and see whether the model agrees with itself. It turns out a model can state one fabrication confidently, but it struggles to invent the exact same fabrication over and over. Sample the same question a handful of times, letting the model vary a little each time. If it actually knows the answer, every sample says the same thing — maybe worded differently, but the same meaning. If it's confabulating, filling a gap it can't recall, each sample tends to invent something different. Knowing converges; making it up scatters. The trick is to compare by meaning, not exact words. "Paris" and "It's Paris" are the same answer, so you group the samples into meaning-clusters. Then you measure the spread across those clusters.
How it works — the demo
The same question asked five times — a known fact yields five answers that mean the same thing; an unknown one yields five different made-up answers — the scatter itself is the signal.
That spread is semantic entropy — uncertainty about what is being said, not merely how it's phrased. One cluster means agreement; many means the model is all over the place. Low semantic entropy: the samples converge, the model is consistent, likely grounded. High semantic entropy: the samples scatter into different names, dates, and facts — a strong sign the model is making it up. Crucially, this catches exactly the confident confabulation that per-token entropy waved right through, because now we're measuring agreement of meaning, not confidence in single words. It works because a model can state one fabrication fluently — but a fresh invention rarely repeats the same way twice. Truth is a single attractor; improvised lies scatter — though a trained-in falsehood can converge too. Sampling repeatedly, then measuring the scatter, turns that difference into a signal you can actually read. You're catching the model quietly contradicting itself across attempts. The trap: trusting a single confident answer.
The trap to avoid
Thinking one confident answer is trustworthy — a single fluent reply hides the scatter; you have to sample and compare meanings to see the model contradicting itself.
Why it matters — and what’s next
Semantic entropy flags likely errors — but flagging isn't a GUARANTEE on the error rate. can we get one?
One fluent reply tells you nothing about whether the model actually knows — it hides the scatter completely. You only see the uncertainty by asking again and comparing meanings. Consistency across samples is the real signal, not confidence in any one reply. A lone answer, however sure it sounds, is exactly what you can't verify. So semantic entropy catches confabulation: sample, cluster by meaning, measure the scatter. It flags the likely hallucinations that surface confidence hides. But "likely" is still a hint, not a promise. Can we turn a confidence score into an actual guarantee on the error rate? Next: conformal prediction.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
Token entropy missed confident lies. The fix for that is beautifully simple: don't ask once — ask several times, and see whether the model agrees with itself. It turns out a model can state one fabrication confidently, but it struggles to invent the exact same fabrication over and over.
Sample the same question a handful of times, letting the model vary a little each time. If it actually knows the answer, every sample says the same thing — maybe worded differently, but the same meaning. If it's confabulating, filling a gap it can't recall, each sample tends to invent something different. Knowing converges; making it up scatters.
The trick is to compare by meaning, not exact words. "Paris" and "It's Paris" are the same answer, so you group the samples into meaning-clusters. Then you measure the spread across those clusters. That spread is semantic entropy — uncertainty about what is being said, not merely how it's phrased. One cluster means agreement; many means the model is all over the place.
Low semantic entropy: the samples converge, the model is consistent, likely grounded. High semantic entropy: the samples scatter into different names, dates, and facts — a strong sign the model is making it up. Crucially, this catches exactly the confident confabulation that per-token entropy waved right through, because now we're measuring agreement of meaning, not confidence in single words.
It works because a model can state one fabrication fluently — but a fresh invention rarely repeats the same way twice. Truth is a single attractor; improvised lies scatter — though a trained-in falsehood can converge too. Sampling repeatedly, then measuring the scatter, turns that difference into a signal you can actually read. You're catching the model quietly contradicting itself across attempts.
The trap: trusting a single confident answer. One fluent reply tells you nothing about whether the model actually knows — it hides the scatter completely. You only see the uncertainty by asking again and comparing meanings. Consistency across samples is the real signal, not confidence in any one reply. A lone answer, however sure it sounds, is exactly what you can't verify.
So semantic entropy catches confabulation: sample, cluster by meaning, measure the scatter. It flags the likely hallucinations that surface confidence hides. But "likely" is still a hint, not a promise. Can we turn a confidence score into an actual guarantee on the error rate? Next: conformal prediction.