hawkes.aiKey ideas
- The one idea — The three flagship benchmarks measure genuinely different things — real-bug-fixing, graduate science reasoning, human preference — and each has a systematic blind spot.
- How it is shown — Three proving grounds toured: what each tests, what each misses; the same model ranking differently across them.
- The trap to avoid — Reading a single benchmark as "how good is it" — each measures one axis, and a model can top one while failing your job.
- What it sets up — And even these can be gamed.
Same model, three famous benchmarks, three different ranks. Real bug-fixing, graduate science, human taste — they measure different things. Here's how to read them.
The one idea
The three flagship benchmarks measure genuinely different things — real-bug-fixing, graduate science reasoning, human preference — and each has a systematic blind spot.
Every release brags in benchmark scores, and most people read them as one thing: good. They're not. Three flagship benchmarks measure three different axes — and the same model can top one and stumble on another. Tour the proving grounds, learn what each tests and misses, and the leaderboard stops fooling you. Proving ground one: SWE-bench. Real bug reports from real projects, fixes held back — the model must produce a patch that passes the project's own tests. Its virtue is realism: genuine software judgment, verifiably scored. Its skew: it leans on particular stacks, depends on scaffolding, and captures a slice of coding. A strong number means something — about that slice. Proving ground two: GPQA. Graduate-level science, deliberately Google-proof — real multi-step reasoning.
How it works — the demo
Three proving grounds toured: what each tests, what each misses; the same model ranking differently across them.
Its virtue is difficulty with clean answers. Its limits: a narrow slice, distant from most jobs, and — a fixed-answer exam — exactly the kind of test that leaks into training, episode one-forty-nine's whole problem. A high score is real reasoning, of one narrow kind. Proving ground three: Arena Elo. Two anonymous models answer one prompt, a human picks the better, thousands of duels become a chess-style rating. Its virtue is relevance: real humans, real preference at scale. Its bias is human nature — the crowd rewards confidence, length, and polish, so episode seventy-nine's sycophancy shows on the scoreboard. Felt helpfulness, which charm can counterfeit. The skill isn't any single ground — it's triangulation. Plot scores across complementary benchmarks and read the shape: a code spike with modest science and high preference is one animal; a rounded profile another. That shape, matched to your need, beats any headline number.
The trap to avoid
Reading a single benchmark as "how good is it" — each measures one axis, and a model can top one while failing your job.
Why it matters — and what’s next
And even these can be gamed.
Benchmarks are a coordinate system. The trap: "it tops the leaderboard, so it's best." Best at that benchmark's one axis — and your job is a different axis. A code champion crowned "state of the art" flops at warm support; a science ace fumbles long-context recall. "State of the art" has a silent tail: at one thing, one way. Read it before you deploy the crown. But the shadow over all three grounds: even good benchmarks can be gamed. A record score might reflect genuine skill — or a model tuned to the test's quirks, or one that saw the answers before. It tops the leaderboard either way, and passes the model-card audit either way. How that's done, and how to spot it: next. Bench-maxxing.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
Every release brags in benchmark scores, and most people read them as one thing: good. They're not. Three flagship benchmarks measure three different axes — and the same model can top one and stumble on another. Tour the proving grounds, learn what each tests and misses, and the leaderboard stops fooling you.
Proving ground one: SWE-bench. Real bug reports from real projects, fixes held back — the model must produce a patch that passes the project's own tests. Its virtue is realism: genuine software judgment, verifiably scored. Its skew: it leans on particular stacks, depends on scaffolding, and captures a slice of coding. A strong number means something — about that slice.
Proving ground two: GPQA. Graduate-level science, deliberately Google-proof — real multi-step reasoning. Its virtue is difficulty with clean answers. Its limits: a narrow slice, distant from most jobs, and — a fixed-answer exam — exactly the kind of test that leaks into training, episode one-forty-nine's whole problem. A high score is real reasoning, of one narrow kind.
Proving ground three: Arena Elo. Two anonymous models answer one prompt, a human picks the better, thousands of duels become a chess-style rating. Its virtue is relevance: real humans, real preference at scale. Its bias is human nature — the crowd rewards confidence, length, and polish, so episode seventy-nine's sycophancy shows on the scoreboard. Felt helpfulness, which charm can counterfeit.
The skill isn't any single ground — it's triangulation. Plot scores across complementary benchmarks and read the shape: a code spike with modest science and high preference is one animal; a rounded profile another. That shape, matched to your need, beats any headline number. Benchmarks are a coordinate system.
The trap: "it tops the leaderboard, so it's best." Best at that benchmark's one axis — and your job is a different axis. A code champion crowned "state of the art" flops at warm support; a science ace fumbles long-context recall. "State of the art" has a silent tail: at one thing, one way. Read it before you deploy the crown.
But the shadow over all three grounds: even good benchmarks can be gamed. A record score might reflect genuine skill — or a model tuned to the test's quirks, or one that saw the answers before. It tops the leaderboard either way, and passes the model-card audit either way. How that's done, and how to spot it: next. Bench-maxxing.