hawkes.aiKey ideas
- The one idea — Record scores get engineered without obvious cheating — test-like training data, quirk-tuning, best-of-N, format-matching — producing leaderboard toppers that fail the real task.
- How it is shown — The crooked gymnasium: a model drilled on the exact exercises it'll be tested on, acing the benchmark, faltering on a fresh variant; the tells laid out.
- The trap to avoid — Trusting a benchmark leap without checking for the gaming tells — the score and the capability can diverge completely.
- What it sets up — So trust nothing but YOUR test.
You can top an AI leaderboard without obviously cheating: train on test-like data, tune to its quirks, report your best of many tries.
The one idea
Record scores get engineered without obvious cheating — test-like training data, quirk-tuning, best-of-N, format-matching — producing leaderboard toppers that fail the real task.
A model tops a benchmark by a stunning margin. Is that a capability leap — or a model coached to the test? Both look identical on the leaderboard. Bench-maxxing is the art of the second: engineering a record score without any obvious cheating, producing champions of the test who flunk the sport. Here's how it's done, and how to catch it. Technique one is the deepest: the model saw the test. Benchmark questions and answers, or paraphrases, sit in the training ocean among trillions of tokens — and the model recites what it memorized while looking like it reasoned. Often no one cheated on purpose; the benchmarks are public, the web is the pile. Contamination — its own episode next. Technique two: tune to the quirks. Every benchmark has incidental habits — phrasings, formats, templates — and a model drilled on those aces the test's shape while its grasp of substance stays shallow.
How it works — the demo
The crooked gymnasium: a model drilled on the exact exercises it'll be tested on, acing the benchmark, faltering on a fresh variant; the tells laid out.
Same skill, different format, and it stumbles. Overfitting wearing capability's face: a veneer polished to the leaderboard's surface, no thicker. Technique three: the sampling games — episode one-thirty-five's conditions, weaponized. Best-of-many: attempt each problem fifty times, report only the luckiest — a number no single real call reproduces. Add a generous harness, matched formats, tools quietly enabled, and the score sits under lab conditions no product ships. Reported and actual can be worlds apart. So the tells. A suspiciously narrow win — soaring on one benchmark, ordinary elsewhere. Numbers no one reproduces outside the vendor's harness. A gap between benchmark glory and community vibe. And best-of-N with no single-shot number behind it.
The trap to avoid
Trusting a benchmark leap without checking for the gaming tells — the score and the capability can diverge completely.
Why it matters — and what’s next
So trust nothing but YOUR test.
The through-line: a real leap generalizes; a maxxed one shatters when the apparatus changes. The trap: deploying on a leap you didn't verify. A team re-architects around a record-topper, and production quality stays flat — the leap was maxxed, their workload a fresh apparatus. A gamed score improves the leaderboard, not your product. Treat every benchmark jump as a hypothesis. Which raises the question — test it how? And the escape from the whole hall of mirrors is one move: the private test. Every public benchmark is gameable — leaked, tuned to, best-of-N'd. But a test built from your own real tasks, kept secret, is an apparatus no model could have drilled on: immune to leakage, quirk-tuning, and theater. Why you must run your own evals: next.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
A model tops a benchmark by a stunning margin. Is that a capability leap — or a model coached to the test? Both look identical on the leaderboard. Bench-maxxing is the art of the second: engineering a record score without any obvious cheating, producing champions of the test who flunk the sport. Here's how it's done, and how to catch it.
Technique one is the deepest: the model saw the test. Benchmark questions and answers, or paraphrases, sit in the training ocean among trillions of tokens — and the model recites what it memorized while looking like it reasoned. Often no one cheated on purpose; the benchmarks are public, the web is the pile. Contamination — its own episode next.
Technique two: tune to the quirks. Every benchmark has incidental habits — phrasings, formats, templates — and a model drilled on those aces the test's shape while its grasp of substance stays shallow. Same skill, different format, and it stumbles. Overfitting wearing capability's face: a veneer polished to the leaderboard's surface, no thicker.
Technique three: the sampling games — episode one-thirty-five's conditions, weaponized. Best-of-many: attempt each problem fifty times, report only the luckiest — a number no single real call reproduces. Add a generous harness, matched formats, tools quietly enabled, and the score sits under lab conditions no product ships. Reported and actual can be worlds apart.
So the tells. A suspiciously narrow win — soaring on one benchmark, ordinary elsewhere. Numbers no one reproduces outside the vendor's harness. A gap between benchmark glory and community vibe. And best-of-N with no single-shot number behind it. The through-line: a real leap generalizes; a maxxed one shatters when the apparatus changes.
The trap: deploying on a leap you didn't verify. A team re-architects around a record-topper, and production quality stays flat — the leap was maxxed, their workload a fresh apparatus. A gamed score improves the leaderboard, not your product. Treat every benchmark jump as a hypothesis. Which raises the question — test it how?
And the escape from the whole hall of mirrors is one move: the private test. Every public benchmark is gameable — leaked, tuned to, best-of-N'd. But a test built from your own real tasks, kept secret, is an apparatus no model could have drilled on: immune to leakage, quirk-tuning, and theater. Why you must run your own evals: next.