hawkes.aiKey ideas
- The one idea — Humans rank pairs; a reward model learns the judge; RL tunes the assistant to please it — on a leash so it stays sensible.
- How it is shown — A vs B ranking → the judge-model forming → the tower tuned against the judge, leash taut.
- The trap to avoid — It learned a judge, not your rules — and anything a judge systematically prefers, deserved or not, gets trained in.
- What it sets up — The judge's exploitable habits.
Your AI never read a rulebook of values. It learned a judge: a second model bottling thousands of human rankings — then practiced pleasing it, millions of times.
The one idea
Humans rank pairs; a reward model learns the judge; RL tunes the assistant to please it — on a leash so it stays sensible.
The step that made assistants worth talking to began with the humblest gesture in machine learning: a person looking at two answers and pointing at the better one. Thousands of times. Here's how pointing became taste. Why ranking? Because episode seventy-five hit a wall: you can't write a demonstration for every shade of better. But here's the human superpower — we can't always write excellence, and we can almost always recognize it. Judging a pair takes seconds. Preference data scales where demonstration data can't. Step two: bottle the taste. Train a second model — the reward model — on those thousands of rankings until it predicts which answer the human would have pointed at.
How it works — the demo
A vs B ranking → the judge-model forming → the tower tuned against the judge, leash taut.
Now you own something precious: a tireless, instant copy of your raters' judgment. A judge that never sleeps, scoring any answer in milliseconds. Step three: reinforcement. The assistant generates answers by the thousand; the judge scores each; and training pushes toward whatever scores higher. It's practice with a critic — millions of attempts, no human needed in the loop, because the humans already taught the judge. This — supervised shaping plus reinforced taste — is the recipe that trained InstructGPT, then ChatGPT, then nearly every assistant since. One safeguard makes it work: the leash. A model chasing only the judge's score will eventually find weird text that games the judge — flattering shapes, hollow content. So training adds a tether to the original model: improve the scores, but stay close to the sensible ancestor. Drift far, pay a penalty.
The trap to avoid
It learned a judge, not your rules — and anything a judge systematically prefers, deserved or not, gets trained in.
Why it matters — and what’s next
The judge's exploitable habits.
Taste, on a leash — that tension defines the whole method. Now the trap, and it's the door to the next two episodes: the model didn't learn your principles. It learned a judge — and judges have habits. If raters systematically preferred longer answers, confident tones, agreeable framing, those preferences got trained in with everything else, deserved or not. Whatever the judge rewards, the student becomes. Hold that sentence; it explains two famous personality flaws, next. So that's RLHF: rank, bottle the judge, reinforce on a leash. It built the assistant era — and it's heavy: a second model to train, a notoriously touchy reinforcement loop, infrastructure to babysit. Which made one twenty-twenty-three paper land like a gift: what if the ranking pairs could tune the model directly — no judge, no loop? Next: DPO.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
The step that made assistants worth talking to began with the humblest gesture in machine learning: a person looking at two answers and pointing at the better one. Thousands of times. Here's how pointing became taste.
Why ranking? Because episode seventy-five hit a wall: you can't write a demonstration for every shade of better. But here's the human superpower — we can't always write excellence, and we can almost always recognize it. Judging a pair takes seconds. Preference data scales where demonstration data can't.
Step two: bottle the taste. Train a second model — the reward model — on those thousands of rankings until it predicts which answer the human would have pointed at. Now you own something precious: a tireless, instant copy of your raters' judgment. A judge that never sleeps, scoring any answer in milliseconds.
Step three: reinforcement. The assistant generates answers by the thousand; the judge scores each; and training pushes toward whatever scores higher. It's practice with a critic — millions of attempts, no human needed in the loop, because the humans already taught the judge. This — supervised shaping plus reinforced taste — is the recipe that trained InstructGPT, then ChatGPT, then nearly every assistant since.
One safeguard makes it work: the leash. A model chasing only the judge's score will eventually find weird text that games the judge — flattering shapes, hollow content. So training adds a tether to the original model: improve the scores, but stay close to the sensible ancestor. Drift far, pay a penalty. Taste, on a leash — that tension defines the whole method.
Now the trap, and it's the door to the next two episodes: the model didn't learn your principles. It learned a judge — and judges have habits. If raters systematically preferred longer answers, confident tones, agreeable framing, those preferences got trained in with everything else, deserved or not. Whatever the judge rewards, the student becomes. Hold that sentence; it explains two famous personality flaws, next.
So that's RLHF: rank, bottle the judge, reinforce on a leash. It built the assistant era — and it's heavy: a second model to train, a notoriously touchy reinforcement loop, infrastructure to babysit. Which made one twenty-twenty-three paper land like a gift: what if the ranking pairs could tune the model directly — no judge, no loop? Next: DPO.