hawkes.aiKey ideas
- The one idea — At each step the model scores every token (a logit); constrained decoding sets forbidden tokens' logits to negative infinity before sampling, making their probability exactly zero — unpickable.
- How it is shown — The logit vector over the vocabulary; the grammar's forbidden tokens crushed to −∞; softmax turns them to zero probability; sampling can't choose them.
- The trap to avoid — Thinking it generates-then-rejects bad tokens — it removes them before the choice, so there's nothing to reject.
- What it sets up — Forcing structure is powerful — but it has a hidden cost.
To ban a token, the system sets its score to negative infinity. Probability: exactly zero. Not filtered out afterward — never choosable in the first place.
The one idea
At each step the model scores every token (a logit); constrained decoding sets forbidden tokens' logits to negative infinity before sampling, making their probability exactly zero — unpickable.
Now the actual mechanism, and it's elegant. To forbid a token, the system doesn't watch the output and reject mistakes. It reaches inside the model's final step and makes the bad tokens unpickable before a choice is made. The trick is called logit masking — and once you see it, constrained decoding stops being magic. Recall how a token gets chosen. At each step the model outputs a logit — a raw score — for every token in its vocabulary. Those scores run through softmax into probabilities, and the next token is sampled from that distribution. Higher logit, higher chance. That scoreboard of logits is where we intervene. Here's the move. Before the scores become probabilities, the grammar marks every token that would break the format — and sets their logits to negative infinity.
How it works — the demo
The logit vector over the vocabulary; the grammar's forbidden tokens crushed to −∞; softmax turns them to zero probability; sampling can't choose them.
Not lowered — negative infinity. Through softmax, that becomes a probability of exactly zero. The forbidden tokens don't just become unlikely; they become impossible to sample. Why negative infinity, not just a low score? Because merely unlikely still means occasionally chosen, and "occasionally" is exactly the crash you're preventing. Zero is the only safe number. Negative infinity guarantees the probability is precisely zero, so no matter how the sampling rolls, a forbidden token can never come out. And this happens at every step. Token by token, the grammar recomputes which options are legal now, masks the rest to negative infinity, and lets the model pick freely among what remains. The model still chooses — its intelligence decides which valid token is best — but the illegal ones aren't on the menu. Freedom within the rails.
The trap to avoid
Thinking it generates-then-rejects bad tokens — it removes them before the choice, so there's nothing to reject.
Why it matters — and what’s next
Forcing structure is powerful — but it has a hidden cost.
The trap: thinking the system generates a token and then checks it, rejecting bad ones. It never generates-then-rejects — that's slow and could still fail. It removes the bad options before the choice, so there's nothing to reject. The mantra: don't check the output; make the wrong output unpickable. So that's the whole stack: a grammar defines valid, and logit masking enforces it by crushing forbidden tokens to zero at every step. Reliable structure, guaranteed. But forcing the model down these rails isn't free — constraining what it can say can quietly change how well it thinks. There's a hidden tax on structure. Next: the hidden cost of forcing structure.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:45 of narration
Now the actual mechanism, and it's elegant. To forbid a token, the system doesn't watch the output and reject mistakes. It reaches inside the model's final step and makes the bad tokens unpickable before a choice is made. The trick is called logit masking — and once you see it, constrained decoding stops being magic.
Recall how a token gets chosen. At each step the model outputs a logit — a raw score — for every token in its vocabulary. Those scores run through softmax into probabilities, and the next token is sampled from that distribution. Higher logit, higher chance. That scoreboard of logits is where we intervene.
Here's the move. Before the scores become probabilities, the grammar marks every token that would break the format — and sets their logits to negative infinity. Not lowered — negative infinity. Through softmax, that becomes a probability of exactly zero. The forbidden tokens don't just become unlikely; they become impossible to sample.
Why negative infinity, not just a low score? Because merely unlikely still means occasionally chosen, and "occasionally" is exactly the crash you're preventing. Zero is the only safe number. Negative infinity guarantees the probability is precisely zero, so no matter how the sampling rolls, a forbidden token can never come out.
And this happens at every step. Token by token, the grammar recomputes which options are legal now, masks the rest to negative infinity, and lets the model pick freely among what remains. The model still chooses — its intelligence decides which valid token is best — but the illegal ones aren't on the menu. Freedom within the rails.
The trap: thinking the system generates a token and then checks it, rejecting bad ones. It never generates-then-rejects — that's slow and could still fail. It removes the bad options before the choice, so there's nothing to reject. The mantra: don't check the output; make the wrong output unpickable.
So that's the whole stack: a grammar defines valid, and logit masking enforces it by crushing forbidden tokens to zero at every step. Reliable structure, guaranteed. But forcing the model down these rails isn't free — constraining what it can say can quietly change how well it thinks. There's a hidden tax on structure. Next: the hidden cost of forcing structure.