hawkes.aiKey ideas
- The one idea — Constrained decoding uses a grammar to allow only tokens that keep the output valid at each step — making invalid output impossible (only truncation can leave it unfinished), not just unlikely.
- How it is shown — A grammar defining valid JSON; at each generation step only tokens that fit are permitted; the model physically cannot emit a malformed character.
- The trap to avoid — Thinking the model "learned to be careful" — the model is unchanged; the system removes invalid options at generation time.
- What it sets up — How does 'only valid tokens' actually happen at each step?
You can make an AI that physically cannot produce broken JSON. Not careful — incapable. And the model itself doesn't change at all. Here's the trick.
The one idea
Constrained decoding uses a grammar to allow only tokens that keep the output valid at each step — making invalid output impossible (only truncation can leave it unfinished), not just unlikely.
Last episode, asking for JSON gave you usually-valid output. Here's how to make it always-valid — literally incapable of being malformed. The trick isn't to check the output afterward and retry failures. It's to constrain the generation itself, so at every step the model can only pick tokens that keep the output valid. Invalid output becomes impossible. This is constrained decoding. Recall how generation works: the model builds output one token at a time, choosing each from its whole vocabulary. Normally any token is fair game — which is exactly how a stray bracket or chatty preamble sneaks in. Constrained decoding changes what's allowed at each step. The freedom that lets it wander is what we'll restrict. The tool is a grammar — a formal set of rules for what counts as valid.
How it works — the demo
A grammar defining valid JSON; at each generation step only tokens that fit are permitted; the model physically cannot emit a malformed character.
For JSON: a value is a string, number, list, or object; an object opens with a brace, keys are quoted, and so on. The grammar maps every legal path through the output and, by exclusion, every illegal one. It's the rulebook for well-formed. Now combine them. At every step, the grammar acts as a gate: given what's been produced so far, it computes which tokens could come next and stay valid — and forbids the others. Just opened a brace? It permits a quote or a closing brace, but not a stray letter or trailing comma. The model chooses only among legal moves. The consequence is strong: the output can't be malformed, because no malformed path was ever available. Not "unlikely to be wrong" — incapable of it. A trailing comma, a missing bracket, a chatty preamble: none are reachable moves, so none can happen.
The trap to avoid
Thinking the model "learned to be careful" — the model is unchanged; the system removes invalid options at generation time.
Why it matters — and what’s next
How does 'only valid tokens' actually happen at each step?
Valid output every time — only a truncation cutoff can leave it unfinished — guaranteed by construction, not by good behavior. And it's not just JSON. A grammar can define any structure: a schema with exact fields, a date format, a SQL query, a chess move, a made-up mini-language. Whatever you can specify as rules, you can force the output to obey. Constrained decoding turns "please follow this format" into "you cannot break this format." The trap: imagining the model learned to be more careful. It didn't — the model is unchanged. Something outside it deletes the invalid options at generation time, so even a model that "wanted" a trailing comma couldn't produce one. The guarantee comes from the constraint, not the intelligence. But how do you actually remove a token mid-generation? Next: how logit masking works.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 3:00 of narration
Last episode, asking for JSON gave you usually-valid output. Here's how to make it always-valid — literally incapable of being malformed. The trick isn't to check the output afterward and retry failures. It's to constrain the generation itself, so at every step the model can only pick tokens that keep the output valid. Invalid output becomes impossible. This is constrained decoding.
Recall how generation works: the model builds output one token at a time, choosing each from its whole vocabulary. Normally any token is fair game — which is exactly how a stray bracket or chatty preamble sneaks in. Constrained decoding changes what's allowed at each step. The freedom that lets it wander is what we'll restrict.
The tool is a grammar — a formal set of rules for what counts as valid. For JSON: a value is a string, number, list, or object; an object opens with a brace, keys are quoted, and so on. The grammar maps every legal path through the output and, by exclusion, every illegal one. It's the rulebook for well-formed.
Now combine them. At every step, the grammar acts as a gate: given what's been produced so far, it computes which tokens could come next and stay valid — and forbids the others. Just opened a brace? It permits a quote or a closing brace, but not a stray letter or trailing comma. The model chooses only among legal moves.
The consequence is strong: the output can't be malformed, because no malformed path was ever available. Not "unlikely to be wrong" — incapable of it. A trailing comma, a missing bracket, a chatty preamble: none are reachable moves, so none can happen. Valid output every time — only a truncation cutoff can leave it unfinished — guaranteed by construction, not by good behavior.
And it's not just JSON. A grammar can define any structure: a schema with exact fields, a date format, a SQL query, a chess move, a made-up mini-language. Whatever you can specify as rules, you can force the output to obey. Constrained decoding turns "please follow this format" into "you cannot break this format."
The trap: imagining the model learned to be more careful. It didn't — the model is unchanged. Something outside it deletes the invalid options at generation time, so even a model that "wanted" a trailing comma couldn't produce one. The guarantee comes from the constraint, not the intelligence. But how do you actually remove a token mid-generation? Next: how logit masking works.