hawkes.aiKey ideas
- The one idea — The formats real models train and run in — and why BF16's range-over-detail split won training.
- How it is shown — Two 16-bit crates split opposite ways; training's wild values falling off FP16's short range but landing safely on BF16's long one.
- The trap to avoid — "Half precision = half quality" — quality tracks whether the format covers what the numbers need, not the bit count.
- What it sets up — If 8 bits can compute, how few can STORE?
Half the bits does not mean half the quality. What matters is whether the format covers the values training throws at it — and one sixteen-bit format got that right.
The one idea
The formats real models train and run in — and why BF16's range-over-detail split won training.
Three formats carry most of modern AI. Two are the same sixteen bits split opposite ways — and that split, range versus detail, decided which one trains the frontier. The working formats are the story of what training actually needs. FP16 spends its budget on detail: fine ticks, lovely exactness — and a short range. Values too large overflow into infinity; too small, they vanish to zero. For graphics work, fine. But training is about to make that short range a problem. Enter BF16 — brain float, born at Google. Same sixteen bits, split range-first: the full stretch of FP32's number line, paid for with coarser detail. Because training taught everyone: gradients swing across wild magnitudes, and a run survives coarse detail — but dies on short range. Range is oxygen.
How it works — the demo
Two 16-bit crates split opposite ways; training's wild values falling off FP16's short range but landing safely on BF16's long one.
Detail is a luxury. BF16 became the frontier's training workhorse. The squeeze continues. FP8 — eight bits, two sub-flavors: one leaning detail for weights and activations, one leaning range for gradient weather. Modern chips run it natively at double the sixteen-bit rate, and frontier labs train substantial parts of runs in it — DeepSeek documented it openly. The staircase from episode ninety-six, descended in real time. So the working map: training runs on a BF16 backbone with FP8 spreading through it. Inference — no gradient storm — tolerates slimmer formats still, which is where the next episode lives. One rule organizes it all: match the format to the values' weather. Storms get range. Calm gets detail.
The trap to avoid
"Half precision = half quality" — quality tracks whether the format covers what the numbers need, not the bit count.
Why it matters — and what’s next
If 8 bits can compute, how few can STORE?
Waste gets fired. The trap: assuming half precision means half quality. Bits aren't quality — coverage is. A format fails when values fall off its ruler, not when its ticks are merely coarse. Many of the models you use daily were trained with seven detail bits — and they're the smartest artifacts our species has produced. The handwriting was never the intelligence. And once training ends, the storm ends. A frozen model's weights just sit there — no gradients, no weather — and the formats can shrink further than any training run would dare. Eight bits. Four. Next, the act's boldest move: quantization — quarter the model, keep the smarts.
This is one short episode in AI: Zero → Frontier, a step-by-step climb through how AI actually works. Each episode builds only on the ones before it.
Full transcript 2:30 of narration
Three formats carry most of modern AI. Two are the same sixteen bits split opposite ways — and that split, range versus detail, decided which one trains the frontier. The working formats are the story of what training actually needs.
FP16 spends its budget on detail: fine ticks, lovely exactness — and a short range. Values too large overflow into infinity; too small, they vanish to zero. For graphics work, fine. But training is about to make that short range a problem.
Enter BF16 — brain float, born at Google. Same sixteen bits, split range-first: the full stretch of FP32's number line, paid for with coarser detail. Because training taught everyone: gradients swing across wild magnitudes, and a run survives coarse detail — but dies on short range. Range is oxygen. Detail is a luxury. BF16 became the frontier's training workhorse.
The squeeze continues. FP8 — eight bits, two sub-flavors: one leaning detail for weights and activations, one leaning range for gradient weather. Modern chips run it natively at double the sixteen-bit rate, and frontier labs train substantial parts of runs in it — DeepSeek documented it openly. The staircase from episode ninety-six, descended in real time.
So the working map: training runs on a BF16 backbone with FP8 spreading through it. Inference — no gradient storm — tolerates slimmer formats still, which is where the next episode lives. One rule organizes it all: match the format to the values' weather. Storms get range. Calm gets detail. Waste gets fired.
The trap: assuming half precision means half quality. Bits aren't quality — coverage is. A format fails when values fall off its ruler, not when its ticks are merely coarse. Many of the models you use daily were trained with seven detail bits — and they're the smartest artifacts our species has produced. The handwriting was never the intelligence.
And once training ends, the storm ends. A frozen model's weights just sit there — no gradients, no weather — and the formats can shrink further than any training run would dare. Eight bits. Four. Next, the act's boldest move: quantization — quarter the model, keep the smarts.