hawkes.aiKey ideas
- The one idea — Backpropagation (popularized 1986) finally trained multi-layer networks — but the 1980s expert-systems boom and its bust drove the second AI winter; the keepers of the flame, like LeCun's convolutional nets reading bank checks, proved the ideas were right and only the compute and data were missing.
- How it is shown — A key turns and a reverse-current wakes the frozen network; a rival "rule cathedral" of hand-coded expert systems rises, then shatters; one warm workshop keeps a convolutional net reading checks through the winter; a balance-scale shows the ideas were right and the ingredients were missing.
- The trap to avoid — Reading the winters as proof the ideas were wrong — backprop and convnets worked; what was missing was scale, not correctness.
Last time, the neuron froze on a missing algorithm — a way to train the hidden layer that could beat XOR. This episode, that algorithm arrives.
The one idea
Backpropagation (popularized 1986) finally trained multi-layer networks — but the 1980s expert-systems boom and its bust drove the second AI winter; the keepers of the flame, like LeCun's convolutional nets reading bank checks, proved the ideas were right and only the compute and data were missing.
And then, almost immediately, the field freezes again. For a completely different reason. The key was backpropagation. Run an example forward, measure the error at the output, then push it backward layer by layer — nudging every weight to reduce it. The method had older roots, in Werbos and others, but a nineteen eighty-six paper by Rumelhart, Hinton, and Williams made it click. Multi-layer networks could finally learn. But the money wasn't watching neural networks. The nineteen eighties belonged to a rival approach: expert systems. Instead of learning from data, you'd interview human experts and hand-code their knowledge as thousands of if-then rules. It worked well enough to sell — and a whole industry of Lisp machines rose to run it. Then it broke.
How it works — the demo
A key turns and a reverse-current wakes the frozen network; a rival "rule cathedral" of hand-coded expert systems rises, then shatters; one warm workshop keeps a convolutional net reading checks through the winter; a balance-scale shows the ideas were right and the ingredients were missing.
Expert systems were brittle — they couldn't handle anything outside their rules, and every update meant more hand-coding. Around nineteen eighty-seven, the specialized-hardware market collapsed as ordinary workstations caught up. The promises had outrun the product again. The second AI winter set in, and stretched into the mid-nineties. A few kept working through the cold. At Bell Labs, Yann LeCun built convolutional networks — neural nets that scan an image in patches. His system, LeNet, learned to read handwritten digits and shipped: by the late nineties it was reading a real share of the checks cashed in America. Deep learning, deployed, while the field called it unfashionable. Here's the lesson, and the trap. It's tempting to say the winters proved the ideas wrong. They didn't.
The trap to avoid
Reading the winters as proof the ideas were wrong — backprop and convnets worked; what was missing was scale, not correctness.
Why it matters — and what’s next
Backprop worked. Convolutional nets worked. What was missing wasn't the idea — it was scale: enough data to learn from, and enough compute to do the learning. The recipe was right; the ingredients didn't exist yet. So the second winter wasn't a refutation. It was a waiting room. The ideas were finished and idle, listening for two things that hadn't arrived: a mountain of labeled data, and a processor fast enough to chew through it. Both were coming. Next: a dataset called ImageNet.
This is a supplement in AI: Zero → Frontier — a side-trip that deepens the act it sits beside, one file and one loop at a time.
Full transcript 2:45 of narration
Last time, the neuron froze on a missing algorithm — a way to train the hidden layer that could beat XOR. This episode, that algorithm arrives. And then, almost immediately, the field freezes again.
For a completely different reason. The key was backpropagation. Run an example forward, measure the error at the output, then push it backward layer by layer — nudging every weight to reduce it.
The method had older roots, in Werbos and others, but a nineteen eighty-six paper by Rumelhart, Hinton, and Williams made it click. Multi-layer networks could finally learn. But the money wasn't watching neural networks.
The nineteen eighties belonged to a rival approach: expert systems. Instead of learning from data, you'd interview human experts and hand-code their knowledge as thousands of if-then rules. It worked well enough to sell — and a whole industry of Lisp machines rose to run it.
Then it broke. Expert systems were brittle — they couldn't handle anything outside their rules, and every update meant more hand-coding. Around nineteen eighty-seven, the specialized-hardware market collapsed as ordinary workstations caught up.
The promises had outrun the product again. The second AI winter set in, and stretched into the mid-nineties. A few kept working through the cold.
At Bell Labs, Yann LeCun built convolutional networks — neural nets that scan an image in patches. His system, LeNet, learned to read handwritten digits and shipped: by the late nineties it was reading a real share of the checks cashed in America. Deep learning, deployed, while the field called it unfashionable.
Here's the lesson, and the trap. It's tempting to say the winters proved the ideas wrong. They didn't.
Backprop worked. Convolutional nets worked. What was missing wasn't the idea — it was scale: enough data to learn from, and enough compute to do the learning.
The recipe was right; the ingredients didn't exist yet. So the second winter wasn't a refutation. It was a waiting room.
The ideas were finished and idle, listening for two things that hadn't arrived: a mountain of labeled data, and a processor fast enough to chew through it. Both were coming. Next: a dataset called ImageNet.