The knowledge hub
Every episode, supplement and Frontier Dispatch as a readable article — with the cinematic render, the full transcript, and the one idea that matters. 276 articles and counting.
05
Architectures & the model zoo
31 articles#122Ep2:45
Dense vs. Mixture-of-Experts.
MoE replaces each feed-forward layer with a committee of experts and wakes only a few per token — huge capacity, modest per-token compute.
#123Ep2:30
'Active' vs. 'total' parameters.
Total = capacity and the memory bill; active = per-token compute and speed — an MoE computes like its active count but must be hosted like…
#124Ep2:30
How the router picks experts.
A tiny learned gate scores experts per token and sends it to the top few — trained jointly, with emergent specialties that are…
#125Ep2:30
Why MoE scales to trillions cheaply.
Adding experts grows capacity without growing per-token compute — the decoupling that made trillion-parameter models affordable.
#126Ep2:30
MoE's headache: keeping experts balanced.
Routing is a rich-get-richer loop — favored experts improve and get favored more — and training must actively enforce the committee's…
#127Ep2:30
Attention got expensive. Enter GQA.
Grouped-Query Attention lets groups of query heads share key/value heads — shrinking the KV cache several-fold at near-zero quality cost.
#128Ep2:30
MLA: DeepSeek's attention compression.
Multi-head Latent Attention stores one small learned compression per token instead of per-head keys/values — an order-of-magnitude cache…
#129Ep2:45
Linear & delta attention (why KDA exists).
Linear/delta attention replaces the growing cache with a fixed-size state — constant cost at any length, bought with lossy recall; hybrids…
#130Ep2:30
How models remember position (RoPE).
Attention is orderless — RoPE weaves position into the vectors themselves by rotating them position-dependent angles, making relative…
#131Ep2:45
Advertised vs. usable context (YaRN).
Context-extension tricks rescale RoPE's angles to stretch the window — but advertised context ≠ usable context, and the gap is measurable.
#132Ep2:45
Open weights vs. open source vs. API-only.
"Open" labels three different offers — endpoint access, downloadable weights, and full-recipe openness — and only the third is open source…
#133Ep2:30
Why 'open weights' isn't 'open source.'
Weights are a compiled artifact — you can run, tune, and study behavior, but not audit data, reproduce the build, or verify provenance.
#134Ep2:30
Licenses that actually bind you.
Model licenses range from genuinely permissive (MIT/Apache) to custom terms with scale thresholds, acceptable-use policies, and naming…
#135Ep2:30
How to read a model card without getting fooled.
A model card mixes verifiable facts with vendor-run claims — separate them, demand the conditions, and trust independent replication.
#136Ep2:45
The family tree: GPT-2 to today.
Nearly every modern model descends from one 2019 recipe — decoder-only transformer, next-token training — through five great branchings.
#137Ep2:15
How GPT-2 shocked everyone.
The 2019 model deemed "too dangerous to release" was tiny by today's measure — and its panic-then-quaintness cycle is the template for…
#138Ep2:30
GPT-3 and the in-context-learning surprise.
At 100× scale, an unplanted ability appeared: show examples in the prompt and the model learns the task on the spot — no weight updates.
#139Ep2:30
InstructGPT: the assistant is born.
RLHF's first commercial landing: a 1.3B model tuned with human feedback was PREFERRED over the raw 175B giant — alignment beat scale, and…
#140Ep2:30
The open frontier: Llama, Qwen, DeepSeek.
Three lineages turned downloadable models from toys into near-frontier tools — and compressed the open-vs-closed gap from years to months.
#141Ep2:45
GLM, Kimi K2/K3, and the trillion-param open era.
Trillion-scale MoE models are now downloadable — mostly from Chinese labs, mostly permissively licensed — a real shift in who ships…
#142Ep2:30
The closed frontier: GPT-5, Fable, Mythos.
The absolute capability ceiling sits behind APIs — closed flagships bundle managed safety, elastic serving, and newest-first capabilities…
#143Ep2:30
Why some models are 'safeguarded' or routed.
Providers gate risky capabilities and sometimes reroute a request to a different (safer or cheaper) model — sometimes protective…
#144Ep2:30
Coding models vs. general models.
Code-specialized training — heavy code diet plus a uniquely verifiable reward — buys large gains on software tasks, so the best coder…
#145Ep2:30
Reasoning-specialized vs. everything models.
Reasoning specialists are trained to think in long verifiable chains — beating larger generalists on hard structured work while feeling…
#146Ep2:45
Benchmarks: SWE-bench, GPQA, Arena Elo.
The three flagship benchmarks measure genuinely different things — real-bug-fixing, graduate science reasoning, human preference — and…
#147Ep2:45
How benchmarks get gamed ('bench-maxxing').
Record scores get engineered without obvious cheating — test-like training data, quirk-tuning, best-of-N, format-matching — producing…
#148Ep2:30
Why you must run your own evals.
A private test built from your real tasks predicts your production quality — immune to the benchmark games, because no model could have…
#149Ep2:30
Data contamination, explained.
When public test answers leak into training pools — usually without anyone lying — scores inflate silently; the durable defense is holding…
O1Supp2:45
The first neuron, and the first winter.
The perceptron (Rosenblatt, 1958) was the first machine that learned from examples — but Minsky & Papert's 1969 proof that a single layer…
O2Supp2:45
Backprop, the second winter, and the keepers of the flame.
Backpropagation (popularized 1986) finally trained multi-layer networks — but the 1980s expert-systems boom and its bust drove the second…
O3Supp2:45
ImageNet, and the spark that lit the era.
In 2012, AlexNet — a deep convolutional net trained on GPUs — won the ImageNet challenge by a landslide, the collision of big data…