The knowledge hub
Every episode, supplement and Frontier Dispatch as a readable article — with the cinematic render, the full transcript, and the one idea that matters. 276 articles and counting.
06
Beyond text
21 articles#150Ep2:45
How an image becomes tokens.
An image is cut into fixed patches, each embedded as a vector — so a transformer reads a picture as a sequence of tokens, exactly like text.
#151Ep2:45
Teaching pictures and words one language (CLIP).
Contrastive training pulls matching image/caption pairs together and pushes mismatches apart in one shared vector space — the alignment…
#152Ep2:30
One brain, two senses (vision-language models).
Image patches and text tokens share one residual stream, so a single model can look and read at once — the architecture behind "describe…
#153Ep2:45
A totally different engine: diffusion.
Image generators learn to remove noise, then denoise pure static into a picture step by step — generation as carving, not painting.
#154Ep2:30
Compress first, then dream (latent diffusion).
Denoising happens in a compressed latent space instead of full pixels — collapsing cost enough to put image generation on consumer hardware.
#155Ep2:30
Diffusion vs. autoregression, head to head.
Text is built token-by-token, left to right (autoregression); images are refined all-at-once, iteratively (diffusion) — opposite…
#156Ep2:30
How text controls the image (conditioning).
A text prompt steers diffusion by injecting its meaning into every denoising step — using CLIP's shared space as a rudder — with a…
#157Ep2:30
Why hands and text in images are hard.
Fine structure (hands) and exact symbols (readable text) are where diffusion struggles — because a vibe-carving denoiser is worst at…
#158Ep2:30
Image editing models (the 'nano-banana' class).
Instruction-driven editors take a real image plus a text command and change only what's asked while preserving the rest — the shift from…
#159Ep2:45
The hardest problem: video generation.
Video generation is diffusion extended across time — the hard part isn't any single frame, it's making hundreds of frames agree with each…
#160Ep2:30
Why video models 'forget' between frames.
Video models "forget" because they re-derive each frame from limited temporal memory instead of truly tracking objects — so anything not…
#161Ep2:30
Veo, Seedance, Wan, Kling — the landscape.
The video-generation landscape is a fast-moving field of contenders best understood by axes — quality, clip length, control, access…
#162Ep2:30
Steering video: first-frame, last-frame (FLF2V).
Pinning the first and last frames (FLF2V) turns video from a lucky roll into directed motion — the model must start and end exactly where…
#163Ep2:30
How AI 'understands' a video you give it.
A video-input model doesn't watch continuously — it samples a handful of frames, turns each into patch-tokens, and reasons over them as a…
#164Ep2:30
Speech in, speech out (audio models).
Audio is just another modality to tokenize — sound becomes tokens, so the same machine can listen (speech-to-text) and speak…
#165Ep2:30
The multimodal failure modes to know.
Multimodal models fail in predictable ways — reading text (OCR) errors, miscounting, weak spatial reasoning, and confident hallucination…
#166Ep2:15
Why 'omni' models are the trajectory.
The trajectory is "omni" models — one model natively handling text, image, audio, and video in a single shared stream — and this whole act…
B1Supp2:45
AlphaFold, and AI as a scientific instrument.
Protein folding — predicting a 3D structure from an amino-acid sequence — was a roughly fifty-year grand challenge, and AlphaFold2…
B2Supp2:45
The recommender engines that already run your day.
The most economically deployed AI on Earth isn't a chatbot — it's ranking and recommendation: collaborative filtering to learned…
B3Supp2:45
Robots, and the reality gap.
Embodied AI is the hard mode: Moravec's paradox, the reality gap and sim-to-real, dexterity far harder than locomotion, and a missing…
B4Supp2:45
Games, weather, chips: superhuman where the reward is clean.
Narrow superhuman AI follows one recipe — a clean, cheap reward plus self-play or simulation — from AlphaGo/AlphaZero/MuZero to weather…