AI: Zero Frontier

The knowledge hub

Every episode, supplement and Frontier Dispatch as a readable article — with the cinematic render, the full transcript, and the one idea that matters. 276 articles and counting.

04

The physical machine

36 articles

Why AI needs GPUs, not CPUs.

GPUs do thousands of identical multiply-adds in parallel; CPUs do a few things fast — and AI's work is the interns' kind.

It's all just matrix multiplication.

Nearly all model compute — layers, attention, everything — is multiplying big grids of numbers.

A GPU is three numbers — compute, VRAM capacity, memory bandwidth — and the famous one is usually the wrong one to watch.

The bottleneck nobody talks about (bandwidth).

Generating a token means hauling every active weight from memory to cores — so bandwidth, not compute, usually sets chat speed.

Registers to disk: the six rooms.

The memory hierarchy — six levels, each bigger and slower — and moving data, not doing math, is the real cost.

FlashAttention: the trick was memory.

The famous speedup computes exact attention without ever writing the giant score-grid to slow memory — new bookkeeping, not new math.

Why VRAM size decides what you can run.

Weights + KV cache + overhead must fit in VRAM — and "fitting" via offload to slower rooms silently destroys speed.

FP32 to FP4: what precision means.

Numeric formats trade per-weight exactness for size and speed — and neural networks need far less exactness than intuition says.

BF16, FP16, FP8 — the working formats.

The formats real models train and run in — and why BF16's range-over-detail split won training.

Quantization: shrink the model, keep the smarts.

Re-storing trained weights in fewer bits quarters the memory — and because generation is bandwidth-bound, it's a speed feature too.

Quantization's dark side.

Over-aggressive quantization degrades models subtly and unevenly — fluency survives while the hardest skills crack, hidden from casual…

What a '.safetensors' file actually is.

A model file is the weights, serialized: a manifest plus billions of named numbers — no code, no readable knowledge.

The file that hacks you back.

Loading a pickle checkpoint can execute arbitrary code — model files were attack vectors, and safetensors exists because of it.

GGUF and the local-model format wars.

GGUF packs quantized weights plus all metadata into one memory-mappable file — the format that made local AI a double-click.

Do the VRAM math yourself.

VRAM needed ≈ params × bytes-per-weight + KV cache + overhead — a 30-second napkin that beats every "will it run?" forum thread.

One GPU isn't enough — now what?

Frontier models are cut across many GPUs — by layers or through every matrix — and the interconnect becomes the new bandwidth wall.

Splitting a model: tensor parallelism.

Saw every layer's matrices across GPUs — all cards compute their shard at once, stitching partial results every layer.

Splitting a model: pipeline parallelism.

Give each GPU different floors of the tower — light chatter that tolerates ordinary links, at the price of idle bubbles.

The other splits: data, experts, ZeRO.

Data parallelism, expert parallelism, and ZeRO's sharded bookkeeping — composed so 10,000-GPU runs survive constant hardware failure.

The hidden cost: the interconnect.

Chip-to-chip links form a hierarchy (NVLink → PCIe → network), and the tier you have decides which model-splits are even viable.

Latency vs. throughput.

Fast-for-one and cheap-for-a-million are opposite optimization targets — one machine can't maximize both at once.

Batching: why servers make you wait a beat.

Grouping requests lets one weight-delivery feed many tokens — multiplying efficiency for a small admission delay.

Continuous batching: no empty seats.

Schedule at the token step, not the request: finished sequences exit mid-flight, new ones board instantly — no empty seats, ever.

Virtual memory for attention (PagedAttention).

Store the KV cache in fixed-size pages with a lookup table — a 1960s OS idea that ended fragmentation and unlocked bigger batches.

A small model drafts, a big one approves.

Speculative decoding — a cheap drafter proposes several tokens; the big model verifies them all in ONE parallel pass; output provably…

Tokens per second, and what's 'fast.'

Tokens/sec is the speed metric — but "fast" depends entirely on who's consuming the tokens: a reader, a skimmer, or a machine.

Why input is cheap and output is expensive.

Input is read in one parallel pass; output is written one serial pass per token; cached input shares pages — the price sheet is the…

A100 → H100 → B200: why the chip matters.

Each GPU generation adds memory, bandwidth, and — crucially — new native number formats that unlock new models and prices.

Why FP4 needs Blackwell.

Unsupported formats get emulated — FP4 on older GPUs keeps the memory/bandwidth win but forfeits the doubled compute rate; know which wall…

Cloud API vs. self-host: the real math.

Rent-per-token vs own-the-hardware breaks even on one dominant variable — utilization — plus honest lines for ops, quality, and privacy.

[LAB] Run a real model tonight.

From download to first token on your own machine in ~20 minutes — the whole act, made physical.

What a datacenter actually is.

"The cloud" is a warehouse: racks of GPU servers, power distribution, and cooling — physical, located, and thirsty.

Why AI's real bottleneck is power.

Electricity — generation, transmission, cooling — increasingly caps AI's growth more than chip supply does; compute strategy is now energy…

Which GPU for which job? The card taxonomy.

GPUs ship in classes — flagship SXM, PCIe datacenter, inference fleet, workstation, consumer, unified-memory, and non-NVIDIA lanes — and…

Running models in production: the ops nobody sees.

Production model-ops is its own discipline: engines, pinned driver stacks, canary rollouts sized to slow model loads, queue-aware…

Inside the building: the power chain and the facilities.

Follow a watt from substation to chip — transformers, UPS, generators, PDUs, PUE, the cooling tiers, water, network fabric, and the lead…