Running models in production: the ops nobody sees.

Episode one-eighteen put "ops people" on the ownership bill as one line. Tonight, the line unfolds — because between a downloaded model and a dependable product sits an entire discipline: engines, rollouts, scaling, monitoring, and the pager.

The one idea

Production model-ops is its own discipline: engines, pinned driver stacks, canary rollouts sized to slow model loads, queue-aware autoscaling, LLM golden signals, redundancy, and quality canaries.

This is what running models in production means. Walk the apparatus with me. Layer one: the engine — vLLM, TensorRT-LLM, llama-dot-cpp-class machinery that packages this act's serving tricks into one appliance. Choose per hardware and workload. Beneath it, the stack nobody romanticizes: driver, CUDA, engine, and model versions that must align exactly. Production fleets pin every version in containers, because "latest" is how Tuesday's driver update becomes Wednesday's outage. Layer two: rollouts. Models are multi-gigabyte cargo — loading one takes minutes, not milliseconds — so updates move with ceremony: a canary slice serves the new version first; shadow traffic compares its answers against the incumbent; an eval gate — your own evals, a later episode insists — decides promotion; traffic shifts gradually, rollback one lever away. The reckless alternative is a bad model everywhere at once, discovered by customers. Layer three: scaling, where web instincts break hardest.

How it works — the demo

A serving fleet's day: a canary rollout gating on evals, a traffic spike absorbed by queue-depth scaling, a quality-drift alarm caught by scheduled canary prompts, a graceful drain.

Web servers cold-start in seconds; a model server needs minutes to load weights — so you scale before you're desperate, triggered by queue depth, holding warm headroom you'll resent until the day it saves you. And the deeper truth: GPUs aren't always summonable at any price. Capacity is a market. Plan it like one. Layer four: the gauges. LLM serving has its own golden signals — time-to-first-token and tokens-per-second, the user's clocks from episode one-fourteen; queue depth; GPU utilization and memory; KV-cache occupancy; error rates; cost per million tokens. And one gauge infrastructure can't provide: quality. Scheduled canary prompts, scored over time, catch the rot — a bad quant, a broken template, a drifted dependency — that every other dial misses. Layer five: staying up. Redundancy with headroom, health checks, and the LLM-specific courtesy of graceful drain — finishing in-flight generations before a machine bows out, because killing someone's half-written answer is a broken promise.

The trap to avoid

Treating LLM serving like stateless web serving — KV state, minutes-long cold starts, and GPU scarcity break the standard playbook.

Why it matters — and what’s next

Rolling patches, since drivers carry CVEs like any software. Provenance checks on every model load — episode one-oh-one's lesson, institutionalized. All of it boring, rehearsed, and the entire difference between a demo and a service. The trap crowning it all: running model fleets on the web-serving playbook. Web serving assumes statelessness, instant starts, and infinite capacity on demand — and language models violate all three: conversations carry state, starts take minutes, and the hardware is a scarce market. Different physics, different playbook. One layer remains — the building itself. Next: follow a watt from the substation to the chip.

This is a supplement in AI: Zero → Frontier — a side-trip that deepens the act it sits beside, one file and one loop at a time.