Rolling out safely to production.
A field guide from the Moon team on how to move from a working deploy to one you can ship on a Friday.
The gap between a deploy that works in staging and a deploy that survives a Friday afternoon is smaller than most teams think, but the shape of the gap changes as your traffic does. This piece is about how the Moon team closes it in production.
Every rollout carries three risks: a regression the eval missed, a latency cliff that only appears under real load, and an operational stumble on the way in or out. The techniques below reduce all three, and the order matters.
Start with a routed 5 percent
The single most reliable safety mechanism we have found is to start every rollout at five percent of live traffic, held there long enough to see two things: the p95 latency curve settle, and the eval verdicts for at least the top five buckets clear.