Rolling out safely to production.
A field guide from the Moon team on how to move from a working deploy to one you can ship on a Friday.
A field guide from the Moon team on how to move from a working deploy to one you can ship on a Friday.
The gap between a deploy that works in staging and a deploy that survives a Friday afternoon is smaller than most teams think, but the shape of the gap changes as your traffic does. This piece is about how the Moon team closes it in production.
Every rollout carries three risks: a regression the eval missed, a latency cliff that only appears under real load, and an operational stumble on the way in or out. The techniques below reduce all three, and the order matters.
The single most reliable safety mechanism we have found is to start every rollout at five percent of live traffic, held there long enough to see two things: the p95 latency curve settle, and the eval verdicts for at least the top five buckets clear.
Five percent is a compromise. It is small enough that a bad deploy will not tank the customer experience, and large enough that the shape of the request mix in production is properly represented within a few minutes.
await moon.deployments.rollout({ version: "v3.3.1", strategy: { type: "routed", share: 0.05 }, hold_until: ["latency_stable", "evals_pass"], });
Aggregate p95 numbers hide the interesting failure modes. What kills a Friday deploy is usually a single request path getting 3x slower, buried inside a distribution that still looks fine on average. Traces show you that. Aggregates do not.
The Moon Trace stream view during a rollout is the single most useful piece of surface area in the product for this. Filter by served_by to isolate the new version, and eye the tail.
If your rollout playbook depends on an aggregate p95 alert, your playbook does not survive Friday.
Rollbacks are a solved problem in theory and a difficult problem in practice. In theory, revert to the last healthy version. In practice, the last healthy version may have a stale cache, a stale prompt, or a stale integration.
The Moon rollback strategy pre-warms the previous version so a revert lands in under five seconds without a cold start. But even five seconds of degraded traffic feels like a long time when a customer is on the other side.