How do you implement automated rollback for a model tha...

Manual rollback fails for a boring reason: production problems often surface gradually, during off-hours, and by the time a human notices a dashboard trending the wrong way, the bad model has already served thousands of degraded predictions. The fix is to make rollback a monitored, automatic reaction rather than a manual escalation.

That means defining concrete thresholds up front — accuracy dropping more than roughly 5% below the established production baseline within an hour window, P99 latency crossing a fixed ceiling, or error rate exceeding a couple of percent — and running a check against live metrics every few minutes rather than waiting for a human to look. The moment a threshold is breached, traffic routes back to the previous model version instantly, with an alert firing to the team after the fact rather than as the trigger.

The precondition that makes this actually work is keeping the previous model warm and ready to receive full traffic for 48–72 hours after any cutover, instead of decommissioning it the moment the new version goes live — a rollback that has to cold-start a replacement is not meaningfully faster than a human noticing the problem in the first place.