What a year of autonomous rollbacks taught us

A rollback feels like the safe choice. The previous version worked, the new one does not, so you go back. For about a year Keel has been allowed to recommend rollbacks during incidents, and for a smaller group of customers to perform them automatically under strict rules. Here is what the data says about when that instinct is right.

Most nearby deploys are not the problem

In the incidents we studied, a deploy had gone out in the hour before the first alert about two thirds of the time. It was the actual cause in fewer than half of those. Deploys are frequent, so one is almost always nearby. Treating proximity as proof leads to rollbacks that change nothing, and every unnecessary rollback delays the real investigation.

When rolling back helps

• The deploy touched the code path that is failing, according to the System Map.
• The failure began within minutes of the rollout reaching the affected hosts.
• The previous version ran cleanly under similar traffic in the last week.

When all three held, automatic rollbacks resolved the incident in 94 percent of cases. When only the timing matched, they helped in under a third.

When to wait

Rollbacks are dangerous when a deploy includes a schema migration, a change to a message format or anything other services have already started to depend on. In those cases Keel now recommends a forward fix and explains why, even when undoing would be faster. Teams can override that, and some do, but the recommendation arrives with the risk spelled out.

The lesson reaches beyond rollbacks. An autonomous system earns trust less by acting quickly than by knowing which actions are reversible, and saying so before it takes them.