Engineering6 min read
How we cut our own pager volume by 61 percent
We pointed Keel at our own production for a quarter. The biggest win was not faster fixes, it was the alerts that stopped firing at all.
By Marcus Lindqvist
For the last quarter we ran our own production the way we ask customers to run theirs: Keel connected to every repository, cloud account and monitor, with Watches on a schedule and the on-call rotation unchanged. We wanted to know what an autonomous system actually changes for the people carrying the pager.
The numbers
| Metric | Before | After 90 days |
|---|---|---|
| Pages per week | 38 | 15 |
| Median time to root cause | 47 min | 11 min |
| Alerts closed as noise | 41% | 9% |
The drop in pages came mostly from three places, and only one of them was faster incident response.
Fewer alerts worth ignoring
A large share of our pages came from monitors that fired, resolved themselves and fired again. Keel grouped those flapping monitors, traced most of them to thresholds set years ago for traffic we no longer have, and drafted new thresholds from the last sixty days of data. Engineers reviewed and merged the changes in a single afternoon.
Problems found before they paged
A Watch caught a slow climb in queue depth on our ingest workers eleven days before it would have crossed its alert threshold. The fix was a two-line change to a batch size. Nobody was paged, and nobody had to remember the incident in a retro.
Shorter incidents when they did happen
When pages did fire, the on-call engineer usually found Keel's investigation waiting in the incident channel: the probable cause, the evidence behind it and a drafted fix. Engineers still made every call. They just started from the answer instead of from a dashboard.
We are not claiming every team will see the same curve. Our system is younger than most and our alerts were noisier than we liked to admit. But the pattern held across every service we measured: the quietest weeks came from work done before anything broke.
