A single failed check is not an outage. It is usually a network path having a bad minute, and paging someone about it is how a team learns to ignore its own alerts.
We require agreement from three regions before an incident opens. This is what that costs, and why we think it is worth it.
On 29 July our webhook queue fell four minutes behind and stayed there for most of a morning. Checks were fine. Alerting was fine. The thing that broke was the part that tells other people’s systems what we already knew.
Here is the timeline, what we changed, and the two assumptions that turned out to be wrong.
Stay in the loop
Signup for release alerts, launch news, and short accessibility notes.