What we learned from the eu-west backlog

On 29 July our webhook queue fell four minutes behind and stayed there for most of a morning. Checks were fine. Alerting was fine. The thing that broke was the part that tells other people’s systems what we already knew.

Here is the timeline, what we changed, and the two assumptions that turned out to be wrong.