Builds are being queued
Incident Report for buddybuild
Resolved
The backlog has been processed, so no jobs are currently being queued.
Posted 7 months ago. Oct 22, 2018 - 23:28 UTC
Monitoring
The rate of incoming webhooks has started to subside.

The queue length is still high, but we'll continue to monitor the situation, and will update soon.
Posted 7 months ago. Oct 22, 2018 - 20:54 UTC
Identified
We are observing an excess quantity of incoming webhooks, caused by GitHub processing their backlog of webhooks (https://status.github.com/messages). This has caused very high load on our build systems.

Our system will auto-cancel obsolete jobs to mitigate many rebuilds, but it will take some time for the level of incoming jobs to normalize.
Posted 7 months ago. Oct 22, 2018 - 19:28 UTC
This incident affected: Build Servers (iOS).