The build-processing pipeline has been stable with plenty of capacity headroom. We'll continue to work on the follow-up tasks, but this incident is closed. Thanks again.
Posted 8 months ago. Dec 12, 2017 - 19:30 UTC
The backlog of iOS build jobs has cleared and we are processing builds at full capacity. We'll continue to monitor system closely but we don't anticipate any other regression.
The initial issue was caused by a component failure in our pipeline which caused some build jobs to queue. The failure was realized quickly by alarms but investigation took longer than usual and the eventual recovery caused a thundering herd of build jobs into the processing system at our peak time-of-day. The length of this incident was a result of our system trying to clear the backlog while dealing with our normal peak traffic.
We have added more capacity to the processing system to support peak-load. We will now make the code change to fix the root-cause of the pipeline-component failure and will bolster our toolset to help us resolve issues like this much more quickly in future - these are priorities for the infrastructure team this morning (PST). We will also double-down on reporting incidents to this status page earlier.
Thank you for your patience while we resolved the issue.
Posted 8 months ago. Dec 12, 2017 - 17:58 UTC
We have identified an issue impacting iOS builds, we are working to alleviate the issue and expect to recover shortly.