Resolved after
3h
30m of downtime.
May 30, 2021 at 2:11 AM UTC
A plugin upgrade has caused allocation of Azure Container Instances to fail on ci.jenkins.io. Investigation is in progress to resolve the issue and identify the root cause.
Issue was resolved by downgrading 5 Azure plugins to the versions that were previously working on ci.jenkins.io
Azure Container Instance outage on ci.jenkins.io
Timeline - May 29, 2021
16:37 Denver - Mark Waite updated plugins on ci.jenkins.io, restarted ci.jenkins.io
16:55 Denver - Updated JDK 8 tool definition from 8u242 to 8u292
17:15 Denver - Confirmed builds with JDK 8u292 tools working on amd64, arm64, s390x, and popc64le
17:39 Denver - Started Jenkins core build, detected that maven and maven-11 agents were not being allocated
17:41 Denver - System log for ACI agents reports Cannot provision: template for label maven is not available now, because it failed to provision last time.
18:00 Denver - Downgraded plugin reports NoSuchMethodError, likely indicator that it requires an API in one of the 50+ plugins that were upgraded at 16:37 Denver time
18:01 Denver - Updated Azure container plugin to latest release
18:17 Denver - Confirmed problem is still visible after upgrade
18:19 Denver - Record plugin file system times in case rollback needed
19:14 Denver - Identify candidate plugins for rollback
azure-sdk from 12.vc102aedd3c66 down to 4.vcb202d9010c1
azure-credentials from 182.v3ccd4a755864 down to 177.v816b81058012
azure-vm-agents from 780.v50d067d02f76 down to 774.v0cee503baa25
windows-azure-storage from 358.v5c001416d74f down to 355.v4da08e72a251
azure-container-agents from 207.v3ad9931bf69e down to 201.v2afdce22b4cf
Review dependencies of those plugins
Used plugin installation manager to install a local copy of those 5 plugins at their previous versions as confirmation that they should be workable at their previous versions as a group. Not a perfect check, but worth the check.
20:01 Denver - Downgrade those 5 plugins and restart ci.jenkins.io
20:03 Denver - Restart is complete, Manage Plugins shows those 5 plugins have upgrades available
20:09 DEnver - Outage resolved, ACI agents are allocated on ci.jenkins.io