Outage on VM agents on ci.jenkins.io

ci.jenkins.io

[Final message] Outage finished, cleanup operations were done the 30 March with no impact.

[Update - 2023-03-29 15h50 UTC] Build queue cleaned-up two times, and reloaded the state from file system. The instance ssems back to “normal”. Watching it.

[Update - 2023-03-29 15h30 UTC] Build queue did not decreased, the instance is unresponsive: proceeding to purge the queue to regain control, as per https://github.com/jenkins-infra/helpdesk/issues/3474#issuecomment-1488838514

[Update - 2023-03-29 14h00 UTC] Increased the Azure VM capacity to 80 (instead 10 or 20) for each kind (Linux, Windows, etc.) of template

[Update - 2023-03-29 07h30 UTC] The build queue is still full (around 1600 build waiting). Caused by https://github.com/jenkins-infra/helpdesk/issues/3474 (organization scanning trigger).

[Update - 2023-03-28 21h00 UTC] The Azure VM agents are working as expected and started treating builds

[Initial message] There is an outage on ci.jenkins.io since 14h00 UTC. Build requiring a virtual machine are being queued.

More details on https://github.com/jenkins-infra/helpdesk/issues/3459#issuecomment-1487228241.