Ci.jenkins.io on Jenkins Status Page

ci.jenkins.io enforced login and users cleanup

Wed, 28 Jan 2026 18:15:00 +0000

[Initial Message] On Wednesday 28 January 2026 at 18h15 UTC, ci.jenkins.io will require authentication with a Jenkins LDAP account to allow reading pages.

As part of this change, the users database will be wiped out, effectively removing all API tokens and personal preferences. Direct consequence: incrementals.jenkins.io won’t be able to deliver new incremental releases until a new API token is generated.

[Resolved] ci.jenkins.io disruption

Tue, 13 Jan 2026 05:10:00 +0000

[Final Message] The ci.jenkins.io controller is back to a normal response time. The AI crawler has stopped sending requests. We had to restart the controller once per hour to sustain the high rate of requests.

[Initial message] –> ci.jenkins.io is failing with Out Of Memory errors, we’ve restarted it. We don’t know yet the exact cause but we’re suspecting AI crawlers.

[Resolved] [publick8s cluster] Azure publick8s Upgrade to Kubernetes 1.33.5

Mon, 15 Dec 2025 08:30:00 +0000

[Final Message] upgrade went well, no impact on production.

[Initial Message] Monday 15 December 2025 at 08:30PM UTC, we’ll upgrade the AKS(Azure) cluster publick8s to Kubernetes 1.33.5. It should take between 60 and 120 min.

The expected impacts are:

Restart of the following services, with a downtime between 2 and 15 min (depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-in/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) on:

get.jenkins.io (mirror redirector)
issue-redirect.jenkins.io
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

More details in https://github.com/jenkins-infra/helpdesk/issues/4820#issuecomment-3548729217.

[Resolved] Maintenance on ci.jenkins.io (Security Advisory)

Wed, 10 Dec 2025 11:30:00 +0000

[Final Message] The ci.jenkins.io controller is running with latest plugins and with latest LTS 2.528.3 Core version.

[Initial message] Wednesday December 10 2025, the ci.jenkins.io controller will be put offline and updated as part of a security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/PN3lF_72LfU.

[Resolved] Azure cluster publick8s migrated to a new location and upgraded to Kubernetes 1.32

Thu, 25 Sep 2025 11:30:00 +0000

[Final Message] Outage finished on Friday 26 September 2025 at 09:35am UTC and post mortem published: https://github.com/jenkins-infra/helpdesk/issues/4813#issuecomment-3339518665.

[Update] Friday 26 September 2025 at 09:35am UTC: ongoing production outage on the cluster (unexpected side effect of the migration): https://github.com/jenkins-infra/helpdesk/issues/4813.

[Update] Migration was finished on Thursday 25 September 2025 at 03:30PM UTC. See https://github.com/jenkins-infra/helpdesk/issues/4617#issuecomment-3334826339.

[Initial Message] Thursday 25 September 2025, starting at 11:30AM UTC, we’ll perform a migration + upgrade of the AKS(Azure) cluster publick8s to a new location and to Kubernetes 1.32.

The expected impacts are:

Restart of the following services, with a downtime between 2 and 15 min (depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-in/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) on:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

More details in https://github.com/jenkins-infra/helpdesk/issues/4617.

[Resolved] ci.jenkins.io authentication required

Fri, 19 Sep 2025 12:44:00 +0000

[Final Message] The ci.jenkins.io controller is now up and running, the brownout is done, we will have to work around permission for the next brownout.

More details in https://github.com/jenkins-infra/helpdesk/issues/4768.

[Initial message] A large set of unhelpful requests are overloading ci.jenkins.io. We are in a brownout. We are testing that disallowing anonymous access resolves the performance issue.

More details in https://github.com/jenkins-infra/helpdesk/issues/4768.

[Resolved] Maintenance on ci.jenkins.io (Security Advisory)

Wed, 17 Sep 2025 09:30:00 +0000

[Final Message] The ci.jenkins.io controller is running with latest plugins and with latest LTS 2.516.3 Core version. Weekly.ci is running with latest weekly 2.528 Core version

[Initial message] Wednesday September 17 2025, the ci.jenkins.io controller will be put offline and updated as part of a security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/rOpnrQxJWEY.

[Resolved] Maintenance on ci.jenkins.io (Security Advisory)

Wed, 09 Jul 2025 11:30:00 +0000

Wednesday July 07 2025, the ci.jenkins.io controller will be put offline and updated as part of a security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/doaM4i0Qf6o.

[Resolved] Migration of ci.jenkins.io to AWS

Mon, 07 Jul 2025 07:00:00 +0000

[Final Message] The migration went well, all tests are green.

[Initial Message] Monday July 07 2025, ci.jenkins.io controller will be migrated from Azure to AWS as per https://github.com/jenkins-infra/helpdesk/issues/4688.

We expect between 2h to 4h of downtime starting at 07:00 AM UTC.

[Resolved] jdk21 upgrade for runtime controllers and agents

Thu, 22 May 2025 13:00:00 +0000

[Final Message] migration went well, no impact on production. few fixes to apply with https://github.com/jenkins-infra/jenkins-infra/pull/4143

[Initial Message] Thursday May 22 2025, starting at ~~17:00~~ 13:00 UTC, all our controllers and there agents will be upgraded to use JDK21 as runtime version instead of JDK17.

This operation expects 1 or 2 restarts per controller https://github.com/jenkins-infra/helpdesk/issues/4121#issuecomment-2890355054.

[Resolved] LDAP offline for reindex

Tue, 06 May 2025 17:50:00 +0000

The LDAP service requires a reindex which needs LDAP server to be offline. Starting the operation at 17h50 UTC, finished at 18h30 UTC with success.

[Resolved] [publick8s cluster] Azure publick8s Upgrade to Kubernetes 1.31.6

Thu, 24 Apr 2025 07:30:00 +0000

[Final Message] upgrade went well, no impact on production.

[Initial Message] Thursday 24 April 2025, at 07:30PM UTC, we’ll upgrade the AKS(Azure) cluster publick8s to Kubernetes 1.31.6. It should take between 60 and 120 min.

The expected impacts are:

⚠️ Last Kubernetes upgrade (to 1.30) we saw an outage on update-center (updates.jenkins.io) for a few seconds !

Restart of the following services, with a downtime between 2 and 15 min (depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-in/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) on:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

More details in https://github.com/jenkins-infra/helpdesk/issues/4546#issuecomment-2674545933.

[Resolved] Migration of ci.jenkins.io to Azure

Wed, 09 Apr 2025 08:00:00 +0000

[Final Message] The migration went well, all tests are green.

[Initial Message] Wednesday April 09 2025, at 08:00 AM UTC, the Jenkins Infra team will migrate the ci.jenkins.io controller from AWS to Azure as per https://github.com/jenkins-infra/helpdesk/issues/4620. We expect a downtime of 2 hours.

[Resolved] Maintenance on ci.jenkins.io (Security Advisory)

Wed, 02 Apr 2025 14:45:00 +0000

[Final Message] The ci.jenkins.io controller is running with latest LTS 2.479.3 Core version. The weekly.ci.jenkins.io controller is running with latest weekly 2.504 Core version.

[Initial message] Wednesday April 02 2025, the ci.jenkins.io and weekly.ci.jenkins.io controllers will be put offline and updated as a result of a security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/gjEkmc02cU0.

[Resolved] Maintenance on ci.jenkins.io (Security Advisory)

Wed, 05 Mar 2025 12:30:00 +0000

[Final Message] The ci.jenkins.io controller is running with latest LTS 2.479.2 Core version.

[Initial message] Wednesday March 05 2025, the ci.jenkins.io controller will be put offline and updated as a result of a security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/7ZO66tLSKa8.

[Resolved] [publick8s cluster] Azure publick8s Upgrade to Kubernetes 1.30.7

Thu, 20 Feb 2025 13:30:00 +0000

[Final Message] upgrade went well but we saw an outage on update center for about 2 to 3 mn we will analyse further the logs to try to understand were it’s coming from

[Initial Message] Thurday 20 February 2025, at 01:30PM UTC, we’ll upgrade the AKS(Azure) cluster publick8s to Kubernetes 1.30.7. It should take between 60 and 120 min.

The expected impacts are:

Restart of the following services, with a downtime between 2 and 15 min (depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-in/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) on:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

More details in https://github.com/jenkins-infra/helpdesk/issues/4258#issuecomment-2663553626.

[Resolved] Outage on ci.jenkins.io

Fri, 14 Feb 2025 08:30:00 +0000

Following ci migration to aws, we are encountering an outage on ci.jenkins.io, currently investigating

[Resolved] Migration of ci.jenkins.io to AWS

Thu, 13 Feb 2025 09:00:00 +0000

[Final Message] The migration went well, all tests are green.

[Initial Message]

Now, Thursday February 13 2025, we will start the ci.jenkins.io controller migration to aws as per https://github.com/jenkins-infra/helpdesk/issues/4315#issuecomment-2655944334.

We expect at least 4h of downtime starting at 09:00 AM UTC

[Resolved] ldap outage

Fri, 07 Feb 2025 01:40:00 +0000

[Final Message] The issue was identified to be an undocumented missing permission between Kubernetes and the LDAP data disk during Kubernetes Node upgrade. It has been fixed and persisted.

[Initial Message] ldap service is down so no authentification for Jira, Artifactory, or Jenkins controllers. Help desk 4531 is tracking progress on the issue.

[Resolved] Azure Outage / ci.jenkins.io Windows Container Agents

Wed, 08 Jan 2025 22:00:00 +0000

[Final Message] Outage is finished: back to normal

[Update] still in outage at this date an time 14 january 2025 1:45PM

[Update] The Azure outage finished the 04:30 UTC on 11 Jan 2025 (as per https://azure.status.microsoft/en-us/status/history/, Tracking ID: PLP3-1W8). Windows (container) builds are now flowing on ci.jenkins.io.

[Initial Message] A network Azure outage in the region US East 2 started at 22:00 UTC the 8 January 2025 impacts the Jenkins infrastructure.

Most notably, Windows container agents (using Azure Container Instances) are not starting, associated builds are stuck in the queue.

Short term: users can set useContainerAgent to false in their buildPlugin() pipeline library function to force usage of VMs instead of containers.

More details in https://github.com/jenkins-infra/helpdesk/issues/4490.

[Resolved] AKS cluster upgrade to 1.29.10 (patch)

Wed, 18 Dec 2024 09:45:00 +0000

We’re performing AKS cluster upgrade from 1.29.7 to 1.29.10 the 18 December 2024, starting at 09:45 UTC.

The expected impacts are:

Restart of the following services, with an outage (between 2 to 15 min: depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-In/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

[Resolved] Maintenance on ci.jenkins.io (Security Advisory)

Wed, 27 Nov 2024 12:30:00 +0000

[Final Message] The ci.jenkins.io controller is running with latest LTS 2.479.2 Core version.

[Initial message] Wednesday November 27 2024, the ci.jenkins.io controller will be put offline and updated as a result of a security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/KPuGGzCkWX0.

[Resolved] Maintenance on ci.jenkins.io (Datadog plugin upgrade)

Thu, 14 Nov 2024 08:30:00 +0000

Thursday November 14 2024, the ci.jenkins.io controller will be restarted to upgrade the datadog plugin as per https://github.com/jenkins-infra/helpdesk/issues/4377.

If this operation ends up in corrupting the build log like the previous ones, we’ll immediately restore the backup and restart the controller.

[Resolved] Maintenance on ci.jenkins.io (Security Advisory)

Wed, 13 Nov 2024 15:30:00 +0000

[Final Message] The ci.jenkins.io controller is running with updated plugins.

[Initial message] Wednesday November 13 2024, the ci.jenkins.io controller will be put offline and restarted if plugin updates are needed as a result of a plugin security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/UmiH1ZpJez4.

[Resolved] Maintenance on ci.jenkins.io

Wed, 02 Oct 2024 11:19:00 +0000

Jenkins core update is being applied for the Jenkins security advisory.

[Resolved] Azure VMs outage

Mon, 09 Sep 2024 15:00:00 +0000

[Final message]

Azure did fix the problem in two steps:

Additional capacity added in US East 2 the 12 September
Software fix the 13 September

[Update on 2024-09-10]

The problem was gone from 2024-09-09 around 11:00pm UTC but it’s back since 2024-09-10 around 02:00pm UTC
Our Azure Service Health Dashboard mentions 2 mixed incidents, one should be fixed the 12!

[initial message] Since Monday 9 September 2024 at 03:00pm UTC, we see an increased rate of failure to spin up Azure VM agents on ci.jenkins.io

Details in https://github.com/jenkins-infra/helpdesk/issues/4287.

[Resolved] Update Azure AKS cluster `publick8s` to 1.29

Fri, 23 Aug 2024 09:00:00 +0000

Friday 23 August at 07:30am UTC, the AKS cluster publick8s will be upgraded to Kubernetes 1.29.7. It should take between 30 and 60 min.

The expected impacts are:

Restart of the following services, with a downtime between 2 and 15 min (depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-in/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) on:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

More details in https://github.com/jenkins-infra/helpdesk/issues/4161.

[Resolved] Maintenance on ci.jenkins.io

Thu, 22 Aug 2024 13:00:00 +0000

Thursday August 22 2024, the ci.jenkins.io controller will be restarted to upgrade the datadog plugin as per https://github.com/jenkins-infra/helpdesk/issues/4236.

If this operation ends up in corrupting the build log like the previous ones, we’ll immediately restore the backup and restart the controller.

[Resolved] Maintenance on ci.jenkins.io

Wed, 07 Aug 2024 09:00:00 +0000

[Final Message] The ci.jenkins.io controller is running with updated plugins and Core.

[Initial message] Wednesday August 07 2024, the ci.jenkins.io controller will be put offline and restarted if plugin updates are needed as a result of a plugin security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/NkXElf7MmcI.

[Resolved] ldap.jio and repo.azure.jio workloads migration to arm64 (publick8s cluster)

Fri, 05 Jul 2024 14:30:00 +0000

We’re doing a workload migration to arm64 on the publick8s AKS cluster starting at 14:30 UTC.

The expected impacts are:

Restart of the following services, with an outage (between 2 to 15 min: depends on Azure Storage systems):

ldap.jenkins.io
repo.azure.jenkins.io (ACP)

Temporary disruptions (no Log-In/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

More details on https://github.com/jenkins-infra/helpdesk/issues/3837#issuecomment-2210532769.

[Resolved] Update azure clusters publick8s Kubernetes version to 1.28

Thu, 04 Jul 2024 06:30:00 +0000

[Final Message] The operation finished successfully at 7:15am UTC.

More details on https://github.com/jenkins-infra/helpdesk/issues/4144#issuecomment-2196274746.

[Initial message] We’re doing an publick8s AZ cluster upgrade from 1.27 to 1.28 tomorrow, 4th of July 2024, starting at 6:30 UTC (8h30 paris time, 12h00 IST).

The expected impacts are:

Restart of the following service, with an outage (between 2 to 15 min: depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-In/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) due to public IP metadata changes and falco + Nginx upgrades:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

[Resolved] High rate of HTTP/502 errors on ci.jenkins.io

Mon, 01 Jul 2024 09:00:00 +0000

[Final message] The issue has been resolved after a few restarts (and fixes of course!)

[Initial message] We are investigating a high rate of HTTP/502 errors on ci.jenkins.io. We’ll keep you posted.

Expect multiple restarts and outage.

More details in https://github.com/jenkins-infra/helpdesk/issues/4158.

[Resolved] Maintenance on ci.jenkins.io

Wed, 26 Jun 2024 11:30:11 +0000

[Final Message] The ci.jenkins.io controller is running with updated plugins.

[Update] The update is taking longer than planned, expect a restart around 16h30 UTC.

[Initial message] Wednesday June 26 2024, the ci.jenkins.io controller will be put offline and restarted if plugin updates are needed as a result of the plugin security advisory.

The restart may happen between 11:30am UTC and 15:30 UTC.

More details in https://groups.google.com/g/jenkinsci-advisories/c/MokuNdWXGfg.

[Resolved] Downgrading `datadog` plugin on ci.jenkins.io

Mon, 06 May 2024 12:00:00 +0000

As per https://github.com/jenkins-infra/helpdesk/issues/4080#issuecomment-2095831304, ci.jenkins.io will be down for an operation trying to fix the corrupted builds.

A first tentative will be done to downgrade the datadog plugin, if it fails then the plugin will be removed.

[Resolved] Maintenance on ci.jenkins.io

Thu, 02 May 2024 10:53:00 +0000

Plugin updates are being applied for the Jenkins security advisory.

[Resolved] Maintenance on ci.jenkins.io

Tue, 23 Apr 2024 16:00:00 +0000

The ci.jenkins.io controller will have builds waiting in the queue during a maintenance for configuration upgrade. Until the Azure credentials are fully rotated, no VM agent will be spawned. Container agents will continue to work as expected.

More details in https://github.com/jenkins-infra/helpdesk/issues/4052.

[Resolved] Artifact Caching Proxy deployment of 1.0.0

Thu, 28 Mar 2024 07:30:00 +0000

[Final message]

Upgrade successfull

See https://github.com/jenkins-infra/helpdesk/issues/3969#issuecomment-2024586947 for details.

[Initial message]

The Artifact Caching Proxy system used by ci.jenkins.io and infra.ci.jenkins.io agents to cache Maven artifacts will upgraded to version 1.0.0 the 28 March 2024 at 07:30am UTC.

Expect eventual slowness or failures in Maven dependency resolutions in the next hour on ci.jenkins.io and infra.ci.jenkins.io.

See https://github.com/jenkins-infra/helpdesk/issues/3969#issuecomment-2024586947 for details.

[Resolved] Update azure clusters publick8s Kubernetes version to 1.27

Tue, 19 Mar 2024 08:30:00 +0000

[Final Message] The operation finished successfully at 9:35am UTC.

More details on https://github.com/jenkins-infra/helpdesk/issues/3948#issuecomment-1991181964.

[Initial message] We’re doing an publick8s AZ cluster upgrade from 1.26 to 1.27 today 19th of Mars 2024, starting at 8:30 UTC (9h30 paris time).

The expected impacts are:

Restart of the following service, with an outage (between 2 to 15 min: depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-In/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) due to public IP metadata changes and falco + Nginx upgrades:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

[Resolved] ci.jenkins.io is down

Wed, 07 Feb 2024 14:40:00 +0000

[Final Message]

The ci.jenkins.io controller is up again.

[Initial message]

The ci.jenkins.io controller is down, we’re working on it.

[Resolved] Migration of ci.jenkins.io VM to another location

Thu, 25 Jan 2024 12:30:00 +0000

A maintenance on ci.jenkins.io will be performed with a full service disruption the Thursday 25 January 2024 starting at 12:30pm (noon time) UTC.

The goal of this maintenance is to migrate ci.jenkins.io VM to a new location in Azure.

See https://github.com/jenkins-infra/helpdesk/issues/3913 for more details.

[Resolved] Maintenance on ci.jenkins.io

Wed, 24 Jan 2024 11:00:00 +0000

[Final Message]

The ci.jenkins.io controller has been updated and is back online.

[Initial message]

The ci.jenkins.io controller is down for maintenance as part of a security upgrade announced in

https://groups.google.com/g/jenkinsci-advisories/c/QZiecB2ArMs/m/sDvJWIdHBQAJ

[Resolved] LDAP (ldap.jenkins.io) maintenance

Fri, 19 Jan 2024 08:45:00 +0000

Friday January 19 2024, starting at 08:45am UTC, we’ll proceed to a maintenance on the LDAP service (ldap.jenkins.io) hosted in the publick8s Azure Kubernetes cluster.

Authentication on all of the *.jenkins.io and *.jenkins-ci.orf services will be unavailable during the operation. We expect a disruption between 5 and 30 min.

The goal is to update the LDAP image to a new version to use a new deployment system.

More details in https://github.com/jenkins-infra/helpdesk/issues/3837#issuecomment-1898378430.

[Resolved] Maintenance on ci.jenkins.io

Wed, 13 Dec 2023 12:54:47 +0000

Outage is being investigated. High CPU load was detected.

[Resolved] Operation on Jenkins Azure VM Controllers

Wed, 22 Nov 2023 14:30:00 +0000

Wednesday November 22 2023, starting at 14:30 UTC, an operation on the Jenkins Azure VM controllers will happen.

No expected impact except delays in the Azure VM and ACI container agent allocation.

[Resolved] Update publick8s AKS cluster to Kubernetes 1.26

Thu, 09 Nov 2023 08:30:00 +0000

[Final Message] The operation finished successfully at 11:00am UTC. The monitoring showed an outage of 6 minutes on the LDAP and 2 minutes on the incremental publisher.

More details on https://github.com/jenkins-infra/helpdesk/issues/3683#issuecomment-1803566680.

[Initial Message] The AKS cluster publick8s will be upgraded from Kubernetes 1.25 to 1.26 the Thursday 09th of November 2023, starting at 08:30am UTC.

The expected impacts are:

Restart of the following service, with an outage (between 2 to 15 min: depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-In/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) due to public IP metadata changes and falco + Nginx upgrades:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

[Resolved] Update of the AWS EKS Kubernetes clusters to 1.26

Mon, 06 Nov 2023 15:00:00 +0000

[Final Message] The operation finished successfully at 16:45 UTC.

[Initial Message] An upgrade of the AWS EKS Kubernetes clusters from 1.25 to 1.26 is scheduled the Monday 06 November 2023 at 15:00 UTC.

Expected impact on ci.jenkins.io:

Usual plugin build workloads using Linux containers will be run in DigitalOcean clusters
No BOM builds can be run until the operation is finished

More details in https://github.com/jenkins-infra/helpdesk/issues/3683#issuecomment-1794913444.

[Resolved] Update of the Digital Ocean Kubernetes clusters to 1.26

Thu, 26 Oct 2023 12:00:00 +0000

[Final Message] The operation finished successfully at 12:45 UTC.

[Initial Message] An upgrade of the DigitalOcean Kubernetes clusters from 1.25 to 1.26 is scheduled the Thursday 26 October 2023 at 12:00 (noon) UTC.

No visible impact is expected on ci.jenkins.io as the container agent workloads will be taken over by the AWS EKS cluster instead of Digital Ocean starting at 11:00 UTC.

More details in https://github.com/jenkins-infra/helpdesk/issues/3683#issuecomment-1780857936.

[Resolved] Maintenance on ci.jenkins.io

Wed, 20 Sep 2023 11:30:00 +0000

A maintenance on ci.jenkins.io will be performed with a full service disruption the Wednesday 20 of September 2023 between 11h30 and 14h30 UTC.

[Resolved] Brownout of the Maven Central mirrored repositories in JFrog Artifactory (repo.jenkins-ci.org)

Mon, 04 Sep 2023 12:00:00 +0000

[Final Message] Brownout is finished.

[Initial Message]

Monday 04 of September 2023, we’ll perform a brownout operation of 4 hours on the Jenkins Artifactory service at repo.jenkins-ci.org at 10:00 UTC.

This operation consists in removing all mirrors of the Maven Central artifact repository from repo.jenkins-ci.org to force Maven builds to download mirrored artefacts from the Maven central instead of our Artifactory instance.

This planned brownout is the 2nd tentative related to https://github.com/jenkins-infra/helpdesk/issues/3599 which goal decrease the amount of outbound bandwidth download of our instance.

[Resolved] Maintenance on ci.jenkins.io

Sat, 02 Sep 2023 14:45:00 +0000

A plugin update to ci.jenkins.io did not complete successfully. The server is currently reporting that it is restarting.

[Resolved] Maintenance on ci.jenkins.io, trusted.ci.jenkins.io and cert.ci.jenkins.io

Mon, 21 Aug 2023 13:45:00 +0000

[Final Message] Operation completed successfully:

Took around 30 minutes as expected.
A few minor unexpected behaviors are tracked in https://github.com/jenkins-infra/helpdesk/issues/3072 (fixed the 22 of October 2023).

[Initial Message] Monday August 21 2023, starting at 13:45 UTC, the Jenkins controllers ci.jenkins.io, trusted.ci.jenkins.io and cert.ci.jenkins.io will be upgraded to use JDK17 instead of JDK11.

This operation expects 1 or 2 restarts per controller but then should be harmless as we worked on the agent images as per https://github.com/jenkins-infra/helpdesk/issues/3072 and already upgraded 3 other controllers.

[Resolved] Maintenance on ci.jenkins.io and trusted.ci.jenkins.io

Thu, 17 Aug 2023 17:00:00 +0000

Thursday August 17 2023, starting at 16:30 UTC, an infrastructure operation on the https://ci.jenkins.io and https://trusted.ci.jenkins.io controller will be done. This operation impacts the network of both services which could lead to network errors when accessing the services and when running build.

The operation took 1 hour and finished with success.

More details in https://github.com/jenkins-infra/azure/pull/454

[Resolved] Maintenance on ci.jenkins.io

Wed, 16 Aug 2023 11:30:00 +0000

[Final Message] The ci.jenkins.io controller is running with updated plugins and core version.

[Initial message] Wednesday August 16 2023, between 11:30 UTC and 13:30 UTC, the https://ci.jenkins.io controller will be unavailable as it will be upgraded and restarted as part of the security advisory announced in https://groups.google.com/g/jenkinsci-advisories/c/xOqgDOYxTAw.

[Resolved] Brownout of the Maven Central mirrored repositories in JFrog Artifactory (repo.jenkins-ci.org)

Wed, 02 Aug 2023 14:30:00 +0000

[Final Message] Brownout is finished after 1h30 (instead of initial 1 hour).

[Initial Message] Wednesday 02 of August, we’ll perform a brownout operation of 1 hour on the Jenkins Artifactory service at repo.jenkins-ci.org at 14:30 UTC for ~1 hour.

This planned brownout is a step of https://github.com/jenkins-infra/helpdesk/issues/3599 to decrease the amount of outbound bandwidth download of our instance.

There should be no functional impact, as Maven as a built-in fallback to the repository central. However, we expect the builds on ci.jenkins.io to be a bit slower as the artifacts downloaded from central won’t be cached anymore.

[Resolved] Maintenance on ci.jenkins.io

Wed, 26 Jul 2023 12:45:00 +0000

[Final Message] The ci.jenkins.io controller is running with updated plugins and core version.

[Initial message]

Wednesday July 26 2023, between 10:30 UTC and 14:30 UTC, the ci.jenkins.io controller will be unavailable as it will be upgraded and restarted as part of the security advisory announced in https://groups.google.com/g/jenkinsci-advisories/c/2GiSsACaT28.

[Resolved] Maintenance on ci.jenkins.io

Wed, 12 Jul 2023 11:38:11 +0000

[Final Message] The ci.jenkins.io controller is running with updated plugins.

[Initial message] Wednesday July 12, 2023 the ci.jenkins.io controller will be restarted if plugin updates are needed as a result of the plugin security advisory.

The restart may happen between 11:30am UTC and 13:30 UTC.

More details in https://groups.google.com/g/jenkinsci-advisories/c/y50ITBfDesw.

[Resolved] Maintenance on ci.jenkins.io

Tue, 04 Jul 2023 13:00:00 +0000

[Final Message] Everything is back to normal on ci.jenkins.io. Follows up in https://github.com/jenkins-infra/helpdesk/issues/3535.

[Update 3] At 21h30 UTC, all builds (except BOM) were able to run, with some runtime errors in some cases due to missing plugins.

[Update 3] At 20h45 UTC, the migration was finished and ci.jenkins.io was available again.

[Update 2] ci.jenkins.io remains down until 5th of July due to I/O issues on the old JENKINS_HOME. Ref. https://github.com/jenkins-infra/helpdesk/issues/3535#issuecomment-1620171502.

[Update 1] Kubernetes 1.25 upgrade on AWS went well and was finished at 15:00 UTC.

[Initial message] Maintenance on ci.jenkins.io is planned the Tuesday 4th of July 2023:

Initial operation starts at 13:00 UTC with an AWS Kubernetes upgrade to 1.25.
- Expected impact is no BOM build will be possible during the maintenance while other builds executed in DigitalOcean and/or Azure.
- More details in https://github.com/jenkins-infra/helpdesk/issues/3582.
Second part of the operation starts at 14h30 UTC for a migration of the virtual machine to a new generation of hardware (Azure VM Gen2) and operating system (Ubuntu 22.04).
- Expected impact is a full downtime during the migration.
- More details in https://github.com/jenkins-infra/helpdesk/issues/3535.

[Resolved] DigitalOcean outage

Wed, 21 Jun 2023 11:45:00 +0000

[Final message] Incident is reported officially by Digital Ocean and is finished: https://status.digitalocean.com/incidents/bclmy32d12p0.

We’ve restored all the nominal configuration after health-checking the clusters.

[Initial message] DigitalOcean infrastructure is having a network outage. We see a high rate of TCP errors on both Kubernetes clusters doks (agents for ci.jenkins.io) and doks-public (Artifact Caching Proxy for these agents). We are disabling temporarily agent scheduling in DigitalOcean: all ci.jenkins.io containers workload will run in AWS or Azure instead.

More details in https://github.com/jenkins-infra/helpdesk/issues/3633.

[Resolved] Maintenance on ci.jenkins.io

Wed, 14 Jun 2023 12:00:00 +0000

[Final message] The maintenance is ended. Advisory published: https://www.jenkins.io/security/advisory/2023-06-14/.

[Initial message] Maintenance on ci.jenkins.io, down for security advisory. (ref. https://groups.google.com/g/jenkinsci-advisories/c/CD7La5U-dIQ).

[Resolved] Outage on VM agents on ci.jenkins.io

Fri, 09 Jun 2023 12:37:00 +0000

[Final message] revert disabled plugin and restart ci.jenkins.io

[Initial message] There is an outage on ci.jenkins.io since 12h35 UTC. following plugin removal there is a configuration as code error message.

https://github.com/jenkins-infra/helpdesk/issues/3573#issuecomment-1584503523

[Resolved] Brownout of the 'jgit' mirrored repository in JFrog Artifactory (repo.jenkins-ci.org)

Thu, 08 Jun 2023 12:30:00 +0000

[Final message] Our experiment is now finished. Projects using the Eclipse jgit had errors downloading artefacts during the operation but it is back to normal

[Initial message] The mirror repository jgit in the JFrog Artifactory service <repo.jenkins-ci.org> will be restricted to only authenticated users the 8 June 2023 at 12:30UTC for 30 min.

This planned brownout is a step of https://github.com/jenkins-infra/helpdesk/issues/3599 to study the behavior of our builds.

There should be no impact, as the artifacts should be consumed through the public virtual repository but we cannot be 100% sure, hence this operation.

[Resolved] ci.jenkins.io scans outage

Mon, 05 Jun 2023 14:00:00 +0000

A change in the GitHub REST API is responsible for a bug in the integration used by Jenkins to retrieve repository maintainers. Repository scans on ci.jenkins.io fail to retrieve pull requests.

The issue will require a new release of the GitHub API library and the GitHub API plugin that packages the library as a Jenkins plugin. A patched GitHub API has been installed on ci.jenkins.io while waiting for the release of the library and the plugin.

See the help desk issue to track progress.

[Resolved] Maintenance on ci.jenkins.io

Tue, 16 May 2023 11:00:00 +0000

[Final message] The maintenance is ended.

[Initial message] Maintenance on ci.jenkins.io, down for security advisory. (ref. https://groups.google.com/g/jenkinsci-advisories/c/_mJEHe9lXTU).

[Resolved] Outage for container agents on ci.jenkins.io

Tue, 25 Apr 2023 13:00:00 +0000

[Final Message] A new cluster was created and used to replace the former one, with additional measures to avoid future lock out. Follow up in https://github.com/jenkins-infra/helpdesk/issues/3542.

[Updated]

Root cause found: the problem was caused by a mix of technical and human errors which led to a lockout of the AWS EKS cluster used by ci.jenkins.io.
The cluster requires a re-creation, which can take 1 to 2 hours.
A remediation was applied: DigitalOcean is now used for container workload until the operation is finished: https://github.com/jenkins-infra/jenkins-infra/commit/2bfbb53123b43317d91a280ce8b8c9ae4c8b35a2

[Initial message] ci.jenkins.io is not able to start container agents.

The builds requiring a container agents are queued, while the other are executed as usual.

[Resolved] Maintenance on ci.jenkins.io

Wed, 12 Apr 2023 11:00:00 +0000

[Final message] The maintenance is ended.

[Initial message] Maintenance on ci.jenkins.io, down for security advisory. (ref. https://groups.google.com/g/jenkinsci-advisories/c/ccmIF1Vs-9A).

[Resolved] Outage on VM agents on ci.jenkins.io

Tue, 28 Mar 2023 14:00:00 +0000

[Final message] Outage finished, cleanup operations were done the 30 March with no impact.

[Update - 2023-03-29 15h50 UTC] Build queue cleaned-up two times, and reloaded the state from file system. The instance ssems back to “normal”. Watching it.

[Update - 2023-03-29 15h30 UTC] Build queue did not decreased, the instance is unresponsive: proceeding to purge the queue to regain control, as per https://github.com/jenkins-infra/helpdesk/issues/3474#issuecomment-1488838514

[Update - 2023-03-29 14h00 UTC] Increased the Azure VM capacity to 80 (instead 10 or 20) for each kind (Linux, Windows, etc.) of template

[Update - 2023-03-29 07h30 UTC] The build queue is still full (around 1600 build waiting). Caused by https://github.com/jenkins-infra/helpdesk/issues/3474 (organization scanning trigger).

[Update - 2023-03-28 21h00 UTC] The Azure VM agents are working as expected and started treating builds

[Initial message] There is an outage on ci.jenkins.io since 14h00 UTC. Build requiring a virtual machine are being queued.

More details on https://github.com/jenkins-infra/helpdesk/issues/3459#issuecomment-1487228241.

[Resolved] Maintenance on ci.jenkins.io

Tue, 21 Mar 2023 12:00:00 +0000

[Final message] The maintenance is ended.

[Initial message] Maintenance on ci.jenkins.io, down for security advisory. (ref. https://groups.google.com/g/jenkinsci-advisories/c/RmkqB1wAY14).

[Resolved] Maintenance Window on ci.jenkins.io

Fri, 17 Mar 2023 18:27:45 +0000

Maintenance of ci.jenkins.io by the infrastructure team.

[Resolved] Maintenance on ci.jenkins.io

Wed, 08 Mar 2023 13:00:00 +0000

[Final message] The maintenance is ended.

[Initial message] Maintenance on ci.jenkins.io, down for security advisory (ref. https://groups.google.com/g/jenkinsci-advisories/c/RmkqB1wAY14).

[Resolved] Maintenance on ci.jenkins.io

Wed, 15 Feb 2023 11:55:00 +0000

[Final message] The maintenance is ended.

[Initial message] Maintenance on ci.jenkins.io, down for plugin security advisory.

[Resolved] s390x agent restart on ci.jenkins.io

Fri, 03 Feb 2023 16:00:00 -0500

The IBM System 390 agent connected to ci.jenkins.io may be restarted during a maintenance window Friday 3 Feb 2023 from 4pm to 10pm ET.

[Resolved] Outage on ci.jenkins.io

Tue, 24 Jan 2023 12:33:00 +0000

Updates for plugin security advisory.

[Resolved] Outage on ci.jenkins.io

Tue, 27 Dec 2022 13:32:00 +0000

Stopping a large scale job BOM PR-1625 was not ending the jobs and was not deleting agents. Restarted to release the jobs and delete the more than 150 agents that have been idling for 20 hours.

[Resolved] Maintenance with downtime of JFrog Artifactory (repo.jenkins-ci.org)

Sun, 18 Dec 2022 09:00:00 +0000

[Final message] The maintenance has been completed with success by JFrog.

repo.jenkins-ci.org is back online, we resumed ci.jenkins.io and trusted.ci.jenkins.io builds.

[Initial message] December 18, 2022 at 09:00am UTC: our sponsor JFrog will proceed to perform maintenance of our “jenkinsci” Artifactory instance.

Expect a complete downtime of 1 to 6 hours due to the nature of this maintenance. The maintenance involves cloud migration from Google Cloud to Amazon Web Services.

Only Jenkins contributors will be impacted, as no releases or builds on ci.jenkins.io or on developer machines will succeed during the downtime. Jenkins users won’t be impacted.

Please note that ci.jenkins.io will be in “maintenance mode” during the operation: new incoming builds will stay in the build queue while the maintenance mode is enabled, to ensure that no build is lost.

More information in https://github.com/jenkins-infra/helpdesk/issues/3288.

[Resolved] Outage on ci.jenkins.io

Tue, 13 Dec 2022 10:16:00 +0000

[End of outage message] service is back, analysing logs

[Initial message] ci.jenkins.io is not responding

[Resolved] Maintenance on ci.jenkins.io

Tue, 15 Nov 2022 12:30:00 +0000

[Final message]
The maintenance is ended, ci.jenkins.io is back online.

[Initial message]
A maintenance on ci.jenkins.io has been started.

[Resolved] Maintenance on ci.jenkins.io

Wed, 02 Nov 2022 10:00:00 +0000

[Final message] The maintenance is ended, ci.jenkins.io is back online.

[Initial message] A maintenance on ci.jenkins.io is planned the 2 of November at 10:00 UTC.

[Resolved] Maintenance on ci.jenkins.io

Tue, 01 Nov 2022 13:00:00 +0000

[Final message] The maintenance is ended, ci.jenkins.io is back online.

[Initial message] A maintenance on ci.jenkins.io is planned the 1st of November at 13:00 UTC.

[Resolved] Maintenance on ci.jenkins.io

Wed, 19 Oct 2022 11:30:00 +0000

[Final message]
The maintenance is ended, ci.jenkins.io is back online.

[Initial message]
A maintenance on ci.jenkins.io has been started.

[Resolved] Maintenance on ci.jenkins.io

Tue, 23 Aug 2022 11:30:00 +0000

[Final message] The maintenance is finished, ci.jenkins.io is back online.

[Initial message] A maintenance on ci.jenkins.io has been started.

[Resolved] Maintenance on ci.jenkins.io

Wed, 27 Jul 2022 11:30:00 +0000

[Final message]
The maintenance is finished, ci.jenkins.io is back online.

[Initial message]
A maintenance on ci.jenkins.io has been started.

[Resolved] Maintenance Window on ci.jenkins.io

Wed, 06 Jul 2022 10:00:00 +0000

[Initial message] The Jenkins Infra. team is going to perform a set of maintenances on ci.jenkins.io as part of https://github.com/jenkins-infra/helpdesk/issues/3031, https://github.com/jenkins-infra/helpdesk/issues/2984, and https://github.com/jenkins-infra/helpdesk/issues/2961.

These actions require multiple controller restart: that should either queue, cancel or fail the running builds.

[Resolved] Maintenance Window on ci.jenkins.io

Wed, 22 Jun 2022 11:30:00 +0000

[Final message] The maintenance is finished, ci.jenkins.io is back online.

[Initial message] A maintenance on ci.jenkins.io has been started.

[Resolved] Maintenance Window on ci.jenkins.io

Tue, 17 May 2022 11:31:00 +0000

An upgrade of the plugins of ci.jenkins.io has been started.

See the 2022-05-17 security advisory.

[Resolved] Outage on ci.jenkins.io

Mon, 02 May 2022 18:15:00 +0000

The ci.jenkins.io service saturated its CPU with Java JIT compilation and Pipeline library loading. Diagnostic information was captured and the service was restarted.

[Resolved] Outage on ci.jenkins.io

Sun, 01 May 2022 12:58:27 +0000

Undiagnosed outage of ci.jenkins.io service. Service was restored without intervention.

[Resolved] Disruptions on ci.jenkins.io

Wed, 20 Apr 2022 16:45:00 +0000

[Closing Incident] The service ci.jenkins.io is now performing as expected. A post-mortem is expected in the upcoming days, please follow up on the issue https://github.com/jenkins-infra/helpdesk/issues/2893 if you are interested.

[Opening Incident] There are disruptions on the service ci.jenkins.io, see https://github.com/jenkins-infra/helpdesk/issues/2893 for more details and progress.

[Resolved] Outage on ci.jenkins.io

Tue, 12 Apr 2022 11:07:00 +0000

The ci.jenkins.io service went offline and returned to service as reported by uptime robot.

[Resolved] Maintenance Window on ci.jenkins.io

Tue, 15 Mar 2022 13:45:00 +0000

An upgrade of the plugins of ci.jenkins.io has been started. More informations to come.

[Resolved] Maintenance window on ci.jenkins.io (2022-02-15)

Tue, 15 Feb 2022 13:30:00 +0000

The 2022-02-15 at 13:30 UTC, expect an outage on ci.jenkins to update the plugins installed on this instance.

Security advisory is available at https://www.jenkins.io/security/advisory/2022-02-15/.

[Resolved] Maven agents not allocating on ci.jenkins.io

Fri, 11 Feb 2022 02:14:00 +0000

[Update 11th of February 2022 - 19:00 UTC] An operation is required on the underlying Kubernetes cluster. We plan to do it monday 14th starting at 08:00am UTC.

It will result in a downtime of all the agents with labels maven and maven-11: builds will be queued until the cluster is working again. VM agents will keep being allocated.

[Update 11th of February 2022 - 15:20 UTC] Cluster had been scaled manually to handle builds until the root cause is fixed. Read more at https://github.com/jenkins-infra/helpdesk/issues/2780.

[Initial Message] Agents with labels maven and maven-11 fail to allocate on AWS. Infrastructure team found that the AWS autoscaler was failing. Investigation is in progress.

[Resolved] Maintenance window on ci.jenkins.io

Wed, 09 Feb 2022 12:14:28 +0000

An upgrade of ci.jenkins.io from 2.319.2 to 2.319.3. See security advisory, changelog, and upgrade guide.

[Resolved] ci.jenkins.io down due to overload

Fri, 21 Jan 2022 17:56:43 +0000

Overload detected on ci.jenkins.io required a service restart. Reported to Mark Waite email by uptimerobot.

[Resolved] Disruptions on ci.jenkins.io

Thu, 13 Jan 2022 15:00:00 +0000

There are disruptions on the service ci.jenkins.io, see https://github.com/jenkins-infra/helpdesk/issues/2733 for more details and progress.

[Resolved] Maintenance Window on ci.jenkins.io

Wed, 12 Jan 2022 14:44:54 +0000

An upgrade of ci.jenkins.io from 2.319.1 to 2.319.2. See security advisory, changelog, and upgrade guide.

[Resolved] Maintenance Window on ci.jenkins.io

Thu, 04 Nov 2021 11:24:40 +0000

An upgrade of ci.jenkins.io from 2.303.2 to 2.303.3. See security advisory, changelog, and upgrade guide.

[Resolved] Maintenance Window on ci.jenkins.io

Wed, 06 Oct 2021 15:45:00 +0000

Jenkins core and plugins were updated on ci.jenkins.io with security fixes from Jenkins security advisory 2021-10-06.

[Resolved] LDAP server down

Fri, 01 Oct 2021 00:00:00 +0000

[Final Message] The root certificate have been rotated, resulting in an automatic repair of all broken services.

archives.jenkins.io was back online at 09:55 UTC
ci.jenkins.io was back online at 09:55 UTC
ldap was back online at 09:55 UTC

[Initial Message] Due to DST Root CA X3 Expiration our LDAP server is currently down, we are working to fix it.

[Resolved] Puppet Outage on VM-hosted services

Thu, 23 Sep 2021 10:01:00 +0000

[Final Message] The Puppet infrastructure had been fixed, resulting in an automatic repair of all broken services.

archives.jenkins.io was back online at 12:58 UTC
ci.jenkins.io was back online at 13:00 UTC

[Initial Message] An outage on our Puppet infrastructure system is currently happening.

A mitigation had been applied resulting in disabling configuration update to avoid breaking services.

The following services had been put down before we could mitigate:

ci.jenkins.io
archives.jenkins.io

We are working on fixing these services and the Puppet infrastructure

[Resolved] Maintenance Window on ci.jenkins.io

Tue, 31 Aug 2021 12:36:00 +0000

Plugins were updated on ci.jenkins.io with security fixes from Jenkins security advisory 2021-08-31.

[Resolved] Maintenance Window on ci.jenkins.io

Mon, 21 Jun 2021 10:04:00 +0000

Update: operation took 2 minutes, and restart was successful

==============================

A restart of ci.jenkins.io is planned with the following changes:

https://issues.jenkins.io/browse/INFRA-3010 => Remove the plugin azure-commons
Plugins upgrade:
- Bump ec2-plugin to 1.62
- Bump Azure VM to 783.v58077630847d
- Bump Code Coverage API to 1.4.0
- Bump JaCoCo to 3.3.0
- Bump Pipeline: Multibranch to 2.26
- Bump Pipeline: SCP Step to 2.13
- Bump Pipeline: Shared Groovy Libraries to 2.20

[Resolved] Agents randomly disconnected on ci.jenkins.io

Tue, 08 Jun 2021 05:35:00 +0000

Update at 2021-06-08 11:20 UTC

ci.jenkins.io is back to normal since 2021-06-08 11:08 UTC after running a series of operations including a service full restart.

Initial Message at 2021-06-08 09:35 UTC

Agents are randomly disconnected on ci.jenkins.io with the following message in pipeline logs:

Channel "hudson.remoting.Channel@<ID>:JNLP4-connect connection from <IP>:<IP:PORT>": Remote call on JNLP4-connect connection from <IP>:<IP:PORT> failed. The channel is closing down or has closed down

Builds using an EC2 or a permanent agents are reconnected most of the time but might fail
Builds on fungible agents such as ACI are failing 100% when this error happens in build logs and need to be replayed

Please note that ci.jenkins.io might be restarted multiple time until the incident is disconnected.

[Resolved] ACI agents fail to allocate for ci.jenkins.io

Thu, 03 Jun 2021 15:41:00 +0000

ci.jenkins.io is unable to start any ACI agent and there is an increasing build queue.

Investigation is in progress to resolve the issue and identify the root cause, ci.jenkins.io is put in shutdown mode to limit the amount of builds to be treated.

[edit]

Plugin and configuration upgrades around EC2 and Azure-* have been done
all the builds have been cleaned
ci.jenkins.io restarted and back to normal state

[Resolved] ACI agents fail to allocate for ci.jenkins.io

Sat, 29 May 2021 22:41:00 +0000

A plugin upgrade has caused allocation of Azure Container Instances to fail on ci.jenkins.io. Investigation is in progress to resolve the issue and identify the root cause.

Issue was resolved by downgrading 5 Azure plugins to the versions that were previously working on ci.jenkins.io

Azure Container Instance outage on ci.jenkins.io

Timeline - May 29, 2021

16:37 Denver - Mark Waite updated plugins on ci.jenkins.io, restarted ci.jenkins.io
16:55 Denver - Updated JDK 8 tool definition from 8u242 to 8u292
17:15 Denver - Confirmed builds with JDK 8u292 tools working on amd64, arm64, s390x, and popc64le
17:39 Denver - Started Jenkins core build, detected that maven and maven-11 agents were not being allocated
17:41 Denver - System log for ACI agents reports Cannot provision: template for label maven is not available now, because it failed to provision last time.
17:50 Denver - Downgraded Azure container plugin to previously installed version, restarted Jenkins
18:00 Denver - Downgraded plugin reports NoSuchMethodError, likely indicator that it requires an API in one of the 50+ plugins that were upgraded at 16:37 Denver time
18:01 Denver - Updated Azure container plugin to latest release
18:17 Denver - Confirmed problem is still visible after upgrade
18:19 Denver - Record plugin file system times in case rollback needed
19:14 Denver - Identify candidate plugins for rollback
- azure-sdk from 12.vc102aedd3c66 down to 4.vcb202d9010c1
- azure-credentials from 182.v3ccd4a755864 down to 177.v816b81058012
- azure-vm-agents from 780.v50d067d02f76 down to 774.v0cee503baa25
- windows-azure-storage from 358.v5c001416d74f down to 355.v4da08e72a251
- azure-container-agents from 207.v3ad9931bf69e down to 201.v2afdce22b4cf
Review dependencies of those plugins
- Used plugin installation manager to install a local copy of those 5 plugins at their previous versions as confirmation that they should be workable at their previous versions as a group. Not a perfect check, but worth the check.
20:01 Denver - Downgrade those 5 plugins and restart ci.jenkins.io
20:03 Denver - Restart is complete, Manage Plugins shows those 5 plugins have upgrades available
20:09 DEnver - Outage resolved, ACI agents are allocated on ci.jenkins.io

Plugins that were upgraded at 16:37 Denver time

-rw-r--r-- 1 jenkins jenkins 4200464 May 29 22:37 plugins/jacoco.jpi
-rw-r--r-- 1 jenkins jenkins 380040 May 29 22:37 plugins/kubernetes.jpi
-rw-r--r-- 1 jenkins jenkins 3091463 May 29 22:37 plugins/ldap.jpi
-rw-r--r-- 1 jenkins jenkins 86675 May 29 22:37 plugins/blueocean-commons.jpi
-rw-r--r-- 1 jenkins jenkins 277630 May 29 22:37 plugins/blueocean-jwt.jpi
-rw-r--r-- 1 jenkins jenkins 39703 May 29 22:37 plugins/command-launcher.jpi
-rw-r--r-- 1 jenkins jenkins 468323 May 29 22:37 plugins/cloudbees-bitbucket-branch-source.jpi
-rw-r--r-- 1 jenkins jenkins 8089625 May 29 22:37 plugins/subversion.jpi
-rw-r--r-- 1 jenkins jenkins 102390 May 29 22:37 plugins/blueocean-rest.jpi
-rw-r--r-- 1 jenkins jenkins 14124 May 29 22:37 plugins/blueocean-jira.jpi
-rw-r--r-- 1 jenkins jenkins 86109 May 29 22:37 plugins/workflow-multibranch.jpi
-rw-r--r-- 1 jenkins jenkins 308017 May 29 22:37 plugins/branch-api.jpi
-rw-r--r-- 1 jenkins jenkins 35870 May 29 22:37 plugins/blueocean-pipeline-scm-api.jpi
-rw-r--r-- 1 jenkins jenkins 2363613 May 29 22:37 plugins/jenkins-design-language.jpi
-rw-r--r-- 1 jenkins jenkins 1076110 May 29 22:37 plugins/blueocean-core-js.jpi
-rw-r--r-- 1 jenkins jenkins 1413507 May 29 22:37 plugins/blueocean-web.jpi
-rw-r--r-- 1 jenkins jenkins 967614 May 29 22:37 plugins/blueocean-rest-impl.jpi
-rw-r--r-- 1 jenkins jenkins 192974 May 29 22:37 plugins/blueocean-pipeline-api-impl.jpi
-rw-r--r-- 1 jenkins jenkins 66662 May 29 22:37 plugins/blueocean-github-pipeline.jpi
-rw-r--r-- 1 jenkins jenkins 2745123 May 29 22:37 plugins/blueocean-dashboard.jpi
-rw-r--r-- 1 jenkins jenkins 703681 May 29 22:37 plugins/blueocean-personalization.jpi
-rw-r--r-- 1 jenkins jenkins 33137044 May 29 22:37 plugins/azure-sdk.jpi
-rw-r--r-- 1 jenkins jenkins 58121 May 29 22:37 plugins/azure-credentials.jpi
-rw-r--r-- 1 jenkins jenkins 5947031 May 29 22:37 plugins/jira.jpi
-rw-r--r-- 1 jenkins jenkins 28517 May 29 22:37 plugins/display-url-api.jpi
-rw-r--r-- 1 jenkins jenkins 2068729 May 29 22:37 plugins/pipeline-utility-steps.jpi
-rw-r--r-- 1 jenkins jenkins 776815 May 29 22:37 plugins/plugin-util-api.jpi
-rw-r--r-- 1 jenkins jenkins 486809 May 29 22:37 plugins/font-awesome-api.jpi
-rw-r--r-- 1 jenkins jenkins 261324 May 29 22:37 plugins/azure-vm-agents.jpi
-rw-r--r-- 1 jenkins jenkins 957359 May 29 22:37 plugins/credentials.jpi
-rw-r--r-- 1 jenkins jenkins 16099656 May 29 22:38 plugins/analysis-model-api.jpi
-rw-r--r-- 1 jenkins jenkins 226188 May 29 22:38 plugins/mercurial.jpi
-rw-r--r-- 1 jenkins jenkins 54575 May 29 22:38 plugins/blueocean-git-pipeline.jpi
-rw-r--r-- 1 jenkins jenkins 127789 May 29 22:38 plugins/matrix-auth.jpi
-rw-r--r-- 1 jenkins jenkins 34438 May 29 22:38 plugins/pubsub-light.jpi
-rw-r--r-- 1 jenkins jenkins 11342134 May 29 22:38 plugins/warnings-ng.jpi
-rw-r--r-- 1 jenkins jenkins 472289 May 29 22:38 plugins/windows-slaves.jpi
-rw-r--r-- 1 jenkins jenkins 1360650 May 29 22:38 plugins/cvs.jpi
-rw-r--r-- 1 jenkins jenkins 2425761 May 29 22:38 plugins/ec2.jpi
-rw-r--r-- 1 jenkins jenkins 634219 May 29 22:38 plugins/workflow-cps.jpi
-rw-r--r-- 1 jenkins jenkins 839972 May 29 22:38 plugins/timestamper.jpi
-rw-r--r-- 1 jenkins jenkins 1102491 May 29 22:38 plugins/windows-azure-storage.jpi
-rw-r--r-- 1 jenkins jenkins 61490 May 29 22:38 plugins/blueocean-config.jpi
-rw-r--r-- 1 jenkins jenkins 92076 May 29 22:38 plugins/blueocean-bitbucket-pipeline.jpi
-rw-r--r-- 1 jenkins jenkins 23581 May 29 22:38 plugins/blueocean-events.jpi
-rw-r--r-- 1 jenkins jenkins 1057838 May 29 22:38 plugins/blueocean-pipeline-editor.jpi
-rw-r--r-- 1 jenkins jenkins 15198 May 29 22:38 plugins/blueocean-i18n.jpi
-rw-r--r-- 1 jenkins jenkins 8105 May 29 22:38 plugins/blueocean.jpi
-rw-r--r-- 1 jenkins jenkins 308385 May 29 22:38 plugins/github-branch-source.jpi
-rw-r--r-- 1 jenkins jenkins 44276 May 29 22:38 plugins/structs.jpi
-rw-r--r-- 1 jenkins jenkins 96874 May 29 22:38 plugins/workflow-durable-task-step.jpi
-rw-r--r-- 1 jenkins jenkins 35277 May 29 22:38 plugins/kubernetes-credentials.jpi
-rw-r--r-- 1 jenkins jenkins 66978 May 29 22:38 plugins/ansicolor.jpi
-rw-r--r-- 1 jenkins jenkins 99985 May 29 22:38 plugins/groovy.jpi
-rw-r--r-- 1 jenkins jenkins 2337135 May 29 22:38 plugins/configuration-as-code.jpi
-rw-r--r-- 1 jenkins jenkins 189508 May 29 22:38 plugins/script-security.jpi
-rw-r--r-- 1 jenkins jenkins 822801 May 29 22:38 plugins/support-core.jpi
-rw-r--r-- 1 jenkins jenkins 98403 May 30 00:00 plugins/azure-container-agents.jpi