Jenkins Status Page

[Resolved] update center issue

Thu, 16 Apr 2026 09:00:00 +0000

[Final message]

The issue was relarted to CNI driver of publick8s cluster, resolved after recycling its nodes.

Update Center is back to normal.

[Initial message]

updates.jenkins.io is returning random HTTP/502 errors, plugin updates are stuck.

We’re working on it, expect an LDAP restart.

Problems on plugins.jenkins.io: search and changelogs

Tue, 14 Apr 2026 11:00:00 +0000

[Update] We’ve disabled the search temporarily to ensure plugins.jenkins.io stays up-to-date, fixing the changelog content problem.

[Update]

Plugins changelogs seems broken since the 13 April 2026 and only reports old versions of plugins while new versions are available.

[Initial message] Plugin search on plugins.jenkins.io is currently not returning results. Individual plugin pages are still accessible via direct URL (e.g. https://plugins.jenkins.io/git/).

The issue appears to be related to Algolia search quota exhaustion, likely caused by increased automated/LLM scraping of the search facility.

More details in https://github.com/jenkins-infra/helpdesk/issues/5066.

[Resolved] Maintenance on ci.jenkins.io (Security Advisory)

Wed, 18 Mar 2026 12:30:00 +0000

[Final Message] The ci.jenkins.io controller is running with latest plugins and with latest LTS 2.541.3 Core version.

[Initial message]

Wednesday March 18 2026, the ci.jenkins.io controller will be put offline and updated as part of a security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/Zkir_LMxL7c.

[Resolved] Maintenance on ci.jenkins.io

Tue, 10 Mar 2026 09:00:00 +0000

[Final Message] Operation completed successfully

[Initial Message] Tuesday March 10 2026, starting at 09:00 UTC, the Jenkins controllers and agents on trusted.ci.jenkins.io, cert.ci.jenkin.io, and ci.jenkins.io will be upgraded to use JDK25 at runtime instead of JDK21.

This operation will require controller restarts and agent reconnections. A short service disruption is expected during the maintenance window.

See more details at https://github.com/jenkins-infra/helpdesk/issues/4941

[Resolved] Maintenance on release.ci.jenkins.io

Mon, 02 Mar 2026 09:00:00 +0000

[Final Message] Operation completed successfully

[Initial Message] Friday 2026, starting at 09:00 UTC, the Jenkins controllers and agents on release.ci.jenkins.io will be upgraded to use JDK25 at runtime instead of JDK21.

This operation will require controller restarts and agent reconnections. A short service disruption is expected during the maintenance window.

See more details at https://github.com/jenkins-infra/helpdesk/issues/4941

[Resolved] Maintenance on weekly.ci.jenkins.io and infra.ci.jenkins.io

Fri, 27 Feb 2026 09:30:00 +0000

[Initial Message] Friday February 27 2026, starting at 09:30 UTC, the Jenkins controllers and agents on weekly.ci.jenkins.io and infra.ci.jenkins.io will be upgraded to use JDK25 at runtime instead of JDK21.

This operation will require controller restarts and agent reconnections. A short service disruption is expected during the maintenance window.

See more details at https://github.com/jenkins-infra/helpdesk/issues/4941

[Resolved] jenkins.io website outage

Mon, 02 Feb 2026 19:40:00 +0000

[Final message]

All services are back online, more details will be posted in https://github.com/jenkins-infra/helpdesk/issues/4978.

[Initial message] Jenkins website and some other services are down, currently investigating.

See details and updates at https://github.com/jenkins-infra/helpdesk/issues/4978.

[Resolved] ci.jenkins.io enforced login and users cleanup

Wed, 28 Jan 2026 18:15:00 +0000

[Initial Message] On Wednesday 28 January 2026 at 18h15 UTC, ci.jenkins.io will require authentication with a Jenkins LDAP account to allow reading pages.

As part of this change, the users database will be wiped out, effectively removing all API tokens and personal preferences. Direct consequence: incrementals.jenkins.io won’t be able to deliver new incremental releases until a new API token is generated.

[Resolved] Update Center slow due to CloudFlare outage

Tue, 20 Jan 2026 07:30:00 +0000

[Final Message] CloudFlare maintenance is finished: we’ve enabled the East America mirror as it is now responding as expected.

[Initial Message] CloudFlare is performing a maintenance (https://www.cloudflarestatus.com/incidents/y5xbgk3fh95y) with re-routed traffic. Since 07:30 UTC, our monitoring detected a lot of timeouts for the Update Center requests targeted at our East US mirror due to the re-routing latency.

We’ve disabled the East US mirror in favor of West europe or archives.jenkins.io at 08:00 UTC for now.

[Resolved] ci.jenkins.io disruption

Tue, 13 Jan 2026 05:10:00 +0000

[Final Message] The ci.jenkins.io controller is back to a normal response time. The AI crawler has stopped sending requests. We had to restart the controller once per hour to sustain the high rate of requests.

[Initial message] –> ci.jenkins.io is failing with Out Of Memory errors, we’ve restarted it. We don’t know yet the exact cause but we’re suspecting AI crawlers.

[Resolved] Migration of pkg.origin.jenkins.io service from AWS to Azure

Fri, 19 Dec 2025 09:30:00 +0000

On December 19th at 09h30am UTC, we’ll migrate the service pkg.origin.jenkins.io from AWS to Azure.

This service is the source of data for the Fastly CDN service pkg.jenkins.io:

We expect a slight slow down for users installing Jenkins Debian/RPM packages (or browsing the pages of this service).
In case of unexpected error, installation of Jenkins Debian/RPM packages (or browsing the pages of this service) will be disrupted until the operation is finished.

See more details at https://github.com/jenkins-infra/helpdesk/issues/3705.

[Resolved] [publick8s cluster] Azure publick8s Upgrade to Kubernetes 1.33.5

Mon, 15 Dec 2025 08:30:00 +0000

[Final Message] upgrade went well, no impact on production.

[Initial Message] Monday 15 December 2025 at 08:30PM UTC, we’ll upgrade the AKS(Azure) cluster publick8s to Kubernetes 1.33.5. It should take between 60 and 120 min.

The expected impacts are:

Restart of the following services, with a downtime between 2 and 15 min (depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-in/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) on:

get.jenkins.io (mirror redirector)
issue-redirect.jenkins.io
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

More details in https://github.com/jenkins-infra/helpdesk/issues/4820#issuecomment-3548729217.

[Resolved] [privatek8s cluster] Azure privatek8s Upgrade to Kubernetes 1.33.5

Fri, 12 Dec 2025 09:00:00 +0000

[Final message] Upgrade completed without any issue. We’ve done it today only as yesterday GitHub was too unstable to perform it without hitting issues when pulling terraform providers or helm repositories.

[Initial message] Friday 12 December 2025 at 09:00 UTC, we’ll upgrade the AKS(Azure) cluster privatek8s to Kubernetes 1.33.5. It should take between 30 and 60 min.

Expect multiple restarts and downtimes for the infra.ci.jenkins.io and release.ci.jenkins.io controllers along with the github-comment-ops bots.

More details in https://github.com/jenkins-infra/helpdesk/issues/4820#issuecomment-3548723638.

[Resolved] Maintenance on ci.jenkins.io (Security Advisory)

Wed, 10 Dec 2025 11:30:00 +0000

[Final Message] The ci.jenkins.io controller is running with latest plugins and with latest LTS 2.528.3 Core version.

[Initial message] Wednesday December 10 2025, the ci.jenkins.io controller will be put offline and updated as part of a security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/PN3lF_72LfU.

[Resolved] [ci.jenkins.io container agents] AWS EKS Upgrade to Kubernetes 1.33.5

Thu, 04 Dec 2025 12:00:00 +0000

[Final Message] The upgrade went well, all tests are green.

[Initial Message] Thursday 4 December 2025, at 12:00 UTC, we’ll upgrade the EKS(AWS) cluster hosting ci.jenkins.io’s container agents and Artifact Caching proxy to Kubernetes 1.33.5

All agents using jnlp-linux-* or jnlp-bom labels, or pipelines with the buildPlugin() function with useContainer: true will be queued until the end of the operation.

More details on https://github.com/jenkins-infra/helpdesk/issues/4820#issuecomment-3607329009.

[Resolved] Jenkins core issues are migrating from Jira to GitHub

Mon, 24 Nov 2025 08:30:00 +0000

All Jira issues for Jenkins core are currently being migrated to GitHub.

More details in https://github.com/jenkinsci/jenkins/issues/11301.

[Resolved] CloudFlare outage

Tue, 18 Nov 2025 11:48:00 +0000

Final message: CloudFlare incident has been resolved:

Cloudflare services are currently operating normally. We are no longer observing elevated errors or latency across the network.

Initial message: CloudFlare is down: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7

This impacts the Update Center, and element channels like #jenkins-infra

We’ll keep this status updated.

[Resolved] Jira (issues.jenkins.io) LTS upgrade maintenance window

Thu, 13 Nov 2025 16:00:00 +0000

The Jira instance at https://issues.jenkins.io, managed by the Linux Foundation team, will be down for up to 8 hours beginning at 16:00 UTC on Thursday November 13, 2025 to perform an LTS upgrade from 9.x to 10.x.

More details in https://github.com/jenkins-infra/helpdesk/issues/4644.

[Resolved] Azure outage on the '*.ci.jenkins.io' private controllers

Wed, 29 Oct 2025 15:45:00 +0000

[Final Message] The outage seems has been resolved at 00:05am on the 30 October.

[Initial Message] Azure is facing a global outage (Tracking ID: YKYN-BWZ at https://azure.status.microsoft/en-us/status/history/) which prevents access to the Azure Portal and to the Azure Identity Management API.

It means that our private controllers might be unable to spin up new VM agents: we expect delays in

The publication of our websites (both previews and production)
Jobs running in private controllers
Execution of infrastructure related tasks (Terraform and Kubernetes jobs)

Jobs will be queued and eventually aborted due to timeouts.

[Resolved] Outage on incrementals.jenkins.io

Mon, 27 Oct 2025 07:56:00 +0000

[Final Message] The error has been identified and fixed (token regeneration). Service is back to normal.

[Initial Message] We are encountering an outage on incrementals.jenkins.io, currently investigating.

More info at https://github.com/jenkins-infra/helpdesk/issues/4852.

[Resolved] Outage on incrementals.jenkins.io

Wed, 15 Oct 2025 15:41:00 +0000

[Final Message] The error has been identified and a fix (token rotation) has immediately been applied. Service is back to normal.

[Initial Message] We are encountering an outage on incrementals.jenkins.io, currently investigating.

More info at https://github.com/jenkins-infra/helpdesk/issues/4840

[Resolved] Outage on get.jenkins.io and updates.jenkins.io

Thu, 09 Oct 2025 07:08:20 +0000

[Final Message] The outage seems resolved https://github.com/jenkins-infra/helpdesk/issues/4827

[Initial Message] we are encountering an outage on get.jenkins.io and updates.jenkins.io our publick8s cluster. We are seeing problems that azure doesn’t acknowledge, probably related to an Azure outage.

see https://github.com/jenkins-infra/helpdesk/issues/4827 for information

[Resolved] Outage on get.jenkins.io and updates.jenkins.io

Tue, 07 Oct 2025 07:03:00 +0000

[Final Message] The outage seems resolved https://github.com/jenkins-infra/helpdesk/issues/4827 . We did a refresh of the 3 nodes of publick8s but were impacted with an azure outage that is not public yet.

[Initial Message] We are encountering an outage on get.jenkins.io, currently investigating. The same problem is visible on updates.jenkins.io.

[Resolved] Migration of usage.jenkins.io to Digital Ocean

Thu, 02 Oct 2025 08:00:00 +0000

[Final Message] The migration went well, all tests are green.

[Initial Message] Thursday October 02 2025, usage.jenkins.io VM will be migrated from AWS to Digital Ocean as per https://github.com/jenkins-infra/helpdesk/issues/1626.

We expect between 30mn and 2h of downtime starting at 08:00 AM UTC.

[Resolved] Azure cluster publick8s migrated to a new location and upgraded to Kubernetes 1.32

Thu, 25 Sep 2025 11:30:00 +0000

[Final Message] Outage finished on Friday 26 September 2025 at 09:35am UTC and post mortem published: https://github.com/jenkins-infra/helpdesk/issues/4813#issuecomment-3339518665.

[Update] Friday 26 September 2025 at 09:35am UTC: ongoing production outage on the cluster (unexpected side effect of the migration): https://github.com/jenkins-infra/helpdesk/issues/4813.

[Update] Migration was finished on Thursday 25 September 2025 at 03:30PM UTC. See https://github.com/jenkins-infra/helpdesk/issues/4617#issuecomment-3334826339.

[Initial Message] Thursday 25 September 2025, starting at 11:30AM UTC, we’ll perform a migration + upgrade of the AKS(Azure) cluster publick8s to a new location and to Kubernetes 1.32.

The expected impacts are:

Restart of the following services, with a downtime between 2 and 15 min (depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-in/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) on:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

More details in https://github.com/jenkins-infra/helpdesk/issues/4617.

[Resolved] ci.jenkins.io authentication required

Fri, 19 Sep 2025 12:44:00 +0000

[Final Message] The ci.jenkins.io controller is now up and running, the brownout is done, we will have to work around permission for the next brownout.

More details in https://github.com/jenkins-infra/helpdesk/issues/4768.

[Initial message] A large set of unhelpful requests are overloading ci.jenkins.io. We are in a brownout. We are testing that disallowing anonymous access resolves the performance issue.

More details in https://github.com/jenkins-infra/helpdesk/issues/4768.

[Resolved] Maintenance on ci.jenkins.io (Security Advisory)

Wed, 17 Sep 2025 09:30:00 +0000

[Final Message] The ci.jenkins.io controller is running with latest plugins and with latest LTS 2.516.3 Core version. Weekly.ci is running with latest weekly 2.528 Core version

[Initial message] Wednesday September 17 2025, the ci.jenkins.io controller will be put offline and updated as part of a security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/rOpnrQxJWEY.

[Resolved] Enforced HTTPS for Update Center

Wed, 06 Aug 2025 07:00:00 +0000

Wednesday 06 August 2025 at 07:00am UTC, the service https://updates.jenkins.io will be upgraded to enforced HTTPS as described in https://www.jenkins.io/blog/2025/08/04/update-center-enforce-https/.

The operation should take from 5 to 30 minutes (depends on the Azure Cloud ability to spin up resources on demand).

All Jenkins users are impacted but should not see any functional change. Eventually, some requests may time out during the deployment of the new settings.

More details in https://github.com/jenkins-infra/helpdesk/issues/4758.

[Resolved] Expired TLS certificate on issues.jenkins.io

Sat, 02 Aug 2025 22:00:00 +0000

[Final Message] Linux foundation shared with Jenkins Infra team the required elements (DNS ACME challenge value) to allow automatic renewal of the TLS certificate. The certificate has been finally renewed at 07h30(am) UTC.

[Initial Message] The JIRA service at issues.jenkins.io (issues.jenkins-ci.org) is unavailable due to an expired TLS certificate.

The Jenkins infrastructure is working with the Linux Foundation team (whom manage the JIRA service) to solve this issue.

More details in https://github.com/jenkins-infra/helpdesk/issues/4757.

[Resolved] infra.ci.jenkins.io down

Mon, 28 Jul 2025 08:00:00 +0000

[Final message] infra.ci is back to business after a migration to another cluster which allowed to find the root cause and fix it.

[Initial message] The infra.ci.jenkins.io (private) Jenkins controller is down with error related to cloud storage. We are currently analyzing the issue and fixing it.

All website previews in pull requests, along with deployment of plugins.jenkins.io, docs.jenkins.io and stats.jenkins.io are down.

See https://github.com/jenkins-infra/helpdesk/issues/4753 for details.

[Resolved] Maintenance on ci.jenkins.io (Security Advisory)

Wed, 09 Jul 2025 11:30:00 +0000

Wednesday July 07 2025, the ci.jenkins.io controller will be put offline and updated as part of a security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/doaM4i0Qf6o.

[Resolved] Migration of ci.jenkins.io to AWS

Mon, 07 Jul 2025 07:00:00 +0000

[Final Message] The migration went well, all tests are green.

[Initial Message] Monday July 07 2025, ci.jenkins.io controller will be migrated from Azure to AWS as per https://github.com/jenkins-infra/helpdesk/issues/4688.

We expect between 2h to 4h of downtime starting at 07:00 AM UTC.

[Resolved] jdk21 upgrade for runtime controllers and agents

Thu, 22 May 2025 13:00:00 +0000

[Final Message] migration went well, no impact on production. few fixes to apply with https://github.com/jenkins-infra/jenkins-infra/pull/4143

[Initial Message] Thursday May 22 2025, starting at ~~17:00~~ 13:00 UTC, all our controllers and there agents will be upgraded to use JDK21 as runtime version instead of JDK17.

This operation expects 1 or 2 restarts per controller https://github.com/jenkins-infra/helpdesk/issues/4121#issuecomment-2890355054.

[Resolved] Uplink service is unavailable

Tue, 20 May 2025 00:00:00 +0000

[Final Message] The service is back with an empty database. Telemtry is now collected properly.

[Initial Message] The service uplink is unavailable due to database issue. Any tentative to log-in ends up in “Internal Server Error”.

More details in https://github.com/jenkins-infra/helpdesk/issues/4667.

[Resolved] LDAP offline for reindex

Tue, 06 May 2025 17:50:00 +0000

The LDAP service requires a reindex which needs LDAP server to be offline. Starting the operation at 17h50 UTC, finished at 18h30 UTC with success.

[Resolved] [privatek8s cluster] Migrate to another Azure subscription

Tue, 29 Apr 2025 08:00:00 +0000

Tuesday 29 April 2025, at 08:00am UTC, the cluster privatek8s will be migrated between Azure subscriptions.

Expect a full downtime of infra.ci.jenkins.io and release.ci.jenkins.io from 2 hours up to 8 hours.

More details in https://github.com/jenkins-infra/helpdesk/issues/4250.

[Resolved] [publick8s cluster] Azure publick8s Upgrade to Kubernetes 1.31.6

Thu, 24 Apr 2025 07:30:00 +0000

[Final Message] upgrade went well, no impact on production.

[Initial Message] Thursday 24 April 2025, at 07:30PM UTC, we’ll upgrade the AKS(Azure) cluster publick8s to Kubernetes 1.31.6. It should take between 60 and 120 min.

The expected impacts are:

⚠️ Last Kubernetes upgrade (to 1.30) we saw an outage on update-center (updates.jenkins.io) for a few seconds !

Restart of the following services, with a downtime between 2 and 15 min (depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-in/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) on:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

More details in https://github.com/jenkins-infra/helpdesk/issues/4546#issuecomment-2674545933.

[Resolved] [privatek8s cluster] Azure privatek8s Upgrade to Kubernetes 1.31.6

Wed, 23 Apr 2025 12:30:00 +0000

Wednesday 23 April 2025, at 12:30 UTC, we’ll upgrade the AKS(Azure) cluster privatek8s to Kubernetes 1.31.6. It should take between 30 and 60 min.

Expect multiple restarts and downtimes for the infra.ci.jenkins.io and release.ci.jenkins.io controllers along with the github-comment-ops bots.

More details in https://github.com/jenkins-infra/helpdesk/issues/4546#issuecomment-2674551720.

[Resolved] Migration of ci.jenkins.io to Azure

Wed, 09 Apr 2025 08:00:00 +0000

[Final Message] The migration went well, all tests are green.

[Initial Message] Wednesday April 09 2025, at 08:00 AM UTC, the Jenkins Infra team will migrate the ci.jenkins.io controller from AWS to Azure as per https://github.com/jenkins-infra/helpdesk/issues/4620. We expect a downtime of 2 hours.

[Resolved] Maintenance on ci.jenkins.io (Security Advisory)

Wed, 02 Apr 2025 14:45:00 +0000

[Final Message] The ci.jenkins.io controller is running with latest LTS 2.479.3 Core version. The weekly.ci.jenkins.io controller is running with latest weekly 2.504 Core version.

[Initial message] Wednesday April 02 2025, the ci.jenkins.io and weekly.ci.jenkins.io controllers will be put offline and updated as a result of a security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/gjEkmc02cU0.

[Resolved] Maintenance on ci.jenkins.io (Security Advisory)

Wed, 05 Mar 2025 12:30:00 +0000

[Final Message] The ci.jenkins.io controller is running with latest LTS 2.479.2 Core version.

[Initial message] Wednesday March 05 2025, the ci.jenkins.io controller will be put offline and updated as a result of a security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/7ZO66tLSKa8.

[Resolved] [publick8s cluster] Azure publick8s Upgrade to Kubernetes 1.30.7

Thu, 20 Feb 2025 13:30:00 +0000

[Final Message] upgrade went well but we saw an outage on update center for about 2 to 3 mn we will analyse further the logs to try to understand were it’s coming from

[Initial Message] Thurday 20 February 2025, at 01:30PM UTC, we’ll upgrade the AKS(Azure) cluster publick8s to Kubernetes 1.30.7. It should take between 60 and 120 min.

The expected impacts are:

Restart of the following services, with a downtime between 2 and 15 min (depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-in/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) on:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

More details in https://github.com/jenkins-infra/helpdesk/issues/4258#issuecomment-2663553626.

[Resolved] [privatek8s cluster] Azure privatek8s Upgrade to Kubernetes 1.30.7

Thu, 20 Feb 2025 08:00:00 +0000

[Final Message] Upgrade went well. Had to wait for the infra.ci mounting volume : Warning FailedAttachVolume 5m35s attachdetach-controller Multi-Attach error for volume "jenkins-infra-pv" Volume is already exclusively attached to one node and can't be attached to another but seems faster than before.

[Initial Message] Thurday 20 February 2025, at 08:00 UTC, we’ll upgrade the AKS(Azure) cluster privatek8s to Kubernetes 1.30.7. It should take between 30 and 60 min.

Expect multiple restarts and downtimes for the infra.ci.jenkins.io and release.ci.jenkins.io controllers along with the github-comment-ops bots.

More details in https://github.com/jenkins-infra/helpdesk/issues/4258#issuecomment-2668175339.

[Resolved] [ci.jenkins.io container agents] AWS EKS Upgrade to Kubernetes 1.30.9

Wed, 19 Feb 2025 08:00:00 +0000

[Final Message] The upgrade went well, all tests are green.

[Initial Message] Wednesday 19 February 2025, at 08:00 UTC, we’ll upgrade the EKS(AWS) cluster hosting ci.jenkins.io’s container agents and Artifact Caching proxy to Kubernetes 1.30.9

All agents using jnlp-linux-* or jnlp-bom labels, or pipelines with the buildPlugin() function with useContainer: true will be queued until the end of the operation.

More details on https://github.com/jenkins-infra/helpdesk/issues/4258#issuecomment-2666362598.

[Resolved] Outage on ci.jenkins.io

Fri, 14 Feb 2025 08:30:00 +0000

Following ci migration to aws, we are encountering an outage on ci.jenkins.io, currently investigating

[Resolved] Migration of ci.jenkins.io to AWS

Thu, 13 Feb 2025 09:00:00 +0000

[Final Message] The migration went well, all tests are green.

[Initial Message]

Now, Thursday February 13 2025, we will start the ci.jenkins.io controller migration to aws as per https://github.com/jenkins-infra/helpdesk/issues/4315#issuecomment-2655944334.

We expect at least 4h of downtime starting at 09:00 AM UTC

[Resolved] ldap outage

Fri, 07 Feb 2025 01:40:00 +0000

[Final Message] The issue was identified to be an undocumented missing permission between Kubernetes and the LDAP data disk during Kubernetes Node upgrade. It has been fixed and persisted.

[Initial Message] ldap service is down so no authentification for Jira, Artifactory, or Jenkins controllers. Help desk 4531 is tracking progress on the issue.

[Resolved] Azure Outage / ci.jenkins.io Windows Container Agents

Wed, 08 Jan 2025 22:00:00 +0000

[Final Message] Outage is finished: back to normal

[Update] still in outage at this date an time 14 january 2025 1:45PM

[Update] The Azure outage finished the 04:30 UTC on 11 Jan 2025 (as per https://azure.status.microsoft/en-us/status/history/, Tracking ID: PLP3-1W8). Windows (container) builds are now flowing on ci.jenkins.io.

[Initial Message] A network Azure outage in the region US East 2 started at 22:00 UTC the 8 January 2025 impacts the Jenkins infrastructure.

Most notably, Windows container agents (using Azure Container Instances) are not starting, associated builds are stuck in the queue.

Short term: users can set useContainerAgent to false in their buildPlugin() pipeline library function to force usage of VMs instead of containers.

More details in https://github.com/jenkins-infra/helpdesk/issues/4490.

[Resolved] Outage on JFrog Artifactory (repo.jenkins-ci.org)

Sun, 22 Dec 2024 12:40:00 +0000

The JFrog Artifactory service at <repo.jenkins-ci.org> is facing an outage on JFrog’s side. It only answers HTTP/500 or HTTP/504 errors when trying to access it.

Maven builds are also affected (both locally and on ci.jenkins.io).

[Resolved] AKS cluster upgrade to 1.29.10 (patch)

Wed, 18 Dec 2024 09:45:00 +0000

We’re performing AKS cluster upgrade from 1.29.7 to 1.29.10 the 18 December 2024, starting at 09:45 UTC.

The expected impacts are:

Restart of the following services, with an outage (between 2 to 15 min: depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-In/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

[Resolved] Jira (issues.jenkins.io) upgrade and restart

Wed, 11 Dec 2024 00:00:00 +0000

The Jira instance at issues.jenkins.io, managed by the Linux Foundation team, will be down for up to 30 minutes beginning at 00:00 (UTC) Tuesday November 11, 2024 so that it can be upgraded and restarted.

[Resolved] Disruption of mirror selections on get.jenkins.io and Update Center

Thu, 05 Dec 2024 13:00:00 +0000

User reported that mirror selection is acting randomly with message in mirrorbit web UI saying “Cannot locate the client IP (ASN0)”.

Requests are still served but it may disrupt download performances.

More details in https://github.com/jenkins-infra/helpdesk/issues/4441.

[Resolved] Outage on the Update Center with 404-502 errors

Thu, 28 Nov 2024 08:45:00 +0000

[Final Message]

Multiple issues opened such as https://github.com/jenkins-infra/helpdesk/issues/4427 or https://github.com/jenkins-infra/helpdesk/issues/4428.

Remediation consisted in cordoning and draining the Kubernetes nodes to bypass the CSI SMB/CIFS errors causing the outage. Long term fix has been applied the 29 Nov.: see https://github.com/jenkins-infra/helpdesk/issues/4402 for details.

[Initial message] The update center is having errors 404 and 502, remediation incoming

More details in https://matrix.to/#/!JLUOInpEYmxJIYXlzs:matrix.org/$Uvn4cn3an3RDVQT_trYGbPT3OKdV8yf0NFfjcW_Tejo?via=g4v.dev&via=gitter.im&via=matrix.org.

[Resolved] Maintenance on ci.jenkins.io (Security Advisory)

Wed, 27 Nov 2024 12:30:00 +0000

[Final Message] The ci.jenkins.io controller is running with latest LTS 2.479.2 Core version.

[Initial message] Wednesday November 27 2024, the ci.jenkins.io controller will be put offline and updated as a result of a security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/KPuGGzCkWX0.

[Resolved] Maintenance on ci.jenkins.io (Datadog plugin upgrade)

Thu, 14 Nov 2024 08:30:00 +0000

Thursday November 14 2024, the ci.jenkins.io controller will be restarted to upgrade the datadog plugin as per https://github.com/jenkins-infra/helpdesk/issues/4377.

If this operation ends up in corrupting the build log like the previous ones, we’ll immediately restore the backup and restart the controller.

[Resolved] Maintenance on ci.jenkins.io (Security Advisory)

Wed, 13 Nov 2024 15:30:00 +0000

[Final Message] The ci.jenkins.io controller is running with updated plugins.

[Initial message] Wednesday November 13 2024, the ci.jenkins.io controller will be put offline and restarted if plugin updates are needed as a result of a plugin security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/UmiH1ZpJez4.

[Resolved] Jira (issues.jenkins.io) upgrade and restart

Mon, 11 Nov 2024 23:00:00 +0000

The JIRA instance at issues.jenkins.io, managed by the Linux Foundation team, will be down for up to 15 minutes beginning at 23:00 (UTC) Monday November 11, 2024 so that it can be upgraded and restarted.

[Resolved] Update Center Brownout (6)

Thu, 07 Nov 2024 09:00:00 +0000

Thursday 07 November 2024 at 09:00am UTC, the service https://updates.jenkins.io will switch its implementation to a new system for 24 hours.

All Jenkins users are impacted but should not see any functional change.

⚠️ Please, check that your organization respects the advertised DNS TTL or you might be stuck in the brownout longer that expected.

Under the hood, any HTTP request made to this service will be redirected to a mirror close to user locations during these brownouts.

More details in https://github.com/jenkins-infra/helpdesk/issues/2649.

[Resolved] Update Center Brownout (5)

Thu, 24 Oct 2024 10:00:00 +0000

Thursday 24 October 2024 at 10:00am UTC, the service https://updates.jenkins.io will switch its implementation to a new system for 24 hours.

All Jenkins users are impacted but should not see any functional change.

⚠️ Please, check that your organization respects the advertised DNS TTL or you might be stuck in the brownout longer that expected.

Under the hood, any HTTP request made to this service will be redirected to a mirror close to user locations during these brownouts.

More details in https://github.com/jenkins-infra/helpdesk/issues/2649.

[Resolved] [get.jenkins.io] Reset mirrors database

Wed, 16 Oct 2024 12:45:00 +0000

Thursday October 30 2024, starting at 12:45 UTC, the Jenkins download website (get.jenkins.io/mirrors.jenkins.io) will be disrupted for a technical operation on its database.

The goal is to solve the HTTP/404 errors on old files caught in the following issue: https://github.com/jenkins-infra/helpdesk/issues/4330.

The Jenkins Infra team does not expect and outage but slower download as all traffic will be sent to the fallback system until mirrors are added back and fully scanned.

More details in https://github.com/jenkins-infra/helpdesk/issues/4330.

[Resolved] Maintenance on ci.jenkins.io

Wed, 02 Oct 2024 11:19:00 +0000

Jenkins core update is being applied for the Jenkins security advisory.

[Resolved] Update Center Brownout (4)

Thu, 26 Sep 2024 07:00:00 +0000

Thursday 26 September 2024 at 07:00am UTC, the service https://updates.jenkins.io will switch its implementation to a new system for 24 hours.

All Jenkins users are impacted but should not see any functional change.

⚠️ Please, check that your organization respects the advertised DNS TTL or you might be stuck in the brownout longer that expected.

Under the hood, any HTTP request made to this service will be redirected to a mirror close to user locations during these brownouts.

More details in https://github.com/jenkins-infra/helpdesk/issues/2649.

[Resolved] Update Center Brownout (3)

Wed, 11 Sep 2024 14:00:00 +0000

Wednesday 11 September 2024 from 02:00pm UTC until Thursday 12 September 2024 02:00pm UTC, the service https://updates.jenkins.io will switch its implementation to a new system.

All Jenkins users are impacted but should not see any functional change.

⚠️ Please, check that your organization respects the advertised DNS TTL or you might be stuck in the brownout longer that expected.

Under the hood, any HTTP request made to this service will be redirected to a mirror close to user locations during these brownouts.

More details in https://github.com/jenkins-infra/helpdesk/issues/2649 and https://www.jenkins.io/blog/2024/09/10/update-center-brownouts-3/.

[Resolved] Azure VMs outage

Mon, 09 Sep 2024 15:00:00 +0000

[Final message]

Azure did fix the problem in two steps:

Additional capacity added in US East 2 the 12 September
Software fix the 13 September

[Update on 2024-09-10]

The problem was gone from 2024-09-09 around 11:00pm UTC but it’s back since 2024-09-10 around 02:00pm UTC
Our Azure Service Health Dashboard mentions 2 mixed incidents, one should be fixed the 12!

[initial message] Since Monday 9 September 2024 at 03:00pm UTC, we see an increased rate of failure to spin up Azure VM agents on ci.jenkins.io

Details in https://github.com/jenkins-infra/helpdesk/issues/4287.

[Resolved] Update Center Brownout (2)

Mon, 09 Sep 2024 14:00:00 +0000

Monday 9 September 2024 at 02:00pm UTC, the service https://updates.jenkins.io will switch its implementation to a new system during 1 hour.

All Jenkins users are impacted but should not see any functional change.

⚠️ Please, check that your organization respects the advertised DNS TTL or you might be stuck in the brownout longer that expected.

Under the hood, any HTTP request made to this service will be redirected to a mirror close to user locations during these brownouts.

More details in https://github.com/jenkins-infra/helpdesk/issues/2649.

[Resolved] Update Center Brownout (1)

Fri, 06 Sep 2024 07:00:00 +0000

Friday 6 September 2024 at 07:00am UTC, the service https://updates.jenkins.io will switch its implementation to a new system during 1 hour.

All Jenkins users are impacted but should not see any functional change.

⚠️ Please, check that your organization respects the advertised DNS TTL or you might be stuck in the brownout longer that expected.

Under the hood, any HTTP request made to this service will be redirected to a mirror close to user locations during these brownouts.

More details in https://github.com/jenkins-infra/helpdesk/issues/2649.

[Resolved] Update Azure AKS cluster `publick8s` to 1.29

Fri, 23 Aug 2024 09:00:00 +0000

Friday 23 August at 07:30am UTC, the AKS cluster publick8s will be upgraded to Kubernetes 1.29.7. It should take between 30 and 60 min.

The expected impacts are:

Restart of the following services, with a downtime between 2 and 15 min (depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-in/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) on:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

More details in https://github.com/jenkins-infra/helpdesk/issues/4161.

[Resolved] Update Azure AKS cluster `privatek8s` to 1.29

Fri, 23 Aug 2024 07:30:00 +0000

[Final message]

Upgrade finished with success.

[Initial message] Friday 23 August at 07:30am UTC, the AKS cluster privatek8s will be upgraded to Kubernetes 1.29.7. It should take between 30 and 60 min.

Expect multiple restarts and downtimes for the infra.ci.jenkins.io and release.ci.jenkins.io controllers along with the rss2twitter, ircbot and github-comment-ops bots.

More details in https://github.com/jenkins-infra/helpdesk/issues/4161.

[Resolved] [ci.jenkins.io container agents] AKS Upgrade to Kubernetes 1.29.7

Thu, 22 Aug 2024 14:00:00 +0000

Thursday 22 August 2024, at 14:00 UTC, we’ll upgrade the AKS cluster hosting ci.jenkins.io’s container agents and Artifact Caching proxy to Kubernetes 1.29.7

All agents using jnlp-linux-* or jnlp-bom labels, or pipelines with the buildPlugin() function with useContainer: true will be queued until the end of the operation.

More details on https://github.com/jenkins-infra/helpdesk/issues/4161#issuecomment-2304516597.

[Resolved] Maintenance on ci.jenkins.io

Thu, 22 Aug 2024 13:00:00 +0000

Thursday August 22 2024, the ci.jenkins.io controller will be restarted to upgrade the datadog plugin as per https://github.com/jenkins-infra/helpdesk/issues/4236.

If this operation ends up in corrupting the build log like the previous ones, we’ll immediately restore the backup and restart the controller.

[Resolved] Maintenance on ci.jenkins.io

Wed, 07 Aug 2024 09:00:00 +0000

[Final Message] The ci.jenkins.io controller is running with updated plugins and Core.

[Initial message] Wednesday August 07 2024, the ci.jenkins.io controller will be put offline and restarted if plugin updates are needed as a result of a plugin security advisory.

More details in https://groups.google.com/g/jenkinsci-advisories/c/NkXElf7MmcI.

[Resolved] ldap.jio and repo.azure.jio workloads migration to arm64 (publick8s cluster)

Fri, 05 Jul 2024 14:30:00 +0000

We’re doing a workload migration to arm64 on the publick8s AKS cluster starting at 14:30 UTC.

The expected impacts are:

Restart of the following services, with an outage (between 2 to 15 min: depends on Azure Storage systems):

ldap.jenkins.io
repo.azure.jenkins.io (ACP)

Temporary disruptions (no Log-In/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

More details on https://github.com/jenkins-infra/helpdesk/issues/3837#issuecomment-2210532769.

[Resolved] Update azure clusters publick8s Kubernetes version to 1.28

Thu, 04 Jul 2024 06:30:00 +0000

[Final Message] The operation finished successfully at 7:15am UTC.

More details on https://github.com/jenkins-infra/helpdesk/issues/4144#issuecomment-2196274746.

[Initial message] We’re doing an publick8s AZ cluster upgrade from 1.27 to 1.28 tomorrow, 4th of July 2024, starting at 6:30 UTC (8h30 paris time, 12h00 IST).

The expected impacts are:

Restart of the following service, with an outage (between 2 to 15 min: depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-In/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) due to public IP metadata changes and falco + Nginx upgrades:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

[Resolved] Jenkins Core Release 2.465 failed

Tue, 02 Jul 2024 13:00:00 +0000

[Final] GH releases published for the Docker Image: https://github.com/jenkinsci/docker/releases/tag/2.466 (and https://github.com/jenkinsci/docker/releases/tag/2.465 marked as burnt)

[Update] The issue seems to be related to a Maven plugin version bump. The release 2.465 is considered burnt and we are starting 2.466 release as a replacement.

[Initial message] The Jenkins Core weekly release 2.465 failed. We are investigating.

[Resolved] Plugins Health Score not computed

Tue, 02 Jul 2024 00:00:00 +0000

[Initial message] The Plugin Health scores are not computed since some times. We are investigating the issue.

More details on https://github.com/jenkins-infra/helpdesk/issues/4152.

[Resolved] Maintenance on weekly.ci.jenkins.io

Mon, 01 Jul 2024 12:30:00 +0000

[Final message] Migration done successfully

[Initial message] The weekly.ci.jenkins.io controller will be down for a data volume operation starting at 12:30pm.

More details in https://github.com/jenkins-infra/helpdesk/issues/4044.

[Resolved] High rate of HTTP/502 errors on ci.jenkins.io

Mon, 01 Jul 2024 09:00:00 +0000

[Final message] The issue has been resolved after a few restarts (and fixes of course!)

[Initial message] We are investigating a high rate of HTTP/502 errors on ci.jenkins.io. We’ll keep you posted.

Expect multiple restarts and outage.

More details in https://github.com/jenkins-infra/helpdesk/issues/4158.

[Resolved] Update azure clusters privatek8s Kubernetes version to 1.28

Thu, 27 Jun 2024 07:00:00 +0000

[Final message]

Upgrade finished with success.

[Initial message]

We’re doing an privatek8s AZ cluster upgrade from 1.27 to 1.28 today 27th of June 2024, starting at 07:00 UTC worst case scenario 20mn downtime

[Resolved] issues.jenkins.io (Jira) upgrade

Wed, 26 Jun 2024 23:00:00 +0000

[Initial Message]

The Jira issue tracker at issues.jenkins.io, managed by the Linux Foundation team, will be down for up to 45 minutes beginning at 23:00 (UTC) Wednesday June 26, 2024. The Jira issue tracker will be upgraded to a newer release.

[Final Message]

Upgrade was successful.

[Resolved] Maintenance on ci.jenkins.io

Wed, 26 Jun 2024 11:30:11 +0000

[Final Message] The ci.jenkins.io controller is running with updated plugins.

[Update] The update is taking longer than planned, expect a restart around 16h30 UTC.

[Initial message] Wednesday June 26 2024, the ci.jenkins.io controller will be put offline and restarted if plugin updates are needed as a result of the plugin security advisory.

The restart may happen between 11:30am UTC and 15:30 UTC.

More details in https://groups.google.com/g/jenkinsci-advisories/c/MokuNdWXGfg.

[Resolved] ci.jenkins.io operation on container agents (Kubernetes 1.28)

Tue, 25 Jun 2024 08:00:00 +0000

[Final message]

Upgrade finished.

[Initial message] We’re (infra team) doing an AKS clusters upgrade from 1.27 to 1.28 the Tuesday 25th of June 2024, starting at 08:00 UTC (10h paris time). It will impact the ci.jenkins.io container agents (both jnlp-linux-* and jnlp-bom templates) but the VM agents will still be available.

[Resolved] Repository Permission Updater on trusted.jenkins.io

Tue, 04 Jun 2024 20:00:00 +0000

As per https://github.com/jenkins-infra/helpdesk/issues/4122, trusted.jenkins.io’s RPU builds are stuck since an update, the token to publish plugins are not delivered.

[Resolved] Downgrading `datadog` plugin on ci.jenkins.io

Mon, 06 May 2024 12:00:00 +0000

As per https://github.com/jenkins-infra/helpdesk/issues/4080#issuecomment-2095831304, ci.jenkins.io will be down for an operation trying to fix the corrupted builds.

A first tentative will be done to downgrade the datadog plugin, if it fails then the plugin will be removed.

[Resolved] Maintenance on ci.jenkins.io

Thu, 02 May 2024 10:53:00 +0000

Plugin updates are being applied for the Jenkins security advisory.

[Resolved] Maintenance on ci.jenkins.io

Tue, 23 Apr 2024 16:00:00 +0000

The ci.jenkins.io controller will have builds waiting in the queue during a maintenance for configuration upgrade. Until the Azure credentials are fully rotated, no VM agent will be spawned. Container agents will continue to work as expected.

More details in https://github.com/jenkins-infra/helpdesk/issues/4052.

[Resolved] Maintenance on infra.ci.jenkins.io

Thu, 11 Apr 2024 08:00:00 +0000

[Final message] Migration face multiple issues:

needed PV/PVC creation on the disk from snapshot before cloning by the jenkins helmchart
needed some optimization to deal with timout on volume mount https://github.com/jenkinsci/helm-charts/tree/a052ad9f6fc0f95e67802edc7cef9ab9c3572bee/charts/jenkins#long-volume-attachmount-times

[Initial message]

The infra.ci.jenkins.io controller will be down to migrate the volume used to ZRS as per

https://github.com/jenkins-infra/helpdesk/issues/3823#issuecomment-2036372624

[Resolved] Artifact Caching Proxy deployment of 1.0.0

Thu, 28 Mar 2024 07:30:00 +0000

[Final message]

Upgrade successfull

See https://github.com/jenkins-infra/helpdesk/issues/3969#issuecomment-2024586947 for details.

[Initial message]

The Artifact Caching Proxy system used by ci.jenkins.io and infra.ci.jenkins.io agents to cache Maven artifacts will upgraded to version 1.0.0 the 28 March 2024 at 07:30am UTC.

Expect eventual slowness or failures in Maven dependency resolutions in the next hour on ci.jenkins.io and infra.ci.jenkins.io.

See https://github.com/jenkins-infra/helpdesk/issues/3969#issuecomment-2024586947 for details.

[Resolved] update center maintenance

Mon, 25 Mar 2024 13:30:00 +0000

[Final message]

Migration finished.

See https://github.com/jenkins-infra/helpdesk/issues/3414 for more details.

[Initial message]

We’ll migrate scripts used by the update center from blobxfer to azcopy today at 13h30 UTC, which might result in a latency of plugins new releases availability.

See https://github.com/jenkins-infra/helpdesk/issues/3414 for more details.

[Resolved] Update azure clusters publick8s Kubernetes version to 1.27

Tue, 19 Mar 2024 08:30:00 +0000

[Final Message] The operation finished successfully at 9:35am UTC.

More details on https://github.com/jenkins-infra/helpdesk/issues/3948#issuecomment-1991181964.

[Initial message] We’re doing an publick8s AZ cluster upgrade from 1.26 to 1.27 today 19th of Mars 2024, starting at 8:30 UTC (9h30 paris time).

The expected impacts are:

Restart of the following service, with an outage (between 2 to 15 min: depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-In/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) due to public IP metadata changes and falco + Nginx upgrades:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

[Resolved] Update azure clusters privatek8s Kubernetes version to 1.27

Mon, 18 Mar 2024 15:30:00 +0000

[Final message]

Upgrade finished.

[Initial message] We’re doing an privatek8s AZ cluster upgrade from 1.26 to 1.27 today 18th of Mars 2024, starting at 15:30 UTC (16h30 paris time). worst case scenario 20mn downtime

[Resolved] issues.jenkins.io (Jira) upgrade

Tue, 12 Mar 2024 23:00:00 +0000

The Jira instance at issues.jenkins.io, managed by the Linux Foundation team, will be down for up to 3 hours beginning at 23:00 (UTC) Tuesday March 12, 2024. The Jira instance will be upgraded to a newer release. Track the upgrade on the Linux Foundation status page.

[Resolved] Update AKS clusters Kubernetes version to 1.27

Thu, 07 Mar 2024 08:00:00 +0000

[Final message]

Upgrade finished.

[Initial message] We’re doing an AKS clusters upgrade from 1.26 to 1.27 the Thursday 07th of Mars 2024, starting at 08:00 UTC (9h paris time).

[Resolved] plugins.jenkins.io storage migration

Wed, 28 Feb 2024 11:00:00 +0000

[Final message]

Migration finished, no service disruption.

[Initial message] We’ll migrate the storage of plugins.jenkins.io today at 11h00 UTC, which might result in a brief service disruption.

See https://github.com/jenkins-infra/helpdesk/issues/3414 for more details.

[Resolved] jenkins.io storage migration

Mon, 26 Feb 2024 13:30:00 +0000

[Final message]

Migration finished, no service disruption.

[Initial message] We’ll migrate the storage of jenkins.io today at 13h30 UTC, which might result in a brief service disruption.

See https://github.com/jenkins-infra/helpdesk/issues/3414 for more details.

[Resolved] ci.jenkins.io is down

Wed, 07 Feb 2024 14:40:00 +0000

[Final Message]

The ci.jenkins.io controller is up again.

[Initial message]

The ci.jenkins.io controller is down, we’re working on it.

[Resolved] get.jenkins.io latest 2.443 issue

Tue, 30 Jan 2024 15:00:00 +0000

[Final Message]

Release Core packaging process finished with success, both services are back to normal.

[Initial Message]

Tuesday January 30 2024, starting at 15:05 UTC, get.jenkins.io and updates.jenkins.io return an error when trying to download the latest Jenkins Weekly release 2.443, due to an issue in the release process.

More details in https://github.com/jenkins-infra/helpdesk/issues/3927

[Resolved] get(mirrors).jenkins.io storage migration

Fri, 26 Jan 2024 12:00:00 +0000

A maintenance on get.jenkins.io (mirrors.jenkins*) will be performed with a partial service disruption the Friday 26 January 2024 starting at 12:00pm (noon time) UTC.

The goal of this maintenance is to migrate the underlying storage to a new system in Azure.

See https://github.com/jenkins-infra/helpdesk/issues/3917 for more details.

[Resolved] Migration of ci.jenkins.io VM to another location

Thu, 25 Jan 2024 12:30:00 +0000

A maintenance on ci.jenkins.io will be performed with a full service disruption the Thursday 25 January 2024 starting at 12:30pm (noon time) UTC.

The goal of this maintenance is to migrate ci.jenkins.io VM to a new location in Azure.

See https://github.com/jenkins-infra/helpdesk/issues/3913 for more details.

[Resolved] Maintenance on ci.jenkins.io

Wed, 24 Jan 2024 11:00:00 +0000

[Final Message]

The ci.jenkins.io controller has been updated and is back online.

[Initial message]

The ci.jenkins.io controller is down for maintenance as part of a security upgrade announced in

https://groups.google.com/g/jenkinsci-advisories/c/QZiecB2ArMs/m/sDvJWIdHBQAJ

[Resolved] LDAP (ldap.jenkins.io) maintenance

Fri, 19 Jan 2024 08:45:00 +0000

Friday January 19 2024, starting at 08:45am UTC, we’ll proceed to a maintenance on the LDAP service (ldap.jenkins.io) hosted in the publick8s Azure Kubernetes cluster.

Authentication on all of the *.jenkins.io and *.jenkins-ci.orf services will be unavailable during the operation. We expect a disruption between 5 and 30 min.

The goal is to update the LDAP image to a new version to use a new deployment system.

More details in https://github.com/jenkins-infra/helpdesk/issues/3837#issuecomment-1898378430.

[Resolved] get.jenkins.io maintenance

Mon, 08 Jan 2024 15:15:00 +0000

[Final Message] THe migration is finally finished at noon UTC the 09th. There were 2 downtimes:

2 min the 8th around 17:30 UTC (TLS errors) due to a missing ingress configuration
5 min the 9th around 11:50 UTC (HTTP/500 errors) due to a race condition in helm chart deployments

[Initial Message] Monday January 8 2024, starting at 15:15 UTC, we’ll proceed to a maintenance on the service get.jenkins.io hosted in the publick8s Azure Kubernetes cluster.

The following services will be slower to respond or even might be disrupted:

get.jenkins.io (Jenkins download redirection to mirrors)
mirrors.jenkins-ci.org (Jenkins download redirection to mirrors)
mirrors.jenkins.io (Jenkins download redirection to mirrors)
fallback.get.jenkins.io

The goal is to change the helm chart use to deploy the service from the legacy mirrorbits to the new mirrorbits-parent to benefit from:

Apache running in arm64 Linux systems (cheaper)
Readonly filesystem for shared data (safety)
Same features as the current updates-jenkins-io service (maintenance and sustainability)

More details in: https://github.com/jenkins-infra/helpdesk/issues/3828

[Resolved] issues.jenkins.io (Jira) upgrade

Fri, 29 Dec 2023 18:00:00 +0000

The Jira instance at issues.jenkins.io, managed by the Linux Foundation team, will be down for up to 30 minutes beginning at 18:00 (UTC) Friday December 29, 2023. The Jira instance will be upgraded to a newer release.

[Resolved] issues.jenkins.io (Jira) upgrade

Wed, 27 Dec 2023 18:00:00 +0000

The Jira instance at issues.jenkins.io, managed by the Linux Foundation team, will be down for up to two hours beginning at 18:00 (UTC) Wednesday December 27, 2023 so that it can be upgraded.

[Resolved] Jira (issues.jenkins.io) outage

Mon, 18 Dec 2023 17:00:00 +0000

The JIRA instance at issues.jenkins.io, managed by the Linux Foundation team, will be down for up to two hours beginning at 17:00 (UTC) Monday December 18, 2023 so that it can be restored from backup. A spammer damaged over 1000 Jenkins issue reports on Dec 6, 2023. A database backup will be restored that precedes the damage.

[Resolved] Removal of the Maven Central mirrored repositories from JFrog Artifactory (repo.jenkins-ci.org)

Fri, 15 Dec 2023 13:00:00 +0000

Friday December 15, 2023 at 13:00 UTC, we’ll change the configuration of the Jenkins Artifactory service at repo.jenkins-ci.org.

We will remove all mirrors of the jcenter artifact repository and the oss.sonatype.org-releases repository from repo.jenkins-ci.org to force Maven builds to download artifacts that are provided by Maven Central from the Maven central repository instead of our Artifactory instance.

This is part of the effort to decrease outbound bandwidth used by the Jenkins Artifactory service. Refer to the removal plan for more details and to the announcement in the developer mailing list.

[Resolved] Maintenance on ci.jenkins.io

Wed, 13 Dec 2023 12:54:47 +0000

Outage is being investigated. High CPU load was detected.

[Resolved] Jira (issues.jenkins.io) outage

Thu, 07 Dec 2023 22:00:00 +0000

The JIRA instance at issues.jenkins.io, managed by the Linux Foundation team, will be down for up to 30 minutes beginning at 10:00 PM (UTC) Thursday December 7, 2023 so that it can be upgraded.

[Resolved] AKS (`publick8s`) operation on the x86 nodes

Thu, 07 Dec 2023 10:30:00 +0000

[FINAL] Operation finished with success

[INITIAL] Thursday December 7 2023, starting at 10:30am UTC, we’ll proceed to the amd/intel migration from x86medium nodepool to x86small nodepool

Related help desk issue: https://github.com/jenkins-infra/helpdesk/issues/3827

[Resolved] Brownout of the Maven Central mirrored repositories in JFrog Artifactory (repo.jenkins-ci.org)

Wed, 06 Dec 2023 13:00:00 +0000

Wednesday December 6, 2023, we’ll perform a brownout operation of 2 hours on the Jenkins Artifactory service at repo.jenkins-ci.org at 13:00 UTC.

The brownout will remove all mirrors of the jcenter artifact repository from repo.jenkins-ci.org to force Maven builds to download artifacts that are provided by Maven Central from the Maven central repository instead of our Artifactory instance.

This planned brownout is part of the effort to decrease outbound bandwidth used by the Jenkins Artifactory service. Refer to the removal plan for more details and to the announcement in the developer mailing list.

[Resolved] operation on systempool publick8s

Wed, 06 Dec 2023 09:15:00 +0000

Wednesday December 6 2023, starting at 9:15 UTC (10:15 Paris), we’ll proceed to the update of terraform with the new systempool with taint.

Related help desk issue: https://github.com/jenkins-infra/helpdesk/issues/3827

[Resolved] migration from nodepool x86medium to x86small

Tue, 05 Dec 2023 13:00:00 +0000

[FINAL] Aborted as not enough team members available to secure the operation

[INITIAL] Tuesday December 5 2023, starting at 13:00 UTC (14:00 Paris), we’ll proceed to the amd/intel migration from x86medium nodepool to x86small nodepool with a migration of the old systempool to the new one with taint.

Related help desk issue: https://github.com/jenkins-infra/helpdesk/issues/3827

[Resolved] migration from nodepool x86medium to x86small

Mon, 04 Dec 2023 08:00:00 +0000

[CLOSING] operation postpone, new systempool have been create, migration will occur later.

[INITIAL] Tuesday December 5 2023, starting at 8:00 UTC (09:00 Paris), we’ll proceed to the amd/intel migration from x86medium nodepool to x86small nodepool

Related help desk issue: https://github.com/jenkins-infra/helpdesk/issues/3827

[Resolved] arm64 migration of weekly.ci.jenkins.io

Mon, 27 Nov 2023 15:00:00 +0000

[FINAL message] Operation successfull

[INITIAL message] Monday November 27 2023, starting at 15:00 UTC (16:00 Paris), we’ll proceed to the arm64 migration of the weekly.ci.jenkins.io service with volume migration

[Resolved] Operation on Jenkins Azure VM Controllers

Wed, 22 Nov 2023 14:30:00 +0000

Wednesday November 22 2023, starting at 14:30 UTC, an operation on the Jenkins Azure VM controllers will happen.

No expected impact except delays in the Azure VM and ACI container agent allocation.

[Resolved] arm64 migration of weekly.ci.jenkins.io

Tue, 21 Nov 2023 16:00:00 +0000

[Final Message] Operation aborted, we’ve identified the issue but it needs more work.

[Initial Message] Tuesday November 21 2023, starting at 16:00 UTC (17h00 Paris Time), we’ll proceed to the second attempt of arm64 migration of the weekly.ci.jenkins.io service.

[Resolved] arm64 migration of weekly.ci.jenkins.io

Tue, 21 Nov 2023 08:30:00 +0000

[Final Message] Operation aborted due to a ‘volume node affinity’ problem, need to find out the problem before next try

[Initial Message] Tuesday November 21 2023, starting at 8:30 UTC, we’ll proceed to the arm64 migration of the weekly.ci.jenkins.io service.

[Resolved] arm64 migration of mirrorbits

Mon, 20 Nov 2023 16:30:00 +0000

done without interruption

Monday November 20 2023, starting at 16:30 UTC, we’ll proceed to the arm64 migration of the mirrorbits service not yet in used in production, this should be seemless.

[Resolved] arm64 migration of plugin-site and plugin-site-issues

Fri, 17 Nov 2023 14:00:00 +0000

[FINAL message] all the pods for plugin-site and plugin-site-issues are now on an arm64 nodepool

[INITIAL message] Friday November 17 2023, starting at 14:15 UTC, we’ll proceed to https://plugin-site-issues.jenkins.io plugins.origin.jenkins.io and migration to arm64.

There might be a brief interruption of the services.

[Resolved] Update publick8s AKS cluster to Kubernetes 1.26

Thu, 09 Nov 2023 08:30:00 +0000

[Final Message] The operation finished successfully at 11:00am UTC. The monitoring showed an outage of 6 minutes on the LDAP and 2 minutes on the incremental publisher.

More details on https://github.com/jenkins-infra/helpdesk/issues/3683#issuecomment-1803566680.

[Initial Message] The AKS cluster publick8s will be upgraded from Kubernetes 1.25 to 1.26 the Thursday 09th of November 2023, starting at 08:30am UTC.

The expected impacts are:

Restart of the following service, with an outage (between 2 to 15 min: depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-In/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) due to public IP metadata changes and falco + Nginx upgrades:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

[Resolved] Update privatek8s AKS cluster to Kubernetes 1.26

Tue, 07 Nov 2023 08:00:00 +0000

[Final Message] The operation finished with success at 09:15am UTC

[Initial Message] We’re doing an AKS cluster upgrade from 1.25 to 1.26 the Tuesday 07th of November 2023, starting at 08:00am UTC.

The expected impacts are:

infra.ci.jenkins.io will be down during 20 to 30 min (time for the upgrade to be performed and persistent volumes to be migrated)
No website preview of *.jenkins.io on pull requests
release.ci.jenkins.io will be down during 20 to 30 min (time for the upgrade to be performed and persistent volumes to be migrated)

[Resolved] arm64 migration of datadog-cluster-agent

Mon, 06 Nov 2023 15:30:00 +0000

[Final message]

The operation finished with success, no interruption of services.

[Initial message]

Monday November 6 2023, starting at 15:30 UTC, we’ll proceed to the datadog cluster agent migration to arm64. (Hosted on publick8s cluster)

There might be a brief interruption of the services.

Related help desk issue: https://github.com/jenkins-infra/helpdesk/issues/3619

[Resolved] Update of the AWS EKS Kubernetes clusters to 1.26

Mon, 06 Nov 2023 15:00:00 +0000

[Final Message] The operation finished successfully at 16:45 UTC.

[Initial Message] An upgrade of the AWS EKS Kubernetes clusters from 1.25 to 1.26 is scheduled the Monday 06 November 2023 at 15:00 UTC.

Expected impact on ci.jenkins.io:

Usual plugin build workloads using Linux containers will be run in DigitalOcean clusters
No BOM builds can be run until the operation is finished

More details in https://github.com/jenkins-infra/helpdesk/issues/3683#issuecomment-1794913444.

[Resolved] Ensure old cached jenkins.io pages are removed

Fri, 03 Nov 2023 13:30:00 +0000

[Final message]

The operation finished with success, no interruption of services.

[Update]

Friday November 3d 2023, starting at 13:30 UTC, we’ll activate again an option of jenkins.io build process to ensure old cached jenkins.io pages are removed and not indexed anymore.

There should not be any interruption of service.

[Update]

As removing the outdated content resulted in many old links returning 403 errors, the change has been reverted and the previous state restored.

Postponing the outdated content deletion while we’re working on the next steps described in https://github.com/jenkins-infra/helpdesk/issues/3360#issuecomment-1790899357.

[Initial message]

Thursday November 2d 2023, starting at 13:00 UTC, we’ll activate an option of jenkins.io build process to ensure old cached jenkins.io pages are removed and not indexed anymore.

There might be a brief interruption of service.

Related help desk issue: https://github.com/jenkins-infra/helpdesk/issues/3360

[Resolved] arm64 migration of cert-manager

Tue, 31 Oct 2023 13:00:00 +0000

[Final message]

The operation finished with success, no interruption of services.

[Initial message]

Monday November 6 2023, starting at 15:30 UTC, we’ll proceed to the cert manager migration to arm64. (Hosted on publick8s cluster)

There might be a brief interruption of the services.

Related help desk issue: https://github.com/jenkins-infra/helpdesk/issues/3619

[Resolved] arm64 migration of uplink

Tue, 31 Oct 2023 13:00:00 +0000

[Final message]

The operation finished with success, no interruption of services.

[Initial message]

Tuesday October 31 2023, starting at 13:00 UTC, we’ll proceed to https://uplink.jenkins.io migration to arm64.

There might be a brief interruption of the services.

Related help desk issue: https://github.com/jenkins-infra/helpdesk/issues/3619

[Resolved] Preparation of uplink arm64 migration

Tue, 31 Oct 2023 10:00:00 +0000

[Final message]

The operation is finished, no interruption of services.

[Initial message]

Tuesday October 31 2023, starting at 10:00 UTC, we’ll proceed to the deployment of a new uplink image in order to prepare arm64 migration of https://uplink.jenkins.io.

There might be a brief interruption of the services.

Related help desk issue: https://github.com/jenkins-infra/helpdesk/issues/3619

[Resolved] Update of the Digital Ocean Kubernetes clusters to 1.26

Thu, 26 Oct 2023 12:00:00 +0000

[Final Message] The operation finished successfully at 12:45 UTC.

[Initial Message] An upgrade of the DigitalOcean Kubernetes clusters from 1.25 to 1.26 is scheduled the Thursday 26 October 2023 at 12:00 (noon) UTC.

No visible impact is expected on ci.jenkins.io as the container agent workloads will be taken over by the AWS EKS cluster instead of Digital Ocean starting at 11:00 UTC.

More details in https://github.com/jenkins-infra/helpdesk/issues/3683#issuecomment-1780857936.

[Resolved] arm64 migration of mirrorbits

Wed, 25 Oct 2023 16:00:00 +0000

[Final message]

The operation is postponed as we encountered issues with some binaries not built for arm64 architecture in our images, no interruption of services.

[Initial message]

Wednesday October 25 2023, starting at 16:00 UTC, we’ll proceed to the arm64 migration of the mirrorbits service responsible for get.jenkins.io, fallback.get.jenkins.io, mirrors.jenkins.io and mirror.jenkins-ci.org.

There might be a brief interruption of these services.

Related help desk issue: https://github.com/jenkins-infra/helpdesk/issues/3619

[Resolved] arm64 migration of public nginx ingress on publick8s cluster

Tue, 10 Oct 2023 12:00:00 +0000

[Final message]

The operation is now finished, no interruption of services.

[Initial message]

Thursday October 19 2023, starting at 12:00 UTC, we’ll proceed to the arm64 migration of the publick8s nginx ingress, hosting most of Jenkins Infrastructure public-facing services in order to reduce costs.

There might be a brief interruption of these services.

Related help desk issue: https://github.com/jenkins-infra/helpdesk/issues/3619

[Resolved] Outage on vpn.jenkins.io

Mon, 02 Oct 2023 09:00:00 +0000

After a hard reboot in azure, we had the VM back.

An outage is currently happening on vpn.jenkins.io we are investigating.

[Resolved] Maintenance on ci.jenkins.io

Wed, 20 Sep 2023 11:30:00 +0000

A maintenance on ci.jenkins.io will be performed with a full service disruption the Wednesday 20 of September 2023 between 11h30 and 14h30 UTC.

[Resolved] weekly.ci.jenkins.io outage

Wed, 13 Sep 2023 10:00:00 +0000

[Final message] after a complete recycle of the publick8s nodes the volumes were able to be correctly unmout/mount on the new nodes.

[Initial message] weekly.ci service is down https://weekly.ci.jenkins.io/

[Resolved] Brownout of the Maven Central mirrored repositories in JFrog Artifactory (repo.jenkins-ci.org)

Mon, 04 Sep 2023 12:00:00 +0000

[Final Message] Brownout is finished.

[Initial Message]

Monday 04 of September 2023, we’ll perform a brownout operation of 4 hours on the Jenkins Artifactory service at repo.jenkins-ci.org at 10:00 UTC.

This operation consists in removing all mirrors of the Maven Central artifact repository from repo.jenkins-ci.org to force Maven builds to download mirrored artefacts from the Maven central instead of our Artifactory instance.

This planned brownout is the 2nd tentative related to https://github.com/jenkins-infra/helpdesk/issues/3599 which goal decrease the amount of outbound bandwidth download of our instance.

[Resolved] Maintenance on ci.jenkins.io

Sat, 02 Sep 2023 14:45:00 +0000

A plugin update to ci.jenkins.io did not complete successfully. The server is currently reporting that it is restarting.

[Resolved] Maintenance on ci.jenkins.io, trusted.ci.jenkins.io and cert.ci.jenkins.io

Mon, 28 Aug 2023 12:00:00 +0000

[Final message]

The operation is now finished. It took 3h30 instead of the initial 2h due to unexpected disk slowness but was completed successfully.

[Initial message]

Monday August 28 2023, starting at 12:00 UTC, the Jenkins controller cert.ci.jenkins.io will be put down for maintenance (migration to a new VM) as per https://github.com/jenkins-infra/helpdesk/issues/3688. Only the Jenkins Security team is impacted.

[Resolved] Maintenance on ci.jenkins.io, trusted.ci.jenkins.io and cert.ci.jenkins.io

Mon, 21 Aug 2023 13:45:00 +0000

[Final Message] Operation completed successfully:

Took around 30 minutes as expected.
A few minor unexpected behaviors are tracked in https://github.com/jenkins-infra/helpdesk/issues/3072 (fixed the 22 of October 2023).

[Initial Message] Monday August 21 2023, starting at 13:45 UTC, the Jenkins controllers ci.jenkins.io, trusted.ci.jenkins.io and cert.ci.jenkins.io will be upgraded to use JDK17 instead of JDK11.

This operation expects 1 or 2 restarts per controller but then should be harmless as we worked on the agent images as per https://github.com/jenkins-infra/helpdesk/issues/3072 and already upgraded 3 other controllers.

[Resolved] Maintenance on ci.jenkins.io and trusted.ci.jenkins.io

Thu, 17 Aug 2023 17:00:00 +0000

Thursday August 17 2023, starting at 16:30 UTC, an infrastructure operation on the https://ci.jenkins.io and https://trusted.ci.jenkins.io controller will be done. This operation impacts the network of both services which could lead to network errors when accessing the services and when running build.

The operation took 1 hour and finished with success.

More details in https://github.com/jenkins-infra/azure/pull/454

[Resolved] Maintenance on ci.jenkins.io

Wed, 16 Aug 2023 11:30:00 +0000

[Final Message] The ci.jenkins.io controller is running with updated plugins and core version.

[Initial message] Wednesday August 16 2023, between 11:30 UTC and 13:30 UTC, the https://ci.jenkins.io controller will be unavailable as it will be upgraded and restarted as part of the security advisory announced in https://groups.google.com/g/jenkinsci-advisories/c/xOqgDOYxTAw.

[Resolved] Brownout of the Maven Central mirrored repositories in JFrog Artifactory (repo.jenkins-ci.org)

Wed, 02 Aug 2023 14:30:00 +0000

[Final Message] Brownout is finished after 1h30 (instead of initial 1 hour).

[Initial Message] Wednesday 02 of August, we’ll perform a brownout operation of 1 hour on the Jenkins Artifactory service at repo.jenkins-ci.org at 14:30 UTC for ~1 hour.

This planned brownout is a step of https://github.com/jenkins-infra/helpdesk/issues/3599 to decrease the amount of outbound bandwidth download of our instance.

There should be no functional impact, as Maven as a built-in fallback to the repository central. However, we expect the builds on ci.jenkins.io to be a bit slower as the artifacts downloaded from central won’t be cached anymore.

[Resolved] reports.jenkins.io migration

Tue, 01 Aug 2023 13:00:00 +0000

[Final Message] The migration went well with no disruption. https://github.com/jenkins-infra/kubernetes-management/pull/4209

[Initial message] A migration of reports.jenkins.io from an Intel x86_64 node pool to an arm64 on the cluster publick8s is planned today, Tuesday 1st of August 2023 at 13h00 UTC.

More details on https://github.com/jenkins-infra/helpdesk/issues/3619

[Resolved] javadoc.jenkins.io migration

Mon, 31 Jul 2023 13:00:00 +0000

[Final Message] The migration went well with no disruption.

[Initial message] A migration of the javadoc website from an arm nodepool to another on the cluster publick8s is planned on Monday 31st of July 2023 at 13h00 UTC.

More details on https://github.com/jenkins-infra/helpdesk/issues/3623#issuecomment-1613504371

[Resolved] Maintenance on ci.jenkins.io

Wed, 26 Jul 2023 12:45:00 +0000

[Final Message] The ci.jenkins.io controller is running with updated plugins and core version.

[Initial message]

Wednesday July 26 2023, between 10:30 UTC and 14:30 UTC, the ci.jenkins.io controller will be unavailable as it will be upgraded and restarted as part of the security advisory announced in https://groups.google.com/g/jenkinsci-advisories/c/2GiSsACaT28.

[Resolved] Maintenance on ci.jenkins.io

Wed, 12 Jul 2023 11:38:11 +0000

[Final Message] The ci.jenkins.io controller is running with updated plugins.

[Initial message] Wednesday July 12, 2023 the ci.jenkins.io controller will be restarted if plugin updates are needed as a result of the plugin security advisory.

The restart may happen between 11:30am UTC and 13:30 UTC.

More details in https://groups.google.com/g/jenkinsci-advisories/c/y50ITBfDesw.

[Resolved] Update AKS clusters Kubernetes version to 1.25

Fri, 07 Jul 2023 08:30:00 +0000

[Final Message] The operation finished at 15:30 UTC after a full downtime of ~3 hours on the publick8s cluster due to a network issue.

More details in https://github.com/jenkins-infra/helpdesk/issues/3582.

[Initial Message] We’re doing an AKS clusters upgrade from 1.24 to 1.25 the Friday 07th of July 2023, starting at 08:30 UTC.

No outage is expected, but there will be application restarts, including a downtime between 3 to 10 minutes for ldap.jenkins.io.

Other services potentially impacted:

accounts.jenkins.io
get.jenkins.io
incrementals.jenkins.io
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
ldap.jenkins.io
plugins.jenkins.io
previews of *.jenkins.io pull requests (infra.ci.jenkins.io)
release.ci.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

[Resolved] Jira (issues.jenkins.io) outage

Thu, 06 Jul 2023 21:00:00 +0000

[Final message] This work is completed.

More details are available from the Linux Foundation in Jira upgrade for Jenkins.

[Initial message] The JIRA instance at issues.jenkins.io, managed by the Linux Foundation team, will be down for up to three hours beginning at 9:00 PM (UTC) Thursday July 6, 2023 so that it can be upgraded.

[Resolved] Maintenance on ci.jenkins.io

Tue, 04 Jul 2023 13:00:00 +0000

[Final Message] Everything is back to normal on ci.jenkins.io. Follows up in https://github.com/jenkins-infra/helpdesk/issues/3535.

[Update 3] At 21h30 UTC, all builds (except BOM) were able to run, with some runtime errors in some cases due to missing plugins.

[Update 3] At 20h45 UTC, the migration was finished and ci.jenkins.io was available again.

[Update 2] ci.jenkins.io remains down until 5th of July due to I/O issues on the old JENKINS_HOME. Ref. https://github.com/jenkins-infra/helpdesk/issues/3535#issuecomment-1620171502.

[Update 1] Kubernetes 1.25 upgrade on AWS went well and was finished at 15:00 UTC.

[Initial message] Maintenance on ci.jenkins.io is planned the Tuesday 4th of July 2023:

Initial operation starts at 13:00 UTC with an AWS Kubernetes upgrade to 1.25.
- Expected impact is no BOM build will be possible during the maintenance while other builds executed in DigitalOcean and/or Azure.
- More details in https://github.com/jenkins-infra/helpdesk/issues/3582.
Second part of the operation starts at 14h30 UTC for a migration of the virtual machine to a new generation of hardware (Azure VM Gen2) and operating system (Ubuntu 22.04).
- Expected impact is a full downtime during the migration.
- More details in https://github.com/jenkins-infra/helpdesk/issues/3535.

[Resolved] DigitalOcean outage

Wed, 21 Jun 2023 11:45:00 +0000

[Final message] Incident is reported officially by Digital Ocean and is finished: https://status.digitalocean.com/incidents/bclmy32d12p0.

We’ve restored all the nominal configuration after health-checking the clusters.

[Initial message] DigitalOcean infrastructure is having a network outage. We see a high rate of TCP errors on both Kubernetes clusters doks (agents for ci.jenkins.io) and doks-public (Artifact Caching Proxy for these agents). We are disabling temporarily agent scheduling in DigitalOcean: all ci.jenkins.io containers workload will run in AWS or Azure instead.

More details in https://github.com/jenkins-infra/helpdesk/issues/3633.

[Resolved] Maintenance on ci.jenkins.io

Wed, 14 Jun 2023 12:00:00 +0000

[Final message] The maintenance is ended. Advisory published: https://www.jenkins.io/security/advisory/2023-06-14/.

[Initial message] Maintenance on ci.jenkins.io, down for security advisory. (ref. https://groups.google.com/g/jenkinsci-advisories/c/CD7La5U-dIQ).

[Resolved] jenkins.io migration

Tue, 13 Jun 2023 09:25:00 +0000

[Final message] Migration successfully completed, no service interruption.

[Initial message] A migration of Jenkins website from prodpublick8s cluster to publick8s cluster is planned on Tuesday 13th of June 2023 at 12h00 UTC.

In order to avoid any concurrent updates, jenkins-infra/jenkins.io builds will be disabled for less than an hour.

More details on https://github.com/jenkins-infra/helpdesk/issues/3351#issuecomment-1588040182.

[Resolved] plugins.jenkins.io migration

Tue, 13 Jun 2023 08:00:00 +0000

[Final message] Migration successfully completed, no service interruption.

[Initial message] A migration of the plugins site website from prodpublick8s cluster to publick8s cluster is planned on Tuesday 13th of June 2023 at 08h00 UTC.

In order to avoid any concurrent updates, jenkins-infra/plugin-site infra builds will be disabled for less than an hour.

More details on https://github.com/jenkins-infra/helpdesk/issues/3351#issuecomment-1588038925.

[Resolved] plugin-site-issues migration

Mon, 12 Jun 2023 13:30:00 +0000

[Final message] Migration completed.

[Initial message] A migration of the “plugin-site-issue” service (plugin-site-issues.jenkins.io - used by plugins.jenkins.io to retrieve release and issue information about a plugin) from prodpublick8s cluster to publick8s cluster is planned today on Monday 12st of June 2023 at 13h00 UTC. While there should not be any interruption, the corresponding service may be down for less than one hour, and plugins.jenkins.io may be impacted.

More details on https://github.com/jenkins-infra/helpdesk/issues/3351#issuecomment-1586972368.

[Resolved] mirrorbits migration

Mon, 12 Jun 2023 12:00:00 +0000

[Final message] Migration completed.

[Initial message] A migration of the “mirrorbits” service from prodpublick8s cluster to publick8s cluster is planned today on Monday 12st of June 2023 at 12h00 UTC. While there should not be any interruption, the corresponding service may be down for less than one hour.

More details on https://github.com/jenkins-infra/helpdesk/issues/3351#issuecomment-1586969797.

[Resolved] Outage on VM agents on ci.jenkins.io

Fri, 09 Jun 2023 12:37:00 +0000

[Final message] revert disabled plugin and restart ci.jenkins.io

[Initial message] There is an outage on ci.jenkins.io since 12h35 UTC. following plugin removal there is a configuration as code error message.

https://github.com/jenkins-infra/helpdesk/issues/3573#issuecomment-1584503523

[Resolved] Brownout of the 'jgit' mirrored repository in JFrog Artifactory (repo.jenkins-ci.org)

Thu, 08 Jun 2023 12:30:00 +0000

[Final message] Our experiment is now finished. Projects using the Eclipse jgit had errors downloading artefacts during the operation but it is back to normal

[Initial message] The mirror repository jgit in the JFrog Artifactory service <repo.jenkins-ci.org> will be restricted to only authenticated users the 8 June 2023 at 12:30UTC for 30 min.

This planned brownout is a step of https://github.com/jenkins-infra/helpdesk/issues/3599 to study the behavior of our builds.

There should be no impact, as the artifacts should be consumed through the public virtual repository but we cannot be 100% sure, hence this operation.

[Resolved] ldap.jenkins.io migration

Thu, 08 Jun 2023 12:00:00 +0000

[Final message] Although the intended redirections from accounts.jenkins.io & accounts.jenkins-ci.org to status.jenkins.io didn’t work as expected, the LDAP migration has been successfully completed, no service interruption.

[Initial message] A migration of the LDAP service (ldap.jenkins.io - used for Jenkins user authentication) from prodpublick8s cluster to publick8s cluster is planned today on Thursday 8th of June 2023 at 12h00 UTC.

In order to avoid any user base modification while migrating this service, accounts.jenkins.io access will be disabled for less than an hour.

Although no downtime is expected, the following services might be impacted:

JFrog Artifactory (repo.jenkins-ci.org)
ci.jenkins.io
infra.ci.jenkins.io
release.ci.jenkins.io
trusted.ci.jenkins.io
cert.ci.jenkins.io
vpn.jenkins.io
issues.jenkins.io

More details on https://github.com/jenkins-infra/helpdesk/issues/3351#issuecomment-1576741071

[Resolved] ldap outage

Tue, 06 Jun 2023 12:45:00 +0000

[Final message] The service has been restored, a misconfiguration was responsible for its outage.

[Initial message] ldap service is down so no authentification working on jira or controllers

[Resolved] ci.jenkins.io scans outage

Mon, 05 Jun 2023 14:00:00 +0000

A change in the GitHub REST API is responsible for a bug in the integration used by Jenkins to retrieve repository maintainers. Repository scans on ci.jenkins.io fail to retrieve pull requests.

The issue will require a new release of the GitHub API library and the GitHub API plugin that packages the library as a Jenkins plugin. A patched GitHub API has been installed on ci.jenkins.io while waiting for the release of the library and the plugin.

See the help desk issue to track progress.

[Resolved] accounts.jenkins.io migration

Fri, 02 Jun 2023 11:25:00 +0000

[Final message] Migration completed.

[Initial message] A migration of the “accountapp” service (accounts.jenkins.io) from prodpublick8s cluster to publick8s cluster is planned today on Friday 2st of June 2023 at 13h00 UTC. The corresponding service may be down for less than one hour.

More details on https://github.com/jenkins-infra/helpdesk/issues/3351#issuecomment-1573486575.

[Resolved] reports.jenkins.io migration

Fri, 02 Jun 2023 08:40:00 +0000

[Final message] Migration completed.

[Initial message] A migration of the “Reports” application (reports.jenkins.io) from prodpublick8s cluster to publick8s cluster is planned today on Friday 2st of June 2023 at 08h45 UTC. The corresponding service may be down for less than one hour.

More details on https://github.com/jenkins-infra/helpdesk/issues/3351#issuecomment-1573354662.

[Resolved] trusted.ci.jenkins.io migration to Azure

Fri, 02 Jun 2023 08:30:00 +0000

[Final message] The migration was finished at 17h00 UTC and everything looks good: Update Center, RPU, jenkins.io and all jobs are successful.

[Initial message] A migration of the “Trusted” controller (trusted.ci.jenkins.io) virtual machines from AWS EC2 to Azure is planned today on Friday 2 of June 2023 starting at 08h30 UTC. The service will be down until the migration is complete and finished.

It has an impact on:

The Update Center (updates.jenkins.io): any release of a plugin won’t be visible until the end of the migration
The CD plugin process: the Repository Permission Updater system won’t be updated until the end of the migration: Publishing a new plugin with the CD process or logging-in to Artifactory (repo.jenkins-ci.org) could be impacted
The Plugin site (plugins.jenkins.io) and the Javadoc (javadoc.jenkins.io) won’t be updated during the migration

More details on https://github.com/jenkins-infra/helpdesk/issues/3486.

[Resolved] uplink.jenkins.io migration

Thu, 01 Jun 2023 13:00:00 +0000

[Final message] Migration completed.

[Initial message] A migration of the “Uplink” application (uplink.jenkins.io - used for telemetry) from prodpublick8s cluster to publick8s cluster is planned today on Thursday 1st of June 2023 at 12h45 UTC. The corresponding service may be down for less than one hour.

More details on <https://github.com/jenkins-infra/helpdesk/issues/3351#issuecomment-1568499546>.

[Resolved] rating.jenkins.io migration

Tue, 30 May 2023 16:45:00 +0000

[Final message] Migration complete with a 3 minutes downtime.

[Initial message] A migration of the application “Rating” (rating.jenkins.io - used by https://www.jenkins.io/changelog-stable/ and https://www.jenkins.io/changelog/ pages) from prodpublick8s cluster to publick8s cluster is planned today on Tuesday 30th of May 2023 at 16h45 UTC. The corresponding service may be down for less than one hour.

More details on https://github.com/jenkins-infra/helpdesk/issues/3351#issuecomment-1568499546.

[Resolved] Incremental Publisher migration

Fri, 26 May 2023 16:30:00 +0000

[Final message]

Migration completed with no disruption

[Initial message] A migration of the Incremental Publisher (incrementals-publisher.jenkins.io) application from prodpublick8s cluster to publick8s cluster is planned today on Friday 26th of May 2023 at 16h15 UTC.

No downtime is expected as the service is stateless but slowness might happen.

More information in https://github.com/jenkins-infra/helpdesk/issues/3351#issuecomment-1564632990.

[Resolved] Plugins Health Score migration

Fri, 26 May 2023 13:30:00 +0000

[Final message] Migration done without downtime on plugins.jenkins.io and only 10mn plugin-health.jenkins.io.

[Initial message] A migration of the application Plugins Health Score from prodpublick8s cluster to publick8s cluster is planned today on Friday 26th of May 2023 at 13h30 UTC, the corresponding services may be down for less than one hour:

plugins.jenkins.io
plugin-health.jenkins.io

More details on https://github.com/jenkins-infra/helpdesk/issues/3351#issuecomment-1564517009.

[Resolved] Maintenance on ci.jenkins.io

Tue, 16 May 2023 11:00:00 +0000

[Final message] The maintenance is ended.

[Initial message] Maintenance on ci.jenkins.io, down for security advisory. (ref. https://groups.google.com/g/jenkinsci-advisories/c/_mJEHe9lXTU).

[Resolved] Outage for container agents on ci.jenkins.io

Tue, 25 Apr 2023 13:00:00 +0000

[Final Message] A new cluster was created and used to replace the former one, with additional measures to avoid future lock out. Follow up in https://github.com/jenkins-infra/helpdesk/issues/3542.

[Updated]

Root cause found: the problem was caused by a mix of technical and human errors which led to a lockout of the AWS EKS cluster used by ci.jenkins.io.
The cluster requires a re-creation, which can take 1 to 2 hours.
A remediation was applied: DigitalOcean is now used for container workload until the operation is finished: https://github.com/jenkins-infra/jenkins-infra/commit/2bfbb53123b43317d91a280ce8b8c9ae4c8b35a2

[Initial message] ci.jenkins.io is not able to start container agents.

The builds requiring a container agents are queued, while the other are executed as usual.

[Resolved] Incident on get.jenkins.io (Mirror Redirector)

Wed, 19 Apr 2023 07:32:00 +0000

[Update] Triggering a node pool image OS upgrade to recycle volume mounts. Expect a 4-5 minutes outage on the LDAP connection (e.g. authentication on issues.jenkins.io and (.*)ci.jenkins.io)

[Initial message] The mirror redirector service (get.jenkins.io) is slow and may responds HTTP/502 errors on some requests as reported by a user at 7:32am UTC - https://github.com/jenkins-infra/helpdesk/issues/3525.

Investigation in progress, this page will be updated accordingly to our findings.

[Resolved] Maintenance on ci.jenkins.io

Wed, 12 Apr 2023 11:00:00 +0000

[Final message] The maintenance is ended.

[Initial message] Maintenance on ci.jenkins.io, down for security advisory. (ref. https://groups.google.com/g/jenkinsci-advisories/c/ccmIF1Vs-9A).

[Resolved] Outage on VM agents on ci.jenkins.io

Tue, 28 Mar 2023 14:00:00 +0000

[Final message] Outage finished, cleanup operations were done the 30 March with no impact.

[Update - 2023-03-29 15h50 UTC] Build queue cleaned-up two times, and reloaded the state from file system. The instance ssems back to “normal”. Watching it.

[Update - 2023-03-29 15h30 UTC] Build queue did not decreased, the instance is unresponsive: proceeding to purge the queue to regain control, as per https://github.com/jenkins-infra/helpdesk/issues/3474#issuecomment-1488838514

[Update - 2023-03-29 14h00 UTC] Increased the Azure VM capacity to 80 (instead 10 or 20) for each kind (Linux, Windows, etc.) of template

[Update - 2023-03-29 07h30 UTC] The build queue is still full (around 1600 build waiting). Caused by https://github.com/jenkins-infra/helpdesk/issues/3474 (organization scanning trigger).

[Update - 2023-03-28 21h00 UTC] The Azure VM agents are working as expected and started treating builds

[Initial message] There is an outage on ci.jenkins.io since 14h00 UTC. Build requiring a virtual machine are being queued.

More details on https://github.com/jenkins-infra/helpdesk/issues/3459#issuecomment-1487228241.

[Resolved] Jenkins 2.397 and 2.387.2: New Linux Repository Signing Keys

Tue, 28 Mar 2023 08:00:00 +0000

Beginning March 28, 2023, the Jenkins weekly releases will use new repository signing keys for the Linux installation packages. The same change will be made in Jenkins LTS releases beginning April 5, 2023. Administrators of Linux systems must install the new signing keys on their Linux servers before installing Jenkins Jenkins weekly 2.397 or Jenkins LTS 2.387.2.

More details and instructions at https://www.jenkins.io/blog/2023/03/27/repository-signing-keys-changing/

[Resolved] release.ci.jenkins.io migration

Thu, 23 Mar 2023 15:00:00 +0000

[Final message] The migration is ended.

[Initial message]

We’ll migrate release.ci.jenkins.io to a new private cluster for a better segregation of release jobs, and easier maintenance as this new cluster and related resources are managed as code.

This migration will start today Thursday 23th of May at 15h00 UTC. Most of the work has been done in a preliminary phase, it should take two to three hours to finish the migration, with a total unavailability of release.ci.jenkins.io until it’s done.

Important notice: every person needing to work on the next releases will need access to the new private VPN before next Tuesday, as the existing VPN doesn’t allow access to this new cluster and thus the next location of release.ci.jenkins.io. We’ll send further instructions for this by email.

More information and detailed migration plan: https://github.com/jenkins-infra/helpdesk/issues/2844#issuecomment-1463863691

[Resolved] Maintenance on ci.jenkins.io

Tue, 21 Mar 2023 12:00:00 +0000

[Final message] The maintenance is ended.

[Initial message] Maintenance on ci.jenkins.io, down for security advisory. (ref. https://groups.google.com/g/jenkinsci-advisories/c/RmkqB1wAY14).

[Resolved] Maintenance Window on ci.jenkins.io

Fri, 17 Mar 2023 18:27:45 +0000

Maintenance of ci.jenkins.io by the infrastructure team.

[Resolved] Jira (issues.jenkins.io) outage

Sat, 11 Mar 2023 00:00:00 +0000

The JIRA instance at issues.jenkins.io, managed by the Linux Foundation team, will be down for up to 30 minutes beginning at midnight (UTC) Saturday March 11, 2023 so that it can be upgraded.

[Resolved] Bots migration

Thu, 09 Mar 2023 09:00:00 +0000

[Final message] The 3 services are back online and working as intended on their new cluster.

[Initial message] A migration of the bots from prodpublick8s cluster to privatek8s cluster is planned on Thursday 9th of March 2023 at 9h00 UTC, the corresponding services will be down for less than one hour:

jenkins-admin IRC bot (#jenkins-hosting channel on IRC)
github-comment-ops, reacting to comments on GitHub
rss2twitter bot (https://twitter.com/jenkins_release)

See https://github.com/jenkins-infra/helpdesk/issues/2844#issuecomment-1460023896 for more details.

[Resolved] Maintenance on ci.jenkins.io

Wed, 08 Mar 2023 13:00:00 +0000

[Final message] The maintenance is ended.

[Initial message] Maintenance on ci.jenkins.io, down for security advisory (ref. https://groups.google.com/g/jenkinsci-advisories/c/RmkqB1wAY14).

[Resolved] Update Azure Service Principal credential for the AKS cluster prodpublick8s

Tue, 07 Mar 2023 13:00:00 +0000

[Closing] The operation went well and took 30 minutes, including 5 minute of restart for ldap.jenkins.io.

[Opening] We’ll perform an operation on the AKS cluster prodpublick8s the 07 of March 2023 at 13:00 UTC.

The goal of this operation is to rotate the (soon-to-be-expired) Azure Service Principal credential for this cluster.

This operation will restart all the machines of the cluster.

The following services will be restarted (e.g. unavailable during the restart phase):

ldap.jenkins.io (⚠️ authentication on ci.jenkins.io, issues.jenkins.io and repo.jenkins-ci.org is affected)
reports.jenkins.io

The following services will be migrated without downtime (but slower response time during the operation):

accounts.jenkins.io
get.jenkins.io
incrementals.jenkins.io
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

See more at https://github.com/jenkins-infra/helpdesk/issues/3433.

[Resolved] Update AKS clusters Kubernetes version to 1.24

Thu, 23 Feb 2023 10:30:00 +0000

We’re doing an AKS clusters upgrade from 1.23 to 1.24 today, Thursday 23th of February 2023, starting at 10:30 UTC.

See more at https://github.com/jenkins-infra/helpdesk/issues/3387

No outage is expected, but there could be an impact on the Azure artifact caching proxy. In the case it’s not responding, the builds will simply fallback to repo.jenkins-ci.org to retrieve artifacts.

Other services potentially impacted:

accounts.jenkins.io
get.jenkins.io
incrementals.jenkins.io
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
ldap.jenkins.io
plugins.jenkins.io
previews of *.jenkins.io pull requests (infra.ci.jenkins.io)
release.ci.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

[Resolved] Update EKS Kubernetes version to 1.24

Wed, 22 Feb 2023 14:00:00 +0000

We’re doing an EKS clusters upgrade from 1.23 to 1.24 this afternoon, Wednesday 22th of February 2023, starting at 14:00 UTC.

See more at https://github.com/jenkins-infra/helpdesk/issues/3387

No outage is expected, but there could be an impact on the building queue for ci.jenkins.io even with the DOKS cluster to take the load.

[Resolved] Maintenance on ci.jenkins.io

Wed, 15 Feb 2023 11:55:00 +0000

[Final message] The maintenance is ended.

[Initial message] Maintenance on ci.jenkins.io, down for plugin security advisory.

[Resolved] s390x agent restart on ci.jenkins.io

Fri, 03 Feb 2023 16:00:00 -0500

The IBM System 390 agent connected to ci.jenkins.io may be restarted during a maintenance window Friday 3 Feb 2023 from 4pm to 10pm ET.

[Resolved] Outage on jenkins.io

Wed, 25 Jan 2023 09:10:00 +0000

[Final message] We manually tested and purged the fastly cache and saw the errors (on team members affected networks) disappearing.

As not all users where impacted, we think it’s an incident on fastly side : https://www.fastlystatus.com/incident/375724

[Initial message] The jenkins.io website responds with HTTP/503 errors for some users and some pages (but not all).

[Resolved] Outage on ci.jenkins.io

Tue, 24 Jan 2023 12:33:00 +0000

Updates for plugin security advisory.

[Resolved] PostgreSQL maintenance

Fri, 20 Jan 2023 11:30:00 +0000

We’re working on moving the PostgreSQL database to a new virtual network following infra.ci.jenkins.io cluster migration. There might be some services interruption.

[Resolved] infra.ci.jenkins.io cluster migration

Thu, 19 Jan 2023 15:30:00 +0000

[Final message] Migration completed, everything is back to normal.

For users wanting to access infra.ci.jenkins.io, you have to add a new OpenVPN configuration like the one detailed in https://github.com/jenkins-infra/docker-openvpn#howto-get-client-access, but with private.vpn.jenkins.io as remote address instead of vpn.jenkins.io

If you’ve got any problem, please open an help desk issue at https://github.com/jenkins-infra/helpdesk/

[Initial message] In order to migrate infra.ci.jenkins.io to its new cluster (see https://github.com/jenkins-infra/helpdesk/issues/2844), we’ll need to shut down this instance.

Apart from infra.ci.jenkins.io jobs concerning jenkins-infra repositories, jenkins.io preview deployments will also be impacted.

Expected downtime: 1 hour

[Resolved] Outage on ci.jenkins.io

Tue, 27 Dec 2022 13:32:00 +0000

Stopping a large scale job BOM PR-1625 was not ending the jobs and was not deleting agents. Restarted to release the jobs and delete the more than 150 agents that have been idling for 20 hours.

[Resolved] Maintenance with downtime of JFrog Artifactory (repo.jenkins-ci.org)

Sun, 18 Dec 2022 09:00:00 +0000

[Final message] The maintenance has been completed with success by JFrog.

repo.jenkins-ci.org is back online, we resumed ci.jenkins.io and trusted.ci.jenkins.io builds.

[Initial message] December 18, 2022 at 09:00am UTC: our sponsor JFrog will proceed to perform maintenance of our “jenkinsci” Artifactory instance.

Expect a complete downtime of 1 to 6 hours due to the nature of this maintenance. The maintenance involves cloud migration from Google Cloud to Amazon Web Services.

Only Jenkins contributors will be impacted, as no releases or builds on ci.jenkins.io or on developer machines will succeed during the downtime. Jenkins users won’t be impacted.

Please note that ci.jenkins.io will be in “maintenance mode” during the operation: new incoming builds will stay in the build queue while the maintenance mode is enabled, to ensure that no build is lost.

More information in https://github.com/jenkins-infra/helpdesk/issues/3288.

[Resolved] Outage on ci.jenkins.io

Tue, 13 Dec 2022 10:16:00 +0000

[End of outage message] service is back, analysing logs

[Initial message] ci.jenkins.io is not responding

[Resolved] Maintenance on ci.jenkins.io

Tue, 15 Nov 2022 12:30:00 +0000

[Final message]
The maintenance is ended, ci.jenkins.io is back online.

[Initial message]
A maintenance on ci.jenkins.io has been started.

[Resolved] Migration of keycloak bdd

Wed, 09 Nov 2022 15:00:00 +0000

[Final message] The maintenance is ended, admin faster.

[Initial message] A bdd migration is planned the 9 of November at 15:00 UTC.

[Resolved] Maintenance on ci.jenkins.io

Wed, 02 Nov 2022 10:00:00 +0000

[Final message] The maintenance is ended, ci.jenkins.io is back online.

[Initial message] A maintenance on ci.jenkins.io is planned the 2 of November at 10:00 UTC.

[Resolved] Maintenance on ci.jenkins.io

Tue, 01 Nov 2022 13:00:00 +0000

[Final message] The maintenance is ended, ci.jenkins.io is back online.

[Initial message] A maintenance on ci.jenkins.io is planned the 1st of November at 13:00 UTC.

[Resolved] AKS publick8s version update

Thu, 27 Oct 2022 08:00:00 +0000

[close] Upgrade done, no issue or problems to report. see more at https://github.com/jenkins-infra/helpdesk/issues/3053

[open] We’ll do an AKS clusters upgrade from 1.22 to 1.23 this Thursday, 27th of October 2022, starting at 08:00 UTC.

See more at https://github.com/jenkins-infra/helpdesk/issues/3053

No outage is expected, but there could be an impact (1-2 min outage) on the following services if there is an unexpected upgrade issue:

accounts.jenkins.io
beta.accounts.jenkins.io
customize.jenkins.io
get.jenkins.io
incrementals.jenkins.io
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
ldap.jenkins.io
plugins.jenkins.io
release.ci.jenkins.io
reports.jenkins.io
www.jenkins.io
uplink.jenkins.io

[Resolved] AKS tempprivatek8s version update

Wed, 26 Oct 2022 14:50:00 +0000

[close issue] Private K8S Upgraded without any problems to kubernetes 1.23.

[open issue] -We’ll do an AKS clusters upgrade from 1.22 to 1.23 this Wednesday, 26th of October 2022, starting at 14:50 UTC.

-See more at https://github.com/jenkins-infra/helpdesk/issues/3053

we expect the following service to get restarted:

infra.ci.jenkins.io

[Resolved] EKS Kubernetes version update to 1.23

Mon, 24 Oct 2022 12:30:00 +0000

EKS upgraded to 1.23, eks and eks-public

See more at https://github.com/jenkins-infra/helpdesk/issues/3053

[Resolved] Maintenance on ci.jenkins.io

Wed, 19 Oct 2022 11:30:00 +0000

[Final message]
The maintenance is ended, ci.jenkins.io is back online.

[Initial message]
A maintenance on ci.jenkins.io has been started.

[Resolved] Jira (issues.jenkins.io) outage

Fri, 02 Sep 2022 19:33:00 +0000

The JIRA instance at issues.jenkins.io, managed by the Linux Foundation team, is currently down.

Issue IT-24484 has been raised with the Linux Foundation.

[Resolved] Outage on vpn.jenkins.io

Tue, 30 Aug 2022 07:00:00 +0000

[Final message] VPN is back

[Initial message] An outage is currently occuring on vpn.jenkins.io we are investigating.

[Resolved] Maintenance on ci.jenkins.io

Tue, 23 Aug 2022 11:30:00 +0000

[Final message] The maintenance is finished, ci.jenkins.io is back online.

[Initial message] A maintenance on ci.jenkins.io has been started.

[Resolved] Kubernetes Maintenance: Nginx Ingress + Falco upgrades

Tue, 09 Aug 2022 09:00:00 +0000

The two main Kubernetes clusters (prodpublick8s and temp-privatek8s) of the Jenkins infrastructure are going under maintenance mode to upgrade their nginx ingress and Falco controllers.

Nginx Ingress Controller: https://github.com/jenkins-infra/kubernetes-management/pull/2591
Falco Controller: https://github.com/jenkins-infra/kubernetes-management/pull/2596

While no major down time is expected, the following services could be restarted leading to a 1 to 5 minute disruption:

www.jenkins.io
plugins.jenkins.io
javadoc.jenkins.io
get.jenkins.io
fallback.get.jenkins.io
reports.jenkins.io
uplink.jenkins.io
customize.jenkins.io
incrementals.jenkins.io
beta.accounts.jenkins.io
mirror.azure.jenkins.io
jenkins-wiki-exporter.jenkins.io
archives.azure.jenkins.io
release.ci.jenkins.io
infra.ci.jenkins.io
admin.accounts.jenkins.io
grafana.publick8s.jenkins.io

[Resolved] Maintenance on ci.jenkins.io

Wed, 27 Jul 2022 11:30:00 +0000

[Final message]
The maintenance is finished, ci.jenkins.io is back online.

[Initial message]
A maintenance on ci.jenkins.io has been started.

[Resolved] Maintenance Window on ci.jenkins.io

Wed, 06 Jul 2022 10:00:00 +0000

[Initial message] The Jenkins Infra. team is going to perform a set of maintenances on ci.jenkins.io as part of https://github.com/jenkins-infra/helpdesk/issues/3031, https://github.com/jenkins-infra/helpdesk/issues/2984, and https://github.com/jenkins-infra/helpdesk/issues/2961.

These actions require multiple controller restart: that should either queue, cancel or fail the running builds.

[Resolved] JFrog's Artifactory high response time (repo.jenkins-ci.org)

Mon, 04 Jul 2022 06:00:00 +0000

[Closing message] The system went back to normal response time without any action on our side. Monitoring confirmed that it had behaved correctly the 24h hours following the incident.

[Initial message]

Our monitoring alerted us that the response time from repo.jenkins-ci.org is highly unusual: from 5 to 8s (instead of 0.5 to 1s). All contributors of Jenkins are impacted by this slowness, either when downloading dependencies on their machines, or when executing builds on ci.jenkins.io.

JFrog was alerted, we are waiting for any feedback from them.

[Resolved] AKS publick8s version update

Thu, 30 Jun 2022 07:00:00 +0000

We’ll do an AKS clusters upgrade from 1.21 to 1.22 this Thursday, 30th of June 2022, starting at 07:00 UTC.

See more at https://github.com/jenkins-infra/helpdesk/issues/2930

No outage is expected, but there could be an impact (1-2 min outage) on the following services if there is an unexpected upgrade issue:

accounts.jenkins.io
beta.accounts.jenkins.io
customize.jenkins.io
get.jenkins.io
incrementals.jenkins.io
infra.ci.jenkins.io
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
ldap.jenkins.io
plugins.jenkins.io
release.ci.jenkins.io
reports.jenkins.io
www.jenkins.io
uplink.jenkins.io

[Resolved] Maintenance Window on ci.jenkins.io

Wed, 22 Jun 2022 11:30:00 +0000

[Final message] The maintenance is finished, ci.jenkins.io is back online.

[Initial message] A maintenance on ci.jenkins.io has been started.

[Resolved] Update Center certificate issue (updates.jenkins.io)

Tue, 14 Jun 2022 12:42:00 +0000

[Initial message]

There is an issue with the Update Center certificate, resulting in this error message in Jenkins instances:

It seems plugin updates still work.

We’re working on it, you can follow the issue here: https://github.com/jenkins-infra/helpdesk/issues/2950

[Resolved] JFrog's Artifactory slow (repo.jenkins-ci.org)

Mon, 13 Jun 2022 08:30:00 +0000

[Final message] Our JFrog’s Artifactory instance is hosted in Google cloud provider, located in region: GCP Us East, it seems that the behavior we have encountered was due to the service disruption google experienced 3 days ago.

For additional information, you may refer to the corresponding Google Cloud status page.

[Initial message]

We were alerted by datadog that the JFrog’s Artifactory service at https://repo.jenkins-ci.org is slow.

[Resolved] Update Center not updated with new released plugins

Tue, 31 May 2022 12:00:00 +0000

[Close Issue] the new certificate had been issued and stored within the trusted controller. Both the crawler and the update-center jobs run smoothly again since 07:34am UTC.

The plugin “gap” was filled at 09:47am UTC as per https://github.com/jenkins-infra/helpdesk/issues/2950#issuecomment-1143374848.

[Open Issue] The service updates.jenkins.io, which provides a catalog of the jenkins plugins, is not updated since the 31th of May 2022, around noon UTC. It is caused by a safety measure that forbid updates of the catalog: the signing certificate expires soon and must be renewed. More details on https://github.com/jenkins-infra/helpdesk/issues/2950.

[Resolved] Update Center (updates.jenkins.io) flaky, slow with timeouts

Sat, 28 May 2022 02:00:00 +0000

[close issue] We were able to fix the configuration of the service to a fine tuned value. Closing the incident after 24 hours without issues.

[edit] Issue to track changes: https://github.com/jenkins-infra/helpdesk/issues/2960.

[Open Issue] The service https://updates.jenkins.io is experimenting flakiness with a lot of HTTP timeouts. We are searching the source of trouble and we’ll keep you informed.

[Resolved] Debian package installs unavailable

Fri, 27 May 2022 16:50:00 +0000

[Close Issue] A combination of external contributor fixes (HTTPS redirection) and configuration fix (rollback to the correct GPG key) fixed the issues. Closing as confirmed by 3 different users (on GitHub and IRC).

[edit] Details and follow up on the following GitHub issue: https://github.com/jenkins-infra/helpdesk/issues/2957.

[Open Issue] The Debian package installer fails to install from pkg.jenkins.io.

The failure message is:

Err:2 https://pkg.jenkins.io/debian binary/ jenkins 2.349
Redirection from https to 'http://mirrors.jenkins.io/debian/jenkins_2.349_all.deb' is forbidden [IP: 146.75.34.133 443]
Fetched 248 kB in 0s (2256 kB/s)
E: Failed to fetch https://pkg.jenkins.io/debian/binary/jenkins_2.349_all.deb
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

A workaround is available by downloading the weekly installer or the LTS installer from get.jenkins.io and then installing it with dpkg.

LTS workaround

# wget https://get.jenkins.io/debian-stable/jenkins_2.332.3_all.deb
# dpkg -i jenkins_2.332.3_all.deb

Weekly workaround

# wget https://get.jenkins.io/debian/jenkins_2.349_all.deb
# dpkg -i jenkins_2.349_all.deb

[Resolved] pkg.jenkins.io SSL certificate expired

Fri, 27 May 2022 16:50:00 +0000

[Open Issue] The SSL certificate on pkg.jenkins.io and updates.jenkins-ci.org have expired. The SSL certificate on updates.jenkins.io has not expired.

[Resolved] Maintenance on updates.jenkins.io

Fri, 27 May 2022 09:00:00 +0000

[Initial message] We are performing a maintenance on the virtual machine hosting the website https://updates.jenkins.io (“Update Center”) as part of https://github.com/jenkins-infra/helpdesk/issues/2888. No disruption is expected (unless a human mistake is made of course).

[Resolved] Outage on updates.jenkins.io - Invalid certificate

Thu, 19 May 2022 08:10:00 +0000

[edit] Fix applied, but it should take up to 1 hour depending on the DNS cache of your internet providers. More details on https://github.com/jenkins-infra/helpdesk/issues/2888#issuecomment-1131453254.

[Initial message] The website updates.jenkins.io is responding either HTTP/404 pages or “Invalid certificate” (when accessed through HTTPS).

[Resolved] AKS outage, impacts on infra.ci.jenkins.io and ci.jenkins.io

Tue, 17 May 2022 15:15:00 +0000

[close] AKS outage finished: everything is back to normal.

[update] There is an AKS outage on US East 2, which impacts the Virtual Machines allocation: https://app.azure.com/h/MSR7-9Z0/dffcf0.

[infra.ci.jenkins.io] No builds using Kubernetes-based agent can be executed. “Website Pull Requests preview” feature is unavailable.
- Cause: The Kubernetes cluster temp-privatek8s is unable to launch new Virtual machines nodes: infra.ci.jenkins.io cannot handle builds.
[ci.jenkins.io] Increased build queue as only AWS is used for Docker Agents virtual machines.

[initial message] Cluster automatic scaling to handle builds is not working anymore on infra.ci.jenkins.io so it may have difficulties with builds and it could also impact the preview system used within www.jenkins.io

[Resolved] Maintenance Window on ci.jenkins.io

Tue, 17 May 2022 11:31:00 +0000

An upgrade of the plugins of ci.jenkins.io has been started.

See the 2022-05-17 security advisory.

[Resolved] JFrog's Artifactory unavailable (repo.jenkins-ci.org)

Fri, 13 May 2022 07:00:00 +0000

[Initial message]

We were alerted by some users that the JFrog’s Artifactory service at https://repo.jenkins-ci.org is unavailable.

See https://github.com/jenkins-infra/helpdesk/issues/2933 for more details.

[Resolved] Maintenance on pkg.jenkins.io and mirrors.jenkins.io

Sat, 07 May 2022 08:30:00 +0000

[Open Issue] The VM hosting the services mirror.jenkins.io and pk.jenkins.io is slow to respond. The person on-call received alerts last night and had to operate.

We’re going to increase the size of the VM and its disk as a short-term measure: both services will be down while the VM is restarted.

[Resolved] JFrog's Artifactory unavailable (repo.jenkins-ci.org)

Thu, 05 May 2022 23:00:00 +0000

[Initial message]

We were alerted by some users that the JFrog’s Artifactory service at https://repo.jenkins-ci.org is unavailable.

Requests to hte service answers HTTP/5XX errors and the error page at https://repo.jenkins-ci.org/ui/ mentions unhealthy services.

We are working with JFrog to put this service back.

See https://github.com/jenkins-infra/helpdesk/issues/2920 for more details.

[Resolved] Migration of rating.jenkins.io from amazon to azure

Tue, 03 May 2022 07:30:00 +0000

Migration done, nothing to report.

We’ll do a migration of rating.jenkins.io from amazon to azure this Tuesday, 6th of May 2022, starting at 7:30 UTC.

See more at https://github.com/jenkins-infra/helpdesk/issues/1627

No outage is expected.

[Resolved] Outage on ci.jenkins.io

Mon, 02 May 2022 18:15:00 +0000

The ci.jenkins.io service saturated its CPU with Java JIT compilation and Pipeline library loading. Diagnostic information was captured and the service was restarted.

[Resolved] Outage on ci.jenkins.io

Sun, 01 May 2022 12:58:27 +0000

Undiagnosed outage of ci.jenkins.io service. Service was restored without intervention.

[Resolved] JFrog's Artifactory WebUI unavailable (repo.jenkins-ci.org)

Thu, 28 Apr 2022 04:47:00 +0000

[Closing message] JFrogs resolve the issue by restarting the server dealing with the UI

[Initial message]

We were alerted by some users that the Web UI of the JFrog’s Artifactory service at https://repo.jenkins-ci.org/ui is unavailable: it answers HTTP/503 errors with an error page.

This outage does NOT affect the ability to download or upload artefacts in the service.

See https://github.com/jenkins-infra/helpdesk/issues/2904 for more details.

[Resolved] AKS publick8s version update

Wed, 27 Apr 2022 11:00:00 +0000

We’ll do an AKS clusters upgrade from 1.20 to 1.21 this Wednesday, 27th of April 2022, starting at 11:00 UTC.

See more at https://github.com/jenkins-infra/helpdesk/issues/2866

No outage is expected, but there could be an impact (1-2 min outage) on the following services if there is an unexpected upgrade issue:

accounts.jenkins.io
beta.accounts.jenkins.io
customize.jenkins.io
get.jenkins.io
incrementals.jenkins.io
infra.ci.jenkins.io
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
ldap.jenkins.io
plugins.jenkins.io
release.ci.jenkins.io
reports.jenkins.io
www.jenkins.io
uplink.jenkins.io

[Resolved] Maintenance on issues.jenkins.io (JIRA)

Thu, 21 Apr 2022 23:00:00 +0000

The Linux Foundation team that maintains our Jira instance will upgrade the JIRA instance at https://issues.jenkins.io to the latest LTS version.

More information in the following issue: https://github.com/jenkins-infra/helpdesk/issues/2872.

[Resolved] Disruptions on ci.jenkins.io

Wed, 20 Apr 2022 16:45:00 +0000

[Closing Incident] The service ci.jenkins.io is now performing as expected. A post-mortem is expected in the upcoming days, please follow up on the issue https://github.com/jenkins-infra/helpdesk/issues/2893 if you are interested.

[Opening Incident] There are disruptions on the service ci.jenkins.io, see https://github.com/jenkins-infra/helpdesk/issues/2893 for more details and progress.

[Resolved] Outage on ci.jenkins.io

Tue, 12 Apr 2022 11:07:00 +0000

The ci.jenkins.io service went offline and returned to service as reported by uptime robot.

[Resolved] Maintenance Window for Ingress Controllers of Kubernetes Clusters

Fri, 08 Apr 2022 10:30:00 +0000

[Final message]

The maintenance has been completed, all services are operational.

[Initial Message]

A maintenance on the ingress controller of the Azure Kubernetes clusters publick8s and temp-privatek8s is planned for the 08th of April 2022, around 10:30am UTC.

No global outage are expected, but there could be minor impacts (1-2 min outage) on the following services if there is an unexpected upgrade issue:

accounts.jenkins.io
beta.accounts.jenkins.io
customize.jenkins.io
get.jenkins.io
incrementals.jenkins.io
infra.ci.jenkins.io
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
ldap.jenkins.io
plugins.jenkins.io
release.ci.jenkins.io
reports.jenkins.io
www.jenkins.io
uplink.jenkins.io

Reference: https://github.com/jenkins-infra/helpdesk/issues/2875.

[Resolved] Artifactory is slow

Mon, 04 Apr 2022 07:50:00 +0000

[Closing message] The incident was automatically closed in our monitoring system at 13:35 UTC, with the average response time going below the alerting threshold.

[Initial message] We were alerted in parallel by some users and our monitoring that JFrog Artifactory service (repo.jenkins-ci.org) is slow to answer to requests, leading to slowness when downloading artifacts, and eventually build failures due to timeout.

See https://github.com/jenkins-infra/helpdesk/issues/2864 for more details.

[Resolved] Maintenance operation on infra.ci.jenkins.io + jenkins.io's pull requests

Wed, 30 Mar 2022 13:30:00 +0000

[Initial message] As part of https://github.com/jenkins-infra/helpdesk/issues/2840, the Jenkins infra team will perform a maintenance to infra.ci.jenkins.io.

Impact is for the contributors: the previews on the GitHub pull requests for jenkins.io will fail until the operation is completed.

[Resolved] Artifactory is slow

Wed, 23 Mar 2022 14:00:00 +0000

[Final message] Incident was tracked and covered by JFrog: https://status.jfrog.io/incidents/8hh7t43bs0xt.

We are working on it, see https://github.com/jenkins-infra/helpdesk/issues/2849 for more details.

[Resolved] Maintenance Window on ci.jenkins.io

Tue, 15 Mar 2022 13:45:00 +0000

An upgrade of the plugins of ci.jenkins.io has been started. More informations to come.

[Resolved] Maintenance Window on AKS

Tue, 08 Mar 2022 08:00:00 +0000

[Final Message]

Maintenance finished. LDAP was unavailable during restart from 10:10 UTC until 10:15 UTC as per https://github.com/jenkins-infra/helpdesk/issues/2810.

[Initial Message]

A maintenance on the ingress controller of our Azure Kubernetes clusters (publick8s and temp-privatek8s) is planned for the 08th of March at 08:00am UTC.

No global outage are expected, but there could be minor impacts (1-2 min outage) on the following services if there are any unexpected upgrade issues:

accounts.jenkins.io
beta.accounts.jenkins.io
customize.jenkins.io
get.jenkins.io
incrementals.jenkins.io
infra.ci.jenkins.io
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
ldap.jenkins.io
plugins.jenkins.io
release.ci.jenkins.io
reports.jenkins.io
www.jenkins.io
uplink.jenkins.io

Reference: jenkins-infra/helpdesk#2809

[Resolved] Problems publishing artifacts in Artifactory (2022-03-04)

Fri, 04 Mar 2022 18:00:00 +0000

[Final Message]

The repository has been cleaned up (~120 Gb) and additional measures are being applied to stay under the storage limit. Deploy to the repository can be done again.

[Initial Message] We’ve observed problems publishing artifacts in Artifactory, logs indicate the disk is full. So if you’re unable to release stuff the next few days, that is probably why.

https://groups.google.com/g/jenkinsci-dev/c/TZakKUjUm9Q/m/Pirq0fqEBAAJ?utm_medium=email&utm_source=footer as followup

[Resolved] Maintenance window on ci.jenkins.io (2022-02-15)

Tue, 15 Feb 2022 13:30:00 +0000

The 2022-02-15 at 13:30 UTC, expect an outage on ci.jenkins to update the plugins installed on this instance.

Security advisory is available at https://www.jenkins.io/security/advisory/2022-02-15/.

[Resolved] Maven agents not allocating on ci.jenkins.io

Fri, 11 Feb 2022 02:14:00 +0000

[Update 11th of February 2022 - 19:00 UTC] An operation is required on the underlying Kubernetes cluster. We plan to do it monday 14th starting at 08:00am UTC.

It will result in a downtime of all the agents with labels maven and maven-11: builds will be queued until the cluster is working again. VM agents will keep being allocated.

[Update 11th of February 2022 - 15:20 UTC] Cluster had been scaled manually to handle builds until the root cause is fixed. Read more at https://github.com/jenkins-infra/helpdesk/issues/2780.

[Initial Message] Agents with labels maven and maven-11 fail to allocate on AWS. Infrastructure team found that the AWS autoscaler was failing. Investigation is in progress.

[Resolved] Maintenance window on ci.jenkins.io

Wed, 09 Feb 2022 12:14:28 +0000

An upgrade of ci.jenkins.io from 2.319.2 to 2.319.3. See security advisory, changelog, and upgrade guide.

[Resolved] TLS error `SSL_ERROR_BAD_CERT_DOMAIN` on mirrors.jenkins.io and mirrors.jenkins-ci

Mon, 07 Feb 2022 12:37:00 +0000

[Final message] The datadog alert on the https call has been removed as this service as never been served as https.

[Initial message] The services mirrors.jenkins-ci.org and mirrors.jenkins.io shows an error SSL_ERROR_BAD_CERT_DOMAIN due to a certificate generated with the wrong name. Only HTTPS (with TLS) access is impacted: plain HTTP is still working as expected.

We are investigating the problem.

[Resolved] ci.jenkins.io down due to overload

Fri, 21 Jan 2022 17:56:43 +0000

Overload detected on ci.jenkins.io required a service restart. Reported to Mark Waite email by uptimerobot.

[Resolved] repo.jenkins-ci.org down

Mon, 17 Jan 2022 10:38:00 +0000

[Final message] The service is back to normal.

[Initial message] The service repo.jenkins-ci.org is down. A ticket has been raised with JFrog support.

[Resolved] Disruptions on ci.jenkins.io

Thu, 13 Jan 2022 15:00:00 +0000

There are disruptions on the service ci.jenkins.io, see https://github.com/jenkins-infra/helpdesk/issues/2733 for more details and progress.

[Resolved] Maintenance Window on ci.jenkins.io

Wed, 12 Jan 2022 14:44:54 +0000

An upgrade of ci.jenkins.io from 2.319.1 to 2.319.2. See security advisory, changelog, and upgrade guide.

[Resolved] Maintenance Window on AKS

Mon, 20 Dec 2021 11:00:00 +0000

[Final message]

The maintenance has been completed, all services are operational.

[Initial Message]

A maintenance on the ingress controller of the Azure Kubernetes cluster publick8s is planned for the 20th of December, around 11:00am UTC.

No global outage are expected, but there could be minor impacts (1-2 min outage) on the following services if there is an unexpected upgrade issue:

accounts.jenkins.io
beta.accounts.jenkins.io
customize.jenkins.io
get.jenkins.io
incrementals.jenkins.io
infra.ci.jenkins.io
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
ldap.jenkins.io
plugins.jenkins.io
release.ci.jenkins.io
reports.jenkins.io
www.jenkins.io
uplink.jenkins.io

Reference: https://github.com/jenkins-infra/kubernetes-management/pull/1748.

[Resolved] Jira (issues.jenkins.io) outage

Wed, 15 Dec 2021 15:50:00 +0000

[Final Message]

The outage was due to an external problem with AWS.

[Initial Message]

The JIRA instance at issues.jenkins.io, managed by the Linux Foundation team, is currently down.

We are escalating and trying to fix the issue.

[Resolved] accounts.jenkins.io down

Fri, 10 Dec 2021 23:00:00 +0000

[Final Message]

Our investigations showed we are not vulnerable to the CVE-2021-44228 nor the CVE-2021-45046.

[Initial Message]

We decided to disable the Jenkins accounts service temporarily out of an abundance of caution.

See the Jenkins blog post about CVE-2021-44228

[Resolved] AKS publick8s version update

Thu, 25 Nov 2021 08:00:00 +0000

INFRA-3118 Notes

[Final Message] The upgrade is finished without any incident, all services are up and running.

Cat tax:

[Initial Message] We’ll do an AKS and EKS clusters upgrade from 1.19 to 1.20 this Thursday, 25th of November 2021, starting at 08:00 UTC.

The service ldap.jenkins.io service might have a downtime due to its PV.

Other services might have a downtime too if their probes take more than one second to respond due to the new timeoutPeriod enforced default value.
Note: the official Jenkins helm chart we’re using has already set the default timeout period to 5 seconds

Apart from that, no outage is expected, but there could be an impact (1-2 min outage) on the following services if there is an unexpected upgrade issue:

accounts.jenkins.io
beta.accounts.jenkins.io
customize.jenkins.io
get.jenkins.io
incrementals.jenkins.io
infra.ci.jenkins.io
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
ldap.jenkins.io
plugins.jenkins.io
release.ci.jenkins.io
reports.jenkins.io
www.jenkins.io
uplink.jenkins.io

[Resolved] jenkins.io outage

Thu, 18 Nov 2021 10:47:00 +0000

[Final Message] The website jenkins.io and all its sub pages are back online.

[Initial Message] The website jenkins.io and all its sub pages (blog, documentation) are experimenting an outage: TLS errors and HTTP/503. We are working on this outage.

[Resolved] HTTP access to archives.jenkins.io unreachable

Fri, 05 Nov 2021 16:49:00 +0000

The system needed an operating system restart to use the most recent kernel patch. The operating system patch install and restart was completed with 60 seconds or less of downtime.

The SSL certificate on archives.jenkins.io was scheduled to expire in less than 10 days. The issue was resolved by using the notes from a ci.jenkins.io certificate renewal failure that is recorded in the Jenkins infra runbook. No outage due to the SSL certificate renewal, just prevented an outage when the certificate would expire.

[Resolved] Maintenance Window on ci.jenkins.io

Thu, 04 Nov 2021 11:24:40 +0000

An upgrade of ci.jenkins.io from 2.303.2 to 2.303.3. See security advisory, changelog, and upgrade guide.

[Resolved] Maintenance on Plugin Site

Tue, 26 Oct 2021 15:45:00 +0000

[Final Message] The service is back online.

[Initial Message] A maintenance on the Plugin site is planned, to upgrade its dependencies and fix an ongoing DNS issue.

The website plugins.jenkins.io should not be impacted as it’s cached by a CDN, worst case scenario is an outage of ~1 min time to rollback
The website plugins.origin.jenkins.io will be stopped for the duration of the maintenance

[Resolved] Maintenance Window on AKS

Tue, 26 Oct 2021 08:30:00 +0000

[Final message]

The maintenance has been completed, all services are operational.

[Edit - Monday 25th]

We had to delay the operation as we discovered last minutes items to be changed. Next operation window is planned Tuesday, the 26th of October, at 08:30am UTC.

[Initial Message - Friday 22th]

A maintenance on the ingress controller of the Azure Kubernetes cluster publick8s is planned for the 25th of October, around 12:00am (noon) UTC.

No outage is expected, but there could be an impact (1-2 min outage) on the following service if there is an unexpected upgrade issue:

accounts.jenkins.io
beta.accounts.jenkins.io
customize.jenkins.io
get.jenkins.io
incrementals.jenkins.io
infra.ci.jenkins.io
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
ldap.jenkins.io
plugins.jenkins.io
release.ci.jenkins.io
reports.jenkins.io
www.jenkins.io
uplink.jenkins.io

Reference: https://github.com/jenkins-infra/charts/pull/1424.

[Resolved] Maintenance Window on ci.jenkins.io

Wed, 06 Oct 2021 15:45:00 +0000

Jenkins core and plugins were updated on ci.jenkins.io with security fixes from Jenkins security advisory 2021-10-06.

[Resolved] Convert Jira database to utf8mb4

Tue, 05 Oct 2021 16:00:00 +0000

The Linux Foundation will convert Jira database and tables from utf8 to utf8mb4 and solve collation errors in Jira. issues.jenkins.io will be unavailable during the maintenance

Twitter Discussion Jira

[Resolved] LDAP server down

Fri, 01 Oct 2021 00:00:00 +0000

[Final Message] The root certificate have been rotated, resulting in an automatic repair of all broken services.

archives.jenkins.io was back online at 09:55 UTC
ci.jenkins.io was back online at 09:55 UTC
ldap was back online at 09:55 UTC

[Initial Message] Due to DST Root CA X3 Expiration our LDAP server is currently down, we are working to fix it.

[Resolved] Puppet Outage on VM-hosted services

Thu, 23 Sep 2021 10:01:00 +0000

[Final Message] The Puppet infrastructure had been fixed, resulting in an automatic repair of all broken services.

archives.jenkins.io was back online at 12:58 UTC
ci.jenkins.io was back online at 13:00 UTC

[Initial Message] An outage on our Puppet infrastructure system is currently happening.

A mitigation had been applied resulting in disabling configuration update to avoid breaking services.

The following services had been put down before we could mitigate:

ci.jenkins.io
archives.jenkins.io

We are working on fixing these services and the Puppet infrastructure

[Resolved] Maintenance with downtime on the AKS (Azure Kubernetes Service) cluster `publick8s`

Mon, 20 Sep 2021 09:45:00 +0000

[Final Message] The AKS cluster was upgraded successfully to ensure that CVE-2021-25741 is fixed.

AKS Control plane is now using Kubernetes 1.19.13
AKS Node pools are all using https://github.com/Azure/AKS/releases/tag/2021-09-09 images

[Initial Message] A maintenance on the AKS cluster that requires a restart of each service and node is currently applied.

The services ldap.jenkins.io and plugins.jenkins.io will be restarted with an expected downtime of 3 to 5 minutes (time for AKS to bind the data volumes)
All other services should remain up and running.

[Resolved] Jenkins wiki offline due to vulnerability

Thu, 02 Sep 2021 08:15:00 +0000

[Final Message] The service is back online in the form of an nginx image serving the exported confluence data as HTML.

Ref: Service to redirect wiki requests to exported Confluence content or to a search on jenkins.io

[Initial Message] Jenkins wiki, https://wiki.jenkins.io, is offline due to a remote code execution security vulnerability.

[Resolved] Restart Jenkins wiki https://wiki.jenkins.io

Wed, 01 Sep 2021 00:00:00 +0000

Jenkins wiki needs to be restarted about once a month.

[Resolved] Maintenance Window on ci.jenkins.io

Tue, 31 Aug 2021 12:36:00 +0000

Plugins were updated on ci.jenkins.io with security fixes from Jenkins security advisory 2021-08-31.

[Resolved] Maintenance with downtime on https://wiki.jenkins.io (Jenkins wiki)

Sun, 29 Aug 2021 18:00:00 +0000

SSL certificate on the Jenkins wiki will expire in 14 days. Restart the Apache service to use the new certificate.

Downtime less than 3 minutes.

[Resolved] Maintenance with downtime on https://repo.jenkins-ci.org (JFrog repository service)

Fri, 27 Aug 2021 13:00:00 +0000

[Update] Intervention done successfully with 5 minutes downtime only.

JFrog support team will perform a maintenance on their service that will impact our repository at https://repo.jenkins-ci.org.

A downtime of almost 30 minutes is expected, starting at 13:00 UTC (01:00pm UTC), the 27th of August 2021.

[Resolved] HTTP access to archives.jenkins.io unreachable

Fri, 16 Jul 2021 23:50:00 +0000

The HTTP server on archives.jenkins.io was not responding to requests. The mirror system on get.jenkins.io excluded archives.jenkins.io because it was unreachable. Issue was resolved by stopping and starting the apache2 service on the host (okra.jenkins.io).

[Resolved] Update center content not refreshed, recent plugin releases not visible

Wed, 07 Jul 2021 10:00:00 +0000

The update center had its content not refreshed from the 07th of July 2021 until the 12th of July 2021.

More details on what went wrong and how it was fixed can be found in the following page: INFRA-3023.

[Resolved] Maintenance Window on ci.jenkins.io

Mon, 21 Jun 2021 10:04:00 +0000

Update: operation took 2 minutes, and restart was successful

==============================

A restart of ci.jenkins.io is planned with the following changes:

https://issues.jenkins.io/browse/INFRA-3010 => Remove the plugin azure-commons
Plugins upgrade:
- Bump ec2-plugin to 1.62
- Bump Azure VM to 783.v58077630847d
- Bump Code Coverage API to 1.4.0
- Bump JaCoCo to 3.3.0
- Bump Pipeline: Multibranch to 2.26
- Bump Pipeline: SCP Step to 2.13
- Bump Pipeline: Shared Groovy Libraries to 2.20

[Resolved] AKS publick8s version update

Thu, 17 Jun 2021 08:00:00 +0000

INFRA-3005 NOTES

[Closing Maintenance]

The update to Kubernetes 1.19.11 was successfully finished at 14:00 UTC
The service ldap.jenkins.io had 2 downtimes post-mortem, resulting in authentication not possible on the JIRA and the plugin releases where not possible during the following time windows:
- From 08:08 UTC until 08:13 UTC (6 min)
- From 10:40 UTC until 10:41 UTC (1 min)

[Initial Message] We’ll do an AKS cluster upgrade.

The service ldap.jenkins.io service will be restarted. All other services should remain up and running.

[Resolved] Websites are down

Tue, 08 Jun 2021 09:58:00 +0000

Due to a Fastly world-wide incident, three Jenkins services that rely on Fastly to delivery content are impacted. This outage affects www.jenkins.io, pkg.jenkins.io and plugins.jenkins.io

[Resolved] Agents randomly disconnected on ci.jenkins.io

Tue, 08 Jun 2021 05:35:00 +0000

Update at 2021-06-08 11:20 UTC

ci.jenkins.io is back to normal since 2021-06-08 11:08 UTC after running a series of operations including a service full restart.

Initial Message at 2021-06-08 09:35 UTC

Agents are randomly disconnected on ci.jenkins.io with the following message in pipeline logs:

Channel "hudson.remoting.Channel@<ID>:JNLP4-connect connection from <IP>:<IP:PORT>": Remote call on JNLP4-connect connection from <IP>:<IP:PORT> failed. The channel is closing down or has closed down

Builds using an EC2 or a permanent agents are reconnected most of the time but might fail
Builds on fungible agents such as ACI are failing 100% when this error happens in build logs and need to be replayed

Please note that ci.jenkins.io might be restarted multiple time until the incident is disconnected.

[Resolved] repo.jenkins-ci.org issues

Mon, 07 Jun 2021 12:37:00 +0000

2021-06-08 07:37:00

This PR persist in time the hotfix we applied.

2021-06-07 16:37:00

Two IPs have been manually added to ldap.jenkins.io configuration.

2021-06-07 12:37:00

Due to a recent upgrade done by Jfrog on repo.jenkins-ci.org, authentication is broken. Everything else seems to be working correctly.

We are contacting their support.

Follow up will be communicated on the jenkins-infra mailing. -> here

[Resolved] ACI agents fail to allocate for ci.jenkins.io

Thu, 03 Jun 2021 15:41:00 +0000

ci.jenkins.io is unable to start any ACI agent and there is an increasing build queue.

Investigation is in progress to resolve the issue and identify the root cause, ci.jenkins.io is put in shutdown mode to limit the amount of builds to be treated.

[edit]

Plugin and configuration upgrades around EC2 and Azure-* have been done
all the builds have been cleaned
ci.jenkins.io restarted and back to normal state

[Resolved] Windows LTS Package Unavailable

Wed, 02 Jun 2021 18:59:00 +0000

Due to a technical issue in the release process, the Windows package for the LTS Jenkins release was unavailable during almost 13 hours.

The release process has been fixed and executed successfully the 03rd of June 2021 at 06:50 UTC and finished at 07:02 UTC.

At 07:00 UTC, the Windows package was available again from https://jenkins.io/download.

[Resolved] AKS (Azure Kubernetes Service) cluster `publick8s` principal security credential expired

Wed, 02 Jun 2021 17:53:00 +0000

The security credential for the AKS cluster expired the 2021-06-02.

Any internal request to the Azure API (volume mounts, network integration, etc.) is refused since 2021-06-02T17:53:00-00:00.

The operation to renew this certificate started at 2021-06-03 - 15:15 UTC and is resulting in all services hosted on the cluster to be restarted.

No public interuption except for the LDAP service should happen.

[Edit]

There have been a disruption of nearly 5 min on the service plugins.jenkins.io around 16:00 UTC, reported in the Gitter channel #jenkinsci/docs by Zbynek (thanks!) Services logs indicate an issue with the Azure file storage while

[Resolved] wiki.jenkins.io restart

Tue, 01 Jun 2021 23:00:00 +0000

The wiki.jenkins.io process will be stopped and restarted.

[Resolved] Update center job on trusted.ci.jenkins.io is blocked

Sun, 30 May 2021 08:13:00 +0000

The Jenkins update center relies on a job that runs periodically on trusted.ci.jenkins.io to record the latest releases. That job blocked on May 30 and has not been performing updates.

Timeline

2021-05-30T08:13 - Update center job on trusted.ci.jenkins.io reports blocked waiting for agent-1
2021-06-01T19:45 - Mark Waite investigates Jenkins 2.296 not being visible on www.jenkins.io, detects blocked update center job
2021-06-01T23:50 - EU team hands over, detects there was a DNS resolution issue in trusted.ci.jenkins.io and changed the configuration of agent-1 accordingly: the update_center build moved from the “waiting” state to “building” state on agent-1
2021-06-02T01:50 - After a manualy apply of the update_center + a successful build, the latest weekly 2.296 appears on the update center.json

[Resolved] ACI agents fail to allocate for ci.jenkins.io

Sat, 29 May 2021 22:41:00 +0000

A plugin upgrade has caused allocation of Azure Container Instances to fail on ci.jenkins.io. Investigation is in progress to resolve the issue and identify the root cause.

Issue was resolved by downgrading 5 Azure plugins to the versions that were previously working on ci.jenkins.io

Azure Container Instance outage on ci.jenkins.io

Timeline - May 29, 2021

16:37 Denver - Mark Waite updated plugins on ci.jenkins.io, restarted ci.jenkins.io
16:55 Denver - Updated JDK 8 tool definition from 8u242 to 8u292
17:15 Denver - Confirmed builds with JDK 8u292 tools working on amd64, arm64, s390x, and popc64le
17:39 Denver - Started Jenkins core build, detected that maven and maven-11 agents were not being allocated
17:41 Denver - System log for ACI agents reports Cannot provision: template for label maven is not available now, because it failed to provision last time.
17:50 Denver - Downgraded Azure container plugin to previously installed version, restarted Jenkins
18:00 Denver - Downgraded plugin reports NoSuchMethodError, likely indicator that it requires an API in one of the 50+ plugins that were upgraded at 16:37 Denver time
18:01 Denver - Updated Azure container plugin to latest release
18:17 Denver - Confirmed problem is still visible after upgrade
18:19 Denver - Record plugin file system times in case rollback needed
19:14 Denver - Identify candidate plugins for rollback
- azure-sdk from 12.vc102aedd3c66 down to 4.vcb202d9010c1
- azure-credentials from 182.v3ccd4a755864 down to 177.v816b81058012
- azure-vm-agents from 780.v50d067d02f76 down to 774.v0cee503baa25
- windows-azure-storage from 358.v5c001416d74f down to 355.v4da08e72a251
- azure-container-agents from 207.v3ad9931bf69e down to 201.v2afdce22b4cf
Review dependencies of those plugins
- Used plugin installation manager to install a local copy of those 5 plugins at their previous versions as confirmation that they should be workable at their previous versions as a group. Not a perfect check, but worth the check.
20:01 Denver - Downgrade those 5 plugins and restart ci.jenkins.io
20:03 Denver - Restart is complete, Manage Plugins shows those 5 plugins have upgrades available
20:09 DEnver - Outage resolved, ACI agents are allocated on ci.jenkins.io

Plugins that were upgraded at 16:37 Denver time

-rw-r--r-- 1 jenkins jenkins 4200464 May 29 22:37 plugins/jacoco.jpi
-rw-r--r-- 1 jenkins jenkins 380040 May 29 22:37 plugins/kubernetes.jpi
-rw-r--r-- 1 jenkins jenkins 3091463 May 29 22:37 plugins/ldap.jpi
-rw-r--r-- 1 jenkins jenkins 86675 May 29 22:37 plugins/blueocean-commons.jpi
-rw-r--r-- 1 jenkins jenkins 277630 May 29 22:37 plugins/blueocean-jwt.jpi
-rw-r--r-- 1 jenkins jenkins 39703 May 29 22:37 plugins/command-launcher.jpi
-rw-r--r-- 1 jenkins jenkins 468323 May 29 22:37 plugins/cloudbees-bitbucket-branch-source.jpi
-rw-r--r-- 1 jenkins jenkins 8089625 May 29 22:37 plugins/subversion.jpi
-rw-r--r-- 1 jenkins jenkins 102390 May 29 22:37 plugins/blueocean-rest.jpi
-rw-r--r-- 1 jenkins jenkins 14124 May 29 22:37 plugins/blueocean-jira.jpi
-rw-r--r-- 1 jenkins jenkins 86109 May 29 22:37 plugins/workflow-multibranch.jpi
-rw-r--r-- 1 jenkins jenkins 308017 May 29 22:37 plugins/branch-api.jpi
-rw-r--r-- 1 jenkins jenkins 35870 May 29 22:37 plugins/blueocean-pipeline-scm-api.jpi
-rw-r--r-- 1 jenkins jenkins 2363613 May 29 22:37 plugins/jenkins-design-language.jpi
-rw-r--r-- 1 jenkins jenkins 1076110 May 29 22:37 plugins/blueocean-core-js.jpi
-rw-r--r-- 1 jenkins jenkins 1413507 May 29 22:37 plugins/blueocean-web.jpi
-rw-r--r-- 1 jenkins jenkins 967614 May 29 22:37 plugins/blueocean-rest-impl.jpi
-rw-r--r-- 1 jenkins jenkins 192974 May 29 22:37 plugins/blueocean-pipeline-api-impl.jpi
-rw-r--r-- 1 jenkins jenkins 66662 May 29 22:37 plugins/blueocean-github-pipeline.jpi
-rw-r--r-- 1 jenkins jenkins 2745123 May 29 22:37 plugins/blueocean-dashboard.jpi
-rw-r--r-- 1 jenkins jenkins 703681 May 29 22:37 plugins/blueocean-personalization.jpi
-rw-r--r-- 1 jenkins jenkins 33137044 May 29 22:37 plugins/azure-sdk.jpi
-rw-r--r-- 1 jenkins jenkins 58121 May 29 22:37 plugins/azure-credentials.jpi
-rw-r--r-- 1 jenkins jenkins 5947031 May 29 22:37 plugins/jira.jpi
-rw-r--r-- 1 jenkins jenkins 28517 May 29 22:37 plugins/display-url-api.jpi
-rw-r--r-- 1 jenkins jenkins 2068729 May 29 22:37 plugins/pipeline-utility-steps.jpi
-rw-r--r-- 1 jenkins jenkins 776815 May 29 22:37 plugins/plugin-util-api.jpi
-rw-r--r-- 1 jenkins jenkins 486809 May 29 22:37 plugins/font-awesome-api.jpi
-rw-r--r-- 1 jenkins jenkins 261324 May 29 22:37 plugins/azure-vm-agents.jpi
-rw-r--r-- 1 jenkins jenkins 957359 May 29 22:37 plugins/credentials.jpi
-rw-r--r-- 1 jenkins jenkins 16099656 May 29 22:38 plugins/analysis-model-api.jpi
-rw-r--r-- 1 jenkins jenkins 226188 May 29 22:38 plugins/mercurial.jpi
-rw-r--r-- 1 jenkins jenkins 54575 May 29 22:38 plugins/blueocean-git-pipeline.jpi
-rw-r--r-- 1 jenkins jenkins 127789 May 29 22:38 plugins/matrix-auth.jpi
-rw-r--r-- 1 jenkins jenkins 34438 May 29 22:38 plugins/pubsub-light.jpi
-rw-r--r-- 1 jenkins jenkins 11342134 May 29 22:38 plugins/warnings-ng.jpi
-rw-r--r-- 1 jenkins jenkins 472289 May 29 22:38 plugins/windows-slaves.jpi
-rw-r--r-- 1 jenkins jenkins 1360650 May 29 22:38 plugins/cvs.jpi
-rw-r--r-- 1 jenkins jenkins 2425761 May 29 22:38 plugins/ec2.jpi
-rw-r--r-- 1 jenkins jenkins 634219 May 29 22:38 plugins/workflow-cps.jpi
-rw-r--r-- 1 jenkins jenkins 839972 May 29 22:38 plugins/timestamper.jpi
-rw-r--r-- 1 jenkins jenkins 1102491 May 29 22:38 plugins/windows-azure-storage.jpi
-rw-r--r-- 1 jenkins jenkins 61490 May 29 22:38 plugins/blueocean-config.jpi
-rw-r--r-- 1 jenkins jenkins 92076 May 29 22:38 plugins/blueocean-bitbucket-pipeline.jpi
-rw-r--r-- 1 jenkins jenkins 23581 May 29 22:38 plugins/blueocean-events.jpi
-rw-r--r-- 1 jenkins jenkins 1057838 May 29 22:38 plugins/blueocean-pipeline-editor.jpi
-rw-r--r-- 1 jenkins jenkins 15198 May 29 22:38 plugins/blueocean-i18n.jpi
-rw-r--r-- 1 jenkins jenkins 8105 May 29 22:38 plugins/blueocean.jpi
-rw-r--r-- 1 jenkins jenkins 308385 May 29 22:38 plugins/github-branch-source.jpi
-rw-r--r-- 1 jenkins jenkins 44276 May 29 22:38 plugins/structs.jpi
-rw-r--r-- 1 jenkins jenkins 96874 May 29 22:38 plugins/workflow-durable-task-step.jpi
-rw-r--r-- 1 jenkins jenkins 35277 May 29 22:38 plugins/kubernetes-credentials.jpi
-rw-r--r-- 1 jenkins jenkins 66978 May 29 22:38 plugins/ansicolor.jpi
-rw-r--r-- 1 jenkins jenkins 99985 May 29 22:38 plugins/groovy.jpi
-rw-r--r-- 1 jenkins jenkins 2337135 May 29 22:38 plugins/configuration-as-code.jpi
-rw-r--r-- 1 jenkins jenkins 189508 May 29 22:38 plugins/script-security.jpi
-rw-r--r-- 1 jenkins jenkins 822801 May 29 22:38 plugins/support-core.jpi
-rw-r--r-- 1 jenkins jenkins 98403 May 30 00:00 plugins/azure-container-agents.jpi

[Resolved] Windows Weekly Package Unavailable

Tue, 25 May 2021 11:30:00 +0000

Due to a technical issue in the release process, the Windows package for the weekly Jenkins release was unavailable from 2021-05-25 11:30 UTC until 2021-05-26 07:51 UTC.

A Post-Mortem is available on hackmd.io.

[Resolved] Nginx ingress upgrade

Mon, 03 May 2021 11:00:00 +0000

**Update 2021-05-03 11:58 UTC

The operation is successful, we are now waiting several hours before deleting the temporary resources that we created during the transition to be sure that no DNS traffics is redirected to those locations. We didn’t notice service downtime during the process.

There is no ETA at the moment.

We are upgrading the nginx ingress controllers on the Kubernetes cluster publick8s. While no down time is expected, the risk during this upgrade is to stop redirecting HTTP/HTTPS traffics to the following services:

Public services:

www.jenkins.io
plugins.jenkins.io
javadoc.jenkins.io
get.jenkins.io
fallback.get.jenkins.io
reports.jenkins.io
uplink.jenkins.io
customize.jenkins.io
incrementals.jenkins.io
beta.accounts.jenkins.io
mirror.azure.jenkins.io
jenkins-wiki-exporter.jenkins.io
archives.azure.jenkins.io

Private services:

release.ci.jenkins.io
infra.ci.jenkins.io
admin.accounts.jenkins.io
grafana.publick8s.jenkins.io

The operation consists of deploying temporary ingress controllers listening on ephemeral IPs.

[Resolved] get.jenkins.io disk quota exceeded

Mon, 19 Apr 2021 18:50:00 +0000

Issue 4270 Infra mailing list NOTES

The get.jenkins.io and fallback.get.jenkins.io pods are unable to mount their Azure file storage. The mount failure prevents the files from being served by get.jenkins.io.

2021-04-19T21:12:00-00:00

The disk quota issue has been solved using Microsoft documentation So mirrorbits containers are back to normal. We tested that service was operational using https://fallback.get.jenkins.io At the moment, the traffic is still routed to pkg.origin.jenkins.io but it has no impact on the availability of the service. We will switch back the traffic at a better time.

We validated that the service was operational using this mirrorlist

[Resolved] Jira maintenance

Thu, 15 Apr 2021 10:00:00 +0000

The Linux Foundation team that maintains our Jira instance will install a Jira update and operating system updates.

Maintenance details:

What:

issues.jenkins.io Why:
upgrade jira to 8.13.5 (latest LTS)
install OS security updates
reboot jira instance Maitenance duration: 1h. Short downtime is expected for 10-20min as we need to reboot that system

Discussion

[Resolved] AKS publick8s version update

Wed, 14 Apr 2021 07:30:00 +0000

INFRA-2944 NOTES

We’ll do an AKS cluster upgrade.

The service ldap.jenkins.io service will be restarted. All other services should remain up and running.

[Resolved] Update sites not updating

Sun, 21 Mar 2021 10:00:00 +0000

Update sites are currently not updating. Plugin updates released since this outage started are not made available to users.

[Resolved] get.jenkins.io dns update

Sun, 13 Dec 2020 13:20:00 +0000

INFRA-2846

Because of the previous get.jenkins.io outage, we temporarily redeployed the service on a virtual machine. The azure file storage issue is now resolved and the service ready to handle the traffic as it used to be. I am going to update the dns record to redirect from the temporary virtual machine to our kubernetes cluster.

No downtime is expected

[Resolved] AKS publick8s version update

Mon, 07 Dec 2020 13:00:00 +0000

We’ll do an AKS cluster upgrade.

[Resolved] get.jenkins.io issues

Thu, 12 Nov 2020 15:54:00 +0000

2020-11-23: Azure is investigating the root cause of the issue
2020-11-19: The Azure blog storage is back without additional information
*2020-11-14: We temporarily deploy get.jenkins.io outside Azure
2020-11-17: We have a call with Azure support today

The network volume used by get.jenkins.io is currently down. This outage affect every services relying on get.jenkins.io like updates.jenkins.io and pkg.jenkins.io

Twitter Discussion

[Resolved] Maintenance Window

Sat, 13 Jun 2020 15:54:00 +0000

Just began - We’re currently shutting down the network.