Cert.ci.jenkins.io on Jenkins Status Page

[Resolved] [publick8s cluster] Azure publick8s Upgrade to Kubernetes 1.33.5

Mon, 15 Dec 2025 08:30:00 +0000

[Final Message] upgrade went well, no impact on production.

[Initial Message] Monday 15 December 2025 at 08:30PM UTC, we’ll upgrade the AKS(Azure) cluster publick8s to Kubernetes 1.33.5. It should take between 60 and 120 min.

The expected impacts are:

Restart of the following services, with a downtime between 2 and 15 min (depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-in/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) on:

get.jenkins.io (mirror redirector)
issue-redirect.jenkins.io
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

More details in https://github.com/jenkins-infra/helpdesk/issues/4820#issuecomment-3548729217.

[Resolved] Azure outage on the '*.ci.jenkins.io' private controllers

Wed, 29 Oct 2025 15:45:00 +0000

[Final Message] The outage seems has been resolved at 00:05am on the 30 October.

[Initial Message] Azure is facing a global outage (Tracking ID: YKYN-BWZ at https://azure.status.microsoft/en-us/status/history/) which prevents access to the Azure Portal and to the Azure Identity Management API.

It means that our private controllers might be unable to spin up new VM agents: we expect delays in

The publication of our websites (both previews and production)
Jobs running in private controllers
Execution of infrastructure related tasks (Terraform and Kubernetes jobs)

Jobs will be queued and eventually aborted due to timeouts.

[Resolved] Azure cluster publick8s migrated to a new location and upgraded to Kubernetes 1.32

Thu, 25 Sep 2025 11:30:00 +0000

[Final Message] Outage finished on Friday 26 September 2025 at 09:35am UTC and post mortem published: https://github.com/jenkins-infra/helpdesk/issues/4813#issuecomment-3339518665.

[Update] Friday 26 September 2025 at 09:35am UTC: ongoing production outage on the cluster (unexpected side effect of the migration): https://github.com/jenkins-infra/helpdesk/issues/4813.

[Update] Migration was finished on Thursday 25 September 2025 at 03:30PM UTC. See https://github.com/jenkins-infra/helpdesk/issues/4617#issuecomment-3334826339.

[Initial Message] Thursday 25 September 2025, starting at 11:30AM UTC, we’ll perform a migration + upgrade of the AKS(Azure) cluster publick8s to a new location and to Kubernetes 1.32.

The expected impacts are:

Restart of the following services, with a downtime between 2 and 15 min (depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-in/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) on:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

More details in https://github.com/jenkins-infra/helpdesk/issues/4617.

[Resolved] jdk21 upgrade for runtime controllers and agents

Thu, 22 May 2025 13:00:00 +0000

[Final Message] migration went well, no impact on production. few fixes to apply with https://github.com/jenkins-infra/jenkins-infra/pull/4143

[Initial Message] Thursday May 22 2025, starting at ~~17:00~~ 13:00 UTC, all our controllers and there agents will be upgraded to use JDK21 as runtime version instead of JDK17.

This operation expects 1 or 2 restarts per controller https://github.com/jenkins-infra/helpdesk/issues/4121#issuecomment-2890355054.

[Resolved] [publick8s cluster] Azure publick8s Upgrade to Kubernetes 1.31.6

Thu, 24 Apr 2025 07:30:00 +0000

[Final Message] upgrade went well, no impact on production.

[Initial Message] Thursday 24 April 2025, at 07:30PM UTC, we’ll upgrade the AKS(Azure) cluster publick8s to Kubernetes 1.31.6. It should take between 60 and 120 min.

The expected impacts are:

⚠️ Last Kubernetes upgrade (to 1.30) we saw an outage on update-center (updates.jenkins.io) for a few seconds !

Restart of the following services, with a downtime between 2 and 15 min (depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-in/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) on:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

More details in https://github.com/jenkins-infra/helpdesk/issues/4546#issuecomment-2674545933.

[Resolved] [publick8s cluster] Azure publick8s Upgrade to Kubernetes 1.30.7

Thu, 20 Feb 2025 13:30:00 +0000

[Final Message] upgrade went well but we saw an outage on update center for about 2 to 3 mn we will analyse further the logs to try to understand were it’s coming from

[Initial Message] Thurday 20 February 2025, at 01:30PM UTC, we’ll upgrade the AKS(Azure) cluster publick8s to Kubernetes 1.30.7. It should take between 60 and 120 min.

The expected impacts are:

Restart of the following services, with a downtime between 2 and 15 min (depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-in/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) on:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

More details in https://github.com/jenkins-infra/helpdesk/issues/4258#issuecomment-2663553626.

[Resolved] AKS cluster upgrade to 1.29.10 (patch)

Wed, 18 Dec 2024 09:45:00 +0000

We’re performing AKS cluster upgrade from 1.29.7 to 1.29.10 the 18 December 2024, starting at 09:45 UTC.

The expected impacts are:

Restart of the following services, with an outage (between 2 to 15 min: depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-In/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

[Resolved] Azure VMs outage

Mon, 09 Sep 2024 15:00:00 +0000

[Final message]

Azure did fix the problem in two steps:

Additional capacity added in US East 2 the 12 September
Software fix the 13 September

[Update on 2024-09-10]

The problem was gone from 2024-09-09 around 11:00pm UTC but it’s back since 2024-09-10 around 02:00pm UTC
Our Azure Service Health Dashboard mentions 2 mixed incidents, one should be fixed the 12!

[initial message] Since Monday 9 September 2024 at 03:00pm UTC, we see an increased rate of failure to spin up Azure VM agents on ci.jenkins.io

Details in https://github.com/jenkins-infra/helpdesk/issues/4287.

[Resolved] Update Azure AKS cluster `publick8s` to 1.29

Fri, 23 Aug 2024 09:00:00 +0000

Friday 23 August at 07:30am UTC, the AKS cluster publick8s will be upgraded to Kubernetes 1.29.7. It should take between 30 and 60 min.

The expected impacts are:

Restart of the following services, with a downtime between 2 and 15 min (depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-in/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) on:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

More details in https://github.com/jenkins-infra/helpdesk/issues/4161.

[Resolved] ldap.jio and repo.azure.jio workloads migration to arm64 (publick8s cluster)

Fri, 05 Jul 2024 14:30:00 +0000

We’re doing a workload migration to arm64 on the publick8s AKS cluster starting at 14:30 UTC.

The expected impacts are:

Restart of the following services, with an outage (between 2 to 15 min: depends on Azure Storage systems):

ldap.jenkins.io
repo.azure.jenkins.io (ACP)

Temporary disruptions (no Log-In/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

More details on https://github.com/jenkins-infra/helpdesk/issues/3837#issuecomment-2210532769.

[Resolved] Update azure clusters publick8s Kubernetes version to 1.28

Thu, 04 Jul 2024 06:30:00 +0000

[Final Message] The operation finished successfully at 7:15am UTC.

More details on https://github.com/jenkins-infra/helpdesk/issues/4144#issuecomment-2196274746.

[Initial message] We’re doing an publick8s AZ cluster upgrade from 1.27 to 1.28 tomorrow, 4th of July 2024, starting at 6:30 UTC (8h30 paris time, 12h00 IST).

The expected impacts are:

Restart of the following service, with an outage (between 2 to 15 min: depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-In/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) due to public IP metadata changes and falco + Nginx upgrades:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
plugin-health.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

[Resolved] Update azure clusters publick8s Kubernetes version to 1.27

Tue, 19 Mar 2024 08:30:00 +0000

[Final Message] The operation finished successfully at 9:35am UTC.

More details on https://github.com/jenkins-infra/helpdesk/issues/3948#issuecomment-1991181964.

[Initial message] We’re doing an publick8s AZ cluster upgrade from 1.26 to 1.27 today 19th of Mars 2024, starting at 8:30 UTC (9h30 paris time).

The expected impacts are:

Restart of the following service, with an outage (between 2 to 15 min: depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-In/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) due to public IP metadata changes and falco + Nginx upgrades:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

[Resolved] LDAP (ldap.jenkins.io) maintenance

Fri, 19 Jan 2024 08:45:00 +0000

Friday January 19 2024, starting at 08:45am UTC, we’ll proceed to a maintenance on the LDAP service (ldap.jenkins.io) hosted in the publick8s Azure Kubernetes cluster.

Authentication on all of the *.jenkins.io and *.jenkins-ci.orf services will be unavailable during the operation. We expect a disruption between 5 and 30 min.

The goal is to update the LDAP image to a new version to use a new deployment system.

More details in https://github.com/jenkins-infra/helpdesk/issues/3837#issuecomment-1898378430.

[Resolved] Operation on Jenkins Azure VM Controllers

Wed, 22 Nov 2023 14:30:00 +0000

Wednesday November 22 2023, starting at 14:30 UTC, an operation on the Jenkins Azure VM controllers will happen.

No expected impact except delays in the Azure VM and ACI container agent allocation.

[Resolved] Update publick8s AKS cluster to Kubernetes 1.26

Thu, 09 Nov 2023 08:30:00 +0000

[Final Message] The operation finished successfully at 11:00am UTC. The monitoring showed an outage of 6 minutes on the LDAP and 2 minutes on the incremental publisher.

More details on https://github.com/jenkins-infra/helpdesk/issues/3683#issuecomment-1803566680.

[Initial Message] The AKS cluster publick8s will be upgraded from Kubernetes 1.25 to 1.26 the Thursday 09th of November 2023, starting at 08:30am UTC.

The expected impacts are:

Restart of the following service, with an outage (between 2 to 15 min: depends on Azure Storage systems):

ldap.jenkins.io
weekly.ci.jenkins.io
incrementals.jenkins.io

Temporary disruptions (no Log-In/log-out possible during the LDAP restart) on:

accounts.jenkins.io
ci.jenkins.io, trusted.ci.jenkins.io, infra.ci.jenkins.io, weekly.ci.jenkins.io, release.ci.jenkins.io, cert.ci.jenkins.io
issues.jenkins.io (JIRA)

Temporary disruptions (slow requests or outage of a few minutes) due to public IP metadata changes and falco + Nginx upgrades:

get.jenkins.io (mirror redirector)
javadoc.jenkins.io
jenkins-wiki-exporter.jenkins.io
plugins.jenkins.io
repo.azure.jenkins.io
reports.jenkins.io
stories.jenkins.io
uplink.jenkins.io
www.jenkins.io

[Resolved] Maintenance on ci.jenkins.io, trusted.ci.jenkins.io and cert.ci.jenkins.io

Mon, 28 Aug 2023 12:00:00 +0000

[Final message]

The operation is now finished. It took 3h30 instead of the initial 2h due to unexpected disk slowness but was completed successfully.

[Initial message]

Monday August 28 2023, starting at 12:00 UTC, the Jenkins controller cert.ci.jenkins.io will be put down for maintenance (migration to a new VM) as per https://github.com/jenkins-infra/helpdesk/issues/3688. Only the Jenkins Security team is impacted.

[Resolved] Maintenance on ci.jenkins.io, trusted.ci.jenkins.io and cert.ci.jenkins.io

Mon, 21 Aug 2023 13:45:00 +0000

[Final Message] Operation completed successfully:

Took around 30 minutes as expected.
A few minor unexpected behaviors are tracked in https://github.com/jenkins-infra/helpdesk/issues/3072 (fixed the 22 of October 2023).

[Initial Message] Monday August 21 2023, starting at 13:45 UTC, the Jenkins controllers ci.jenkins.io, trusted.ci.jenkins.io and cert.ci.jenkins.io will be upgraded to use JDK17 instead of JDK11.

This operation expects 1 or 2 restarts per controller but then should be harmless as we worked on the agent images as per https://github.com/jenkins-infra/helpdesk/issues/3072 and already upgraded 3 other controllers.