Brownout on Update Center (updates.jenkins.io): 26 and 27 September 2024

Note: this is a follow up of the 11 and 12 September 24 hours brownout.

The service https://updates.jenkins.io will switch its implementation to a new system during 1 day:

All Jenkins users are impacted but should not see any functional change as we removed the HTTP to HTTPS forced redirection which caused disruptions during the previous brownout.

⚠️ Please, check that your organization respects the advertised DNS TTL or you might be stuck in the brownout longer than expected.

Under the hood, any HTTP request made to this service will be redirected to a mirror close to user locations during these brownouts.

Jenkins Update Center is a web server at the core of the Jenkins public infrastructure which distributes the plugins, tool installers, and versions index to all Jenkins servers across the world.

From the installation wizard to regular plugin updates, if you run Jenkins, then you use this service under the hood.

Today, it serves around 50 Tb of data (outbound bandwidth) each month from a single virtual machine on AWS, which costs around $6,000 per month.

In order to sustain this service and improve it, the Jenkins infrastructure team has worked relentlessly during the past years to have a new sustainable implementation for this service.

The new Update Center implementation features a highly available system that redirects user requests to a download mirror close to their location. Additional information is available in the GitHub issue.

Our functional tests and performance tests are meeting our expectations as described in "Why this brownout" section of previous blog post.

However the previous brownout taught us that enforcing HTTP to HTTPS redirections caused disruptions for users . We decided to support full HTTP temporarily to avoid bad surprises: users are (and will be) encouraged to use the HTTPS URL in the future but let’s do one step at a time!

As such, the fourth brownout serves to verify we don’t have last minute bad surprises.

Both current and new Update Centers are updated at the same time and serve the same index.

Starting 7 weeks ago, the Jenkins infrastructure has been using the new Update Center (link:https://azure.updates.jenkins.io) with a client-side DNS override (updates.jenkins.io hostname points to this new service).

During this brownout, we’ll simply switch the DNS entry updates.jenkins.io to this new service and watch for the logs and error rate. At the end, we’ll switch DNS back to the normal service and then analyze metrics and logs to see how the system behaved.

We are confident the new system will perform as expected

Please refer to the helpdesk ticket for more information.