The modular pipeline library (MPL) we created is a highly-flexible shared library for a Jenkins Pipeline that enables easy sharing of best practices across the entire company. It has a clear modular structure, an advanced testing framework, multi-level nesting, a pipeline configuration system, improved error handling, and many other useful components.

We will take a look under the hood and explain how our solution works in several parts:

So now let’s jump right into an explanation of the crucial features we used to build our solution.

Jenkins, our main automation platform, recently received some updates toJenkins Pipeline. These updates allow us to create one Jenkinsfile that describes the entire pipeline, and the steps that need to be executed with a series of self-explanatory scripts. This increases the visibility of CI/CD automation processes for end users, and improves supportability by DevOps teams.

However, there’s a large issue with Pipeline: it’s hard to support multiple Jenkinsfiles (and therefore multiple projects) with unique pipelines. We need to store the common logic somewhere, which is where Jenkins Shared Libraries come in. They are included in the Jenkinsfile, and allow the use of prepared interfaces to simplify automation and store common pieces.

While shared libraries allow you to store logic and manipulate Jenkins, they don’t provide a good way to utilize all the common information. Therefore, the MPL optimizes the pipeline and shared libraries by allowing users to create easy-to-follow descriptions for processes, which are then stored for later use by other teams.

With the MPL, we are now able to collaborate and share our DevOps practices across teams, easily adopt existing pipelines for specific projects, and debug and test features before we actually integrate them into the library. Each team can create a nested library, add a number of pipelines and modules inside, and use it with pipeline automation to create great visibility of the processes for the end user. The MPL can also work on any project to prepare a Jenkinsfile, and manage it as flexibly as the project team wants.

At its core, the MPL provides a simple way to:

There are a lot of other features in the MPL, but it’s essentially a platform to solve general DevOps collaboration issues. To simplify development and manual testing, the MPL provides modules overriding and an inheritance model, allowing users to test specific fixes in the project without affecting anything else. In Jenkins, a module is a file with scripted steps and logic to reach a simple goal (build an artifact, run tests, create an image, etc.). These modules are combined in the pipeline stages, and are easily readable for anyone who knows the Jenkins Pipeline syntax.

The MPL allows users to use the core features of the library (structure, modules, pipelines) and create nested libraries for specific DevOps team needs. A DevOps team can prepare complete pipelines with any custom logic and use it for their projects. They can also override and inherit the core MPL modules in a number of ways, or prepare custom modules which are easy to share with other teams. Check out the infographic below to see how modules fit in:

You can also specify certain pipeline required poststeps in a module. For example, a dynamic deployment module creates the test environment, which needs to be destroyed when the pipeline ends. To take a closer look at the MPL calling process, check out the infographic below:

This infographic shows how calls are executed in the MPL. First, you need a job on your Jenkins, which will call a Jenkinsfile (for example, when the source code is changed), after which the Jenkinsfile will call a pipeline. The pipeline could be described on the MPL side, in the pipeline script in the job, in the nested library, or in the project Jenkinsfile. Finally, the stages of the pipeline will call the modules, and these modules will use features, which could be groovy logic, pipeline steps, or steps in the shared libraries.

Now that we’ve done an overview of the solution, let’s take a look at a simple pipeline execution to see how the MPL works in action.

For example, let’s say you have a common Java Maven project. You are creating a Jenkinsfile in the repo, and want to use the default pipeline prepared by your DevOps team. The MPL already has a simple pipeline: the core MPLPipeline. It’s a really simple pipeline, but it’s a good start for anyone who wants to try the MPL. Let’s look at a simple Jenkinsfile:

This Jenkinsfile contains a single line to load the MPL, and another line to run the pipeline. Most of the shared libraries implement an interface like this, calling one step and providing some parameters. MPLPipeline is merely a custom Pipeline step, as it lies in the vars directory, and its structure is very simple, following these steps:

Stages of the main MPL usually have just one step, the MPLModule. This step contains the core functionality of the MPL: executing the modules which contain the pipeline logic. You can find default modules in the MPL repository, which are placed in resources/com/griddynamics/devops/mpl/modules. Some of the folders include: Checkout, Build, Deploy, and Test, and in each of them we can find Groovy files with the actual logic for the stages. This infographic is a good example of a simplified MPL repository structure:

When the Checkout stage starts, MPLModule loads the module by name (by default a stage name), and runs the Checkout/Checkout.groovy logic:

If the configuration contains the git.url option, it will load a Git Checkout module; otherwise, it will run the Default Checkout module. All the called modules use the same configuration as the parent module, which is why CFG was passed to the MPLModule call. In this case, we have no specific configuration, so it will run theCheckout/DefaultCheckout.groovy logic. The space in the name is a separator to place the module into a specific folder.

In the Default Checkout module, there is just one line with checkout scm execution, which clones the repository specified in the Jenkins job. That’s all the Checkout stage does, as the MPL functionality is excessive for such a small stage, and we only need to talk about it here to show how the MPL works in modules.

The same process applies to the Build stage, as the pipeline runs theMaven Build module:

This stage is a little bit more complicated, but the action is simple: we take the tool with the default name Maven 3, and use it to run mvn clean install. The modules are scripted pipelines, so you can do the same steps usually available in the Jenkins Pipeline. The files don’t need any specific and complicated syntax, just a plain file with steps and CFG as a predefined variable with a stage configuration. The MPL modules inherited the sandbox from the parent, so your scripts will be safe and survive the Jenkins restart, just like a plain Jenkins pipeline.

In the Deploy folder, we find the sample structure of the Openshift Deploy module. Its main purpose here is to show how to use poststep definitions in the modules:

First, we define the always poststep. It is stored in the MPLManager, and is called when poststeps are executed. We can call MPLPostStep with always as many times as we want: all the poststeps will be stored and executed in FILO order. Therefore, we can store poststep logic for actions that need to be done, and then undone, in the same module, such as the decommission of the dynamic environment. This ensures that the actions will be executed when the pipeline is complete.

After the deploy stage, the pipeline executes the Test stage, but nothing too interesting happens there. However, there is an aspect of testing which is very important, and that’s the testing framework of the MPL itself.

The testing framework of the MPL is based on theJenkinsPipelineUnit from LesFurets, with the one small difference being its ability to test the MPL modules. Testing the whole pipeline doesn’t work, as pipelines can be really complicated, and writing tests for such monsters is a Sisyphean task. It is much easier to test a black box with a small amount of steps, ensuring that this particular task is working correctly.

In the MPL, you can find Build module testing examples: all the tests are stored in thetest/groovy/com/griddynamics/devops/mpl/modules directory, and you can find theBuild/BuildTest.groovy file with a number of test cases there. Tests are executed during the MPL build process, allowing users to see traces like this:

The test runs the MPLModule with custom configuration and mocked steps to check that, during execution, the tool was changed to Maven 2 according to the provided configuration. We cover all test cases with such tests, ensuring that the modules are working as expected, and that the pipeline will work properly. You can test the whole pipeline if you want, but testing by modules is just an additional way to simplify the testing process.

Now that we’ve looked at how to test the MPL modules, it’s time to look at one of the key features of the MPL, which is nested libraries.

When working with a large company, supporting one big library makes no sense. Each department requires multiple configuration options and tuning for a somewhat standard pipeline, which creates extra work. The MPL solves such problems by introducing nested libraries. This infographic displays how a nested library compares to just using the main library:

A nested library is the same as a shared library that imports the MPL and uses its functionality, modules, and pipelines. Also, it allows the separation of some team-related logic from the company common logic. Here is the structure of the MPL with nested libraries:

You can import the MPL in the overridden pipeline, specify the path of some additional modules, override module logic, and use Jenkins power moves: there are no limitations. When another team needs your unique module, you can just create a change request to the basic company MPL repo, and share your functional module with the others.

With nested libraries, it’s possible to debug and modify MPL-provided steps (MPLModule for example) and pipelines. This is because nested libraries can override low-level functionalities of the MPL or the Jenkins Pipeline. There are no limitations to what you can or can’t change, as these overrides only affect your own pipeline. This enables experimentation to be done, and then discussed with other teams to see if it will work in other nested libraries as well.

There are also no limits to the number of nesting levels created, but we recommend using just two (MPL and nested), because additional levels make configuration and testing of the nested libraries on lower levels very complicated.

Further into the nested libraries or project-side modules, it’s possible to store a module with the same name as one in the upper-level library. This is a good way to override the logic - you can just replace Build/Build.groovy with your own - as the functional module will be executed instead of the upper-level module. For example, this infographic shows module overriding:

Even better, one of the strengths of the MPL is that you still can use the upper-level module! The MPL has mechanisms to prevent loops, so the same module can’t be executed in the same executing branch again. However, you can easily call the original module a name from another module to use the upper-level logic.

The Petclinic-Selenium example above uses the default MPLPipeline (you can find it on the MPL Wiki-page), and contains project-side modules in a .jenkins directory. These modules will be called before the library modules. For example, the Checkout module is not placed on the project side, so it will be called from the MPL, but the Build module exists in a .jenkins directory on the project side, and it will be called:

As you can see, the Build module from the project registers the poststep, calls the original Build module from the MPL, and then calls the additionalDocker Build module. The following stages of the pipeline are more complicated, but all module overriding essentially works like this. Some projects can be tricky, and need some small tunings for the existing modules. However, you can easily implement those changes on the project level, and think about how to move the functionality to the nested library or MPL later.

Many DevOps teams and companies work with bloated, restrictive, and buggy CI/CD automation platforms. These increase the learning curve for users, cause teams to work slower, and raise production costs. DevOps teams frequently run into similar issues on different projects, but a lack of collaboration means that they have to be individually fixed each time.

However, with the MPL, DevOps teams have a shared, simple, and flexible CI/CD platform to improve user support, collaboration, and overall project source code to the production process. By utilizing the MPL, your company can find an automation consensus, reach cross-company collaboration goals, and reuse the best practices from a large community, all with open source tools. If you’re interested in building an MPL, please contact us to learn more!

Example & demo videos:

On Friday, February 1, we’ll start off with a couple workshops:

Aside from the workshops, from 9am to 5pm a bunch of people will be working out of Hilton Brussels Grand Place, hanging out as travelers come in. It’ll be a casual, unstructured day. Sign up on this meetup page to be notified what meeting room we’re in.

After the office hours and workshops, we’ll have a happy hour Friday evening before FOSDEM at Cafe Le Roy d’Espagne. See the meetup page for details.

Finally, a Jenkins Hackfest will be held the day after FOSDEM 2019 on Monday (February 4). Those who would like to join us for the hackfest should register for the meetup.

Meals, snacks, and beverages will be provided for the hackfest. Come join us, and let’s write some code!

Questions? feel free to contactAlyssa Tong orBaptiste Mathus or join us on theadvocacy-and-outreach gitter channel.

The current installer has a few issues that the Platform SIG wanted to fix in a new install experience for users.

With the experimental Jenkins Windows Installer, most of these issues have been resolved!

Below are screenshots of the new installer sequence:

The new installer is under review by the members of the Platform SIG, but we need people to test the installer and give feedback. If you are interested in testing the new installer, please join the Platform SIG gitter room for more information.

There are still some things that are being researched and implemented in the new installer (e.g., keeping port and other selections when doing an upgrade), but it is getting close to release.

In addition to updates to the MSI based Windows installer, the Platform SIG is working on taking over the Chocolatey Jenkins package and releasing a version for each update.

One of the most common steps defined in a basic pipeline job is the Deploy step. The deployment stage encompasses everything from publishing build artifacts to pushing code into pre-production and production environments. This deployment stage usually involves both development and operations teams logging onto various remote nodes to run commands and/or scripts to deploy code and configuration. While there are a couple of existing ssh plugins for Jenkins, they currently don’t support the functionality such as logging into nodes for pipelines. Thus, there was a need for a plugin that supports these steps.

Below is a simple demonstration on how to use above steps. More documentation can be found on GitHub.

At Cerner, we always strive to have simple configuration files for CI/CD pipelines whenever possible. With that in mind, my team built a wrapper on top of these steps from this plugin. After some design and analysis, we came up with the following YAML structure to run commands across various remote groups:

The above example runs commands from c_group_1 on remote nodes within r_group_1 in parallel before it moves on to the next group using sshUserAcct (from the Jenkins Credentials store) to logon to nodes.

We have created a shared pipeline library that contains a sshDeploy step to support the above mentioned YAML syntax. Below is the code snippet for the sshDeploy step from the library. The full version can be found here on Github.

By using the step (as described in the snippet above) from this shared pipeline library, a Jenkinsfile can be reduced to:

An example execution of the above pipeline code in Blue Ocean looks like this:

Steps from the SSH Steps Plugin are deliberately generic enough that they can be used for various other use-cases as well, not just for deploying code. Using SSH Steps has significantly reduced the time we spend on deployments and has given us the possibility of easily scaling our deployment workflows to various environments.

Help us make this plugin better by contributing. Whether it is adding or suggesting a new feature, bug fixes, or simply improving documentation, contributions are always welcome.

One very common example of that is the role of credentials masking in Jenkins, typically involving a pipeline snippet that looks like this:

Credentials that are in scope are made available to the pipeline without limitation. To prevent accidental exposure in the build log, credentials are masked from regular output, so an invocation of env (Linux) or set (Windows), or programs printing their environment or parameters would not reveal them in the build log to users who would not otherwise have access to the credentials.

The misconception here is that Jenkins will prevent other, perhaps deliberate ways to reveal the password. Some examples:

Both of these snippets circumvent credentials masking in the build log, and show that people with control over the build script can use credentials in ways not necessarily intended or approved by admins.

Obviously these are just the most straightforward examples illustrating the problem. Others could involve the proc file system, sending it to an HTTP server in response to a 401 authentication challenge, embedding it in the (otherwise legitimate) build result, etc.

It would be great if Jenkins could allow the flexible use of credentials with no risk of exposing them through straightforward build script modifications, but realistically, it is impossible for Jenkins to police use of the credential by a build script without the support of a very specific environment setup (e.g. restrictive network configuration).

It should also be noted that credentials aren’t just at risk from users able to control the pipeline, typically by editing the Jenkinsfile. Actual build scripts invoked by pipelines, either shell scripts as in the example above, or more standard build tools such as Maven (controlled by pom.xml) are just as much of a risk if they are run inside a withCredentials block, or executing on the same agent as another block that passed such credentials.

Disclosure of secrets can also happen inadvertently: Jenkins will prevent exact matches of the password or other secret to appear in the log file. Consider that the secret may contain shell metacharacters that bash +x would escape by adding a \ before those characters. The sequence of characters to be printed is no longer identical to the secret, so would not be masked.

Credentials can be defined in different scopes: Credentials defined on the root Jenkins store (the default) will be available to all jobs on the instance. The only exception are credentials with System scope, intended for the global configuration only, for example, to connect to agents. Credentials defined in a folder are only available within that folder (transitively, i.e. also in folders inside this folder).

This allows defining sensitive credentials, such as deployment credentials, on specific folders whose contents only users trusted with those credentials are allowed to configure: Directly in Jenkins using Matrix Authorization Plugin and by limiting write access to repositories defining pipelines as code.

Pipelines inside this folder can use the (e.g. deployment) credentials without limitation, while they’re inaccessible to pipelines outside the folder. Those would need to use the build step or similar approaches to invoke the pipelines inside the folder to deploy their output.

While the previous section outlines a solution to the problem of restricting access to credentials, care needs to be taken so that credentials are not captured anyway. For example, a deployment pipeline that allows its users to define where to deploy to as a build parameter might still be used to send credentials to a maliciously set up host to capture them.A blog post explaining the design of some Jenkins project infrastructure discusses some of these concerns around trust.

It should also be noted that credential domains are a UI hint only — defining a credential to only be valid for github.com does not actually prevent its use elsewhere.

Voice-first technology is revolutionizing how we interact with technology because the interaction is simple, frictionless, and time-saving. For me, voice is an easier way to control Jenkins and retrieve results about my deployments without having to touch a keyboard. In this use case, voice is another access point for data and is a way to further automate the process of building, testing, and deploying a Java project to the cloud, improving efficiency.

If you’re working with DevOps, you understand the need for Continuous Integration and Continuous Delivery (CI/CD) to automate the software delivery pipeline in a reproducible way. CI/CD is the practice of continuously building, testing, and deploying code once it’s committed to version control. DevOps and CI/CD provides software engineering teams with confidence in the code being pushed to production and shorter development lifecycles, which in the end produces happier users, clients, and customers.

DevOps Pal is a private Alexa for Business skill that is used to kick off a Jenkins pipeline job. Alexa for Business was the perfect way for me to distribute DevOps Pal since I have the ability to enable the skill on an organization-by-organization basis, which gives me complete control over who has access. Once DevOps Pal invokes the job, the pipeline status displays in real-time via the Blue Ocean Pipeline Run Details View Page.

I used several components and tools to create DevOps Pal. Let’s review the architecture in detail.

The flow begins by saying, "Alexa, open DevOps Pal and deploy my code", to the Echo device.

The Echo device listens for the wake word (e.g. Alexa, Echo, Computer, or Amazon), which employs deep learning technology running on the device to recognize the wake word the user has chosen. Once the wake word is detected, what I say is recorded and sent to the Alexa Voice Service (AVS), which uses speech to text and natural language understanding (NLU) to identify my intent. My intent is sent to DevOps Pal; the skill acts accordingly by kicking off the Jenkins job and sending a response back using text-to-speech synthesis (TTS), which makes the response natural sounding.

Let’s explore each component in more detail:

The Voice User Interface (VUI) describes the overall conversational flow and is setup via the Alexa Developer Console.

A few important components of the VUI are the Invocation Name (how users launch your skill) and the Intents (phrases a user says to "talk to" or interact with your skill).

Specifically, the "DeployCodeIntent" is invoked when a user says one of several phrases (e.g. run jenkins pipeline, run jenkins job, deploy the code, deploy code, or deploy ) or a variation of the phrase like, "deploy my code".

The endpoint is the destination where the skill requests are sent for fulfillment. In this case, the backend logic is an AWS Lambda authored in Python. The business logic in the Python Lambda uses the Jenkins remote access API to trigger the job remotely. The format of the URL to trigger the job is jenkins_url/job/job_name/build. The API call uses BASIC authentication and a Jenkins Crumb passed in the HTTP request header for CSRF protection. Alternatively, since Jenkins 2.96, you can use an API token instead of a Jenkins Crumb and password to authenticate your API call.

The Jenkins job, 'alexa-cicd', is the job invoked from DevOps Pal. Although, the Jenkins Classic User Interface (UI) is functional, I prefer the Blue Ocean interface because it rethinks the user experience of Jenkins by making it visually intuitive. Blue Ocean is easily enabled via a plugin and leaves the option to continue using the Jenkins Classic UI should you so choose.

After Alexa kicks off the 'alexa-cicd' job, I navigate to the Pipeline Run Details View Page, which allows me to watch the job status in realtime. This job has four stages: Initialize, Build, Test, and Deploy. The final stage, Deploy, uses the AWS Command Line Interface (CLI) on the Jenkins server to copy the artifact to Amazon Simple Storage Service (S3) and create a new Elastic Beanstalk application version based on the artifact located on S3.

The ability to deploy code with voice is just the beginning. There are several cool features that can easily be added:

I hope you’ve enjoyed learning more about the architecture of DevOps Pal and deploying code to the cloud using Jenkins and voice. For more detailed steps, I’ve collaborated with Cloud Academy to author a course, AWS Alexa for CI/CD on the subject.

See the Information for students page for full application guidelines.

We encourage interested students to reach out to the Jenkins community early and to start exploring project ideas. All project ideas have chats and mailing lists referenced on their pages. We will be also organizing office hours for students, and you can use these meetings to meet org admins and mentors and to ask questions. Also, join our Gitter channel and themailing list to receive information about such incoming events in the project.

The application period starts on March 25th, but you can prepare now! Use the time before the application period to discuss and improve your project proposals. We also recommend that you become familiar with Jenkins and start exploring your proposal areas. Project ideas include quick-start guidelines and reference newbie-friendly issues which may help with initial study. If you do not see anything interesting, you can propose your own project idea or check out ideas proposed by other organizations participating in GSoC.

It’s not! We are looking for more project ideas and for Jenkins contributors/users who are passionate about Jenkins and want to mentor students. No hardcore experience required, mentors can study the project internals together with students and technical advisors. We are especially interested in ideas beyond the Java stack, and in ideas focusing new technologies and areas (e.g. Kubernetes, IoT, Python, Go, whatever).

You can either propose a new project idea or join an existing one. See the Call for Mentors post and Information for mentors for details. If you want to propose a new project, please do so by March 11th so that students have time to explore them and to prepare their proposals.

This year mentorship does NOT require strong expertise in Jenkins development. The objective is to guide students and to get involved into the Jenkins community. GSoC org admins will help to find advisers if special expertise is required.

See the GSoC Timeline for more info. In the Jenkins project we will also organize special events during and after GSoC (e.g. at Jenkins world).

We’d like to take a moment to thank everyone involved in these tasks: code contributors, issue reporters, testers, event planners and attendees and all those in the community who have generously lent their time and support to this effort. Thank you all!

Here are some of the contributors who helped with this task (alphabetical order):

Alex Earl, Alyssa Tong, Ashton Treadway, Baptiste Mathus, Carlos Sanchez, Daniel Beck, David Aldrich, Denis Digtyar, Devin Nusbaum, Emeric Vernat, Evaristo Gutierrez, Gavin Mogan, Gianpaolo Macario, Isabel Vilacides, James Howe, Jeff Pearce, Jeff Thompson, Jenn Briden, Jesse Glick, Jonah Graham, Kevin Earls, Ksenia Nenasheva, Kohsuke Kawaguchi, Liam Newman, Mandy Chung, Mark Waite, Nicolas De Loof, Oleg Nenashev, Oliver Gondža, Olivier Lamy, Olivier Vernin, Parker Ennis, Paul Sandoz, Ramón León, Sam Van Oort, Tobias Getrost, Tracy Miranda, Ulli Hafner, Vincent Latombe, Wadeck Follonier

(We are deeply sorry if we missed anyone in this list.)

In order to keep it simple, here is how you can start Jenkins on Java 11 using the Docker image. You can select a Java 11 based image by suffixing the tag of the image with -jdk11. If you are upgrading an existing instance please read the Upgrading Jenkins Java version from 8 to 11 page before upgrading.

So you can run Jenkins on Java 11 with:

However, and as always, you can still start Jenkins with other methods. Please see the more detailed documentation at Running Jenkins on Java 11.

For developers involved in Jenkins development, you can find details on developing and testing Jenkins to run on Java 11 on the Java 11 Developer Guidelines.

This resource regroups the modifications which might need to be done in order to validate the compatibility of plugins for Java 11.

Even though this is a big achievement, we still have work to do.

Our first priority is adding Java 11 support to JenkinsFile Runner project. From there, we will move on to port Java 11 support to the Jenkins X project and the Evergreen project.

So, even if this is a big deal to us, this is not the end of the story. It is a major step that will benefit users, developers, and members of the Jenkins community.

When looking at the “Participate and contribute” page, I noticed a couple of things in that page that I could help improve. And I was actually planning to pick one of those as the first example of a contribution for this post. But when I was reading the contributing guidelines of the repository, I found an even easier contribution I could make, which I thought would be a great example to illustrate how simple it could be to start contributing. So I decided to go ahead with it.

Jenkins uses the Stapler web framework for HTTP request handling. Stapler’s basic premise is that it uses reflective access to code elements matching its naming conventions. For example, any public method whose name starts with get, and that has a String, int, long, or no argument can be invoked this way on objects that are reachable through these means. As these naming conventions closely match common code patterns in Java, accessing crafted URLs could invoke methods never intended to be invoked this way.

A simple example of that is a URL every Jenkins user would be familiar with: /job/jobname. This ends up invoking a method called #getJob(String), with the argument being "jobname", on the root application object, and having it handle the rest of the URL, if any. Of course, this is a URL intended to be accessed this way. How about invoking Object#getClass(), followed by Class#getClassLoader(), by accessing the URL /class/classLoader? While this particular chain would not result in a useful response, this doesn’t change that the methods were invoked. We identified a number of URLs that could be abused to access otherwise inaccessible jobs, or even invoke internal methods in the web application server to invalidate all sessions. The security advisory provides an overview of the issues we’d identified by then.

To solve this problem inherent in the Stapler framework’s design, we defined rules that restrict invocation beyond what would be allowed by Stapler. For example, the declared return type of getters now needed to be one defined in Jenkins core or a Jenkins plugin and have either clearly Stapler-related methods (with Stapler annotations, parameter types, etc.) or Stapler-related resource files associated with it. Otherwise, the type wouldn’t be aware of Stapler, and couldn’t produce a meaningful response anyway.

This meant that getters just declaring Object (or List, Map, etc.) would no longer be allowed by default. It was clear to the developers working on this problem that we needed the ability to be able to override the default rules for specific getters. But allowing plugin developers to adapt their plugins after we published the fix wasn’t going to cut it; Jenkins needed to ship with a comprehensive default whitelist for methods known to not conform to the new rules, so that updating would not result in problems for users.

While there is tooling like Plugin Compatibility Tester and Acceptance Test Harness, many Jenkins plugins do not have comprehensive tests of their UI — the Jenkins UI is fairly stable after all. We did not expect to have sufficient test coverage to deliver a change like this with confidence. The only way we would be able to build such a comprehensive whitelist would be to add telemetry to Jenkins.

While Jenkins instances periodically report usage statistics to the Jenkins project, the information included is very bare bones and mostly useful to know the number of installations, the popularity of plugins, and the general size of Jenkins instances through number and types of jobs and agents. We also didn’t want to just collect data without a clear goal, so we set ourselves some limitations — collect as little data as possible, no personally identifiable information, have a specific purpose for each kind of information we would collect, and define an end date for the collection in advance. We defined all of this in JEP-214, created the Uplink service that would receive submissions, and added the basic client framework to Jenkins. The implementation is fairly basic — we just submit an arbitrary JSON object with some added metadata to a service. This system would inform tweaks to a security fix we were anxious to get out, after all.

Starting in mid October for weekly releases, and early November for LTS, tens of thousands of Jenkins instances would submit Stapler request dispatch telemetry daily, and we would keep identifying code incompatible with the new rules and amending the fix. Ultimately, the whitelist would include a few dozen entries, preventing serious regressions in popular plugins like Credentials Plugin, JUnit Plugin, or the Pipeline plugins suite, down to Google Health Check Plugin, a plugin with just 80 installations when we published the fix.

Learning what requests would result in problems also allowed us to write better developer documentation — we already knew what code patterns would break, and how popular each of them was in the plugin ecosystem.

I wrote above:

While this was true for the fix during most of development, it isn’t how the fix that we published actually works. About a month before the intended release date, internal design/code review feedback criticized the complicated and time-consuming implementation that at the time required scanning the class path of Jenkins and all plugins and looking for related resources, and suggested a different approach.

So we tried to require that the declared type or any of its ancestors be annotated with the new annotation @StaplerAccessibleType, annotated a bunch of types in Jenkins itself (ModelObject being the obvious first choice), and ran our scripts that check to see whether Stapler would be allowed to dispatch methods identified in telemetry. We’d long since automated the daily update of dispatch telemetry processing, so it was a simple matter of changing which Jenkins build we were working with.

After a few iterations of adding the annotation to more classes, the results were very positive: Very few additional types needed whitelisting, while many more were no longer (unnecessarily) allowed to be dispatched to. This experiment, late during development, ended up being essentially the fix we delivered. We didn’t need to perform costly scanning of the class path on startup — we didn’t need to scan the class path at all — , and the rules governing request dispatch in Stapler, while different from before, are still pretty easy to understand and independent of how components are packaged.

As usual when delivering a fix we expect could result in regressions in plugins, we created a wiki page that users could report problems on. Right now, there’s one entry on that wiki page. It is one we were aware of well before release, decided against whitelisting it, and the affected, undocumented feature in Git Plugin ended up being removed. The situation in our issue tracker is only slightly worse, with two apparently minor issues having been reported in Jira.

Without telemetry, delivering a fix like this one would have been difficult to begin with. Tinkering with the implementation just a few weeks before release and having any confidence in the result? Not causing any significant regressions? I think this would simply be impossible.

Previously, language localization files were distributed in core and in each plugin. For this proposal, each language has a single localization plugin, such as Chinese Localization plugin. Finally, Localization Support Plugin andChinese Localization plugin are able to support all types of localization resource files. From the plugins website, you can see that there are already 13 000 installations. We removed all Chinese localization files at the PR-4008.

I really appreciate Daniel Beck for helping me to add localization support,Liam Newman helping me to review JEP-216, and many other community members.

We believe that this SIG could help to improve Jenkins experience for Chinese users and gather more contributors from China. This SIG is responsible for maintaining the Chinese Jenkins website, promoting the Jenkins community in China in the social media with WeChat account. We publish translated blog articles, Jenkins release notes, JAM or other events at the WeChat account. For now, there are 1800 followers that can read our news from the last half a year.

Especially, I want to say thanks to Wang Donghui, Zhai Zhijun, and other contributors. They did a lot of contributions. I wish I could see more and more folks join us.