I have been working on improving the performance of the Role Strategy Plugin as a part of my Google Summer of Code project. Since there was no existing way to measure performance and do benchmarks on Jenkins Plugins, my work for the first phase of the project was to create a framework for running benchmarks in Jenkins plugins with a Jenkins instance available. To make our job a bit easier, we chose Java Microbenchmark Harness for running these benchmarks. This allows us to reliably measure performance of our time-critical functions and will help make Jenkins perform faster for everyone.
The micro-benchmarking framework was recently released in the Jenkins Unit Test Harness 2.50. The blog post below shows how to run benchmarks in your plugins.
Introduction
The framework runs works by starting a temporary Jenkins instance for each fork of the JMH benchmark,
just like JenkinsRule from Jenkins Test Harness. Benchmarks are run directly from your JUnit Tests which allows
you to fail builds on the fly and easily run benchmarks from your IDE, just like unit tests. You can easily
configure your benchmarks by either using your Java methods, or by using Jenkins Configuration-as-Code plugin
and passing the path to your YAML file.
To run benchmarks from your plugins, you need to do the following:
bump up the minimum required Jenkins version to 2.60.3 or above
bump Plugin-POM to a version ≥ 3.46 or manually upgrade to Jenkins Test Harness ≥ 2.51.
Now, to run the benchmarks, you need to have a benchmark runner that contains a @Test so it can run
like a JUnit test. From inside a test method, you can use the OptionsBuilder provided by JMH to
configure your benchmarks. For example:
publicclassBenchmarkRunner {@Testpublicvoid runJmhBenchmarks() throwsException {
ChainedOptionsBuilder options = new OptionsBuilder()
.mode(Mode.AverageTime)
.forks(2)
.result("jmh-report.json");// Automatically detect benchmark classes annotated with @JmhBenchmarknew BenchmarkFinder(getClass()).findBenchmarks(options);new Runner(options.build()).run();
}
}Sample benchmarks
Now, you can write your first benchmark:
Without any special setup
@JmhBenchmarkpublicclassJmhStateBenchmark {publicstaticclassMyStateextends JmhBenchmarkState {
}@Benchmarkpublicvoid benchmark(MyState state) {// benchmark code goes here
state.getJenkins().setSystemMessage("Hello world");
}
}Using Configuration as Code
To use configuration as code, apart from the dependencies above you also need to add the following
to your pom.xml:
<dependency><groupId>io.jenkins</groupId><artifactId>configuration-as-code</artifactId><version>1.21</version><optional>true</optional></dependency><dependency><groupId>io.jenkins</groupId><artifactId>configuration-as-code</artifactId><version>1.21</version><classifier>tests</classifier><scope>test</scope></dependency>Now configuring a benchmark is as simple as providing path to your YAML file and specifying the class containing the benchmark state.
@JmhBenchmarkpublicclassSampleBenchmark {publicstaticclassMyStateextends CascJmhBenchmarkState {@Nonnull@OverrideprotectedString getResourcePath() {return"config.yml";
}@Nonnull@OverrideprotectedClass<?> getEnclosingClass() {return SampleBenchmark.class;
}
}@Benchmarkpublicvoid benchmark(MyState state) {
Jenkins jenkins = state.getJenkins(); // jenkins is configured and ready to be benchmarked.// your benchmark code goes here...
}
}More Samples
As a part of this project, a few benchmarks have been created in the Role Strategy Plugin which show configuring the instances for various situations. You can find themhere.
Running Benchmarks
Running benchmarks from Maven
To easily run benchmarks from Maven, a Maven profile to run the benchmarks has been created
and is available starting Plugin-POM version 3.45. You can then run your benchmarks from the
command line using mvn test -Dbenchmark.
Running benchmarks on ci.jenkins.io
If you have your plugins hosted on ci.jenkins.io, you can easily run benchmarks directly from your Jenkinsfile
by using the runBenchmarks() method after the buildPlugin() step in your which is now available inJenkins Pipeline library.
This function also accepts the path to your generated JMH benchmark reports as an optional
parameter and archives the benchmark results. Running benchmarks in pull request builds allows you to constantly
monitor the performance implications of a given change. For example, the Jenkinsfile from Role Strategy Plugin:
buildPlugin()
runBenchmarks('jmh-report.json')Visualizing benchmark results
Benchmark reports generated (in JSON) can be visualized using the either the JMH Report Plugin or by passing the benchmark reports to the JMH visualizer web service. As an example, here is a visualized report of some benchmarks from the Role Strategy Plugin:
These improvements seen above were obtained through a small pull request to the plugin and shows how even seemingly small changes can bring major performance improvements. Microbenchmarks help to find these hot-spots and estimate the impact of changes.
Some tips and tricks
Since
BenchmarkRunnerclass name in the example above does not qualify as a test according to Maven surefire plugin’s naming conventions, the benchmarks will not interfere with your JUnit tests.Benchmark methods need to be annotated by
@Benchmarkfor JMH to detect them.Classes containing benchmarks are found automatically by the
BenchmarkFinderwhen annotated with@JmhBenchmark.A reference to the Jenkins instance is available through either
JmhBenchmarkState#getJenkins()or throughJenkins.getInstance()like you would otherwise do.JmhBenchmarkStateprovidessetup()andtearDown()methods which can be overridden to configure the Jenkins instance according to your benchmark’s requirements.The benchmark builds on ci.jenkins.io are currently throttled because of the limited availability of
highmemnodes.The benchmark framework was made available in Jenkins Test Harness 2.50, it is recommended to use version 2.51 as it includes some bug fixes.
Links and Feedback
If you have any feedback, comments or questions, please feel free to reach out to me through either the Role Strategy Plugin Gitter chat or through the Jenkins Developer Mailing list.