Just search for BlueOcean beta in the Update Center, install it, browse to the dashboard, and then click the Try BlueOcean UI button on the dashboard.

Back in April we open sourced Blue Ocean and shared our vision with the community. We’re very happy that all the things we showed you then have shipped in the beta (software projects run on time?!).

For a refresher on Blue Ocean, watch this short video:

We have heard from the community about the usability of Jenkins Pipeline. Much of the feedback we received was to a desire toconfigure Pipelines rather than script them, and to make it easy for beginners to get started with their first Pipeline.

This is how Declarative Pipeline was born. We’ve introduced a new method whereby you declare how you want your Pipeline to look rather than using Pipeline Script - it’s configuration rather than code.

Here’s a small example of a Declarative Pipeline for nodejs that runs the whole Pipeline inside a Docker container:

Docker support in Declarative Pipeline allows you to version your application code, Jenkins Pipeline configuration, and the environment where your pipeline will run, all in a single repository. It’s a crazy powerful combination.

Declarative Pipeline introduces the postBuild section that makes it easy to run things conditionally at the end of your Pipeline without the complexity of the try... catch of Pipeline script.

And there is so much more!

If you have the Blue Ocean beta installed you already have Declarative Pipeline. While Declarative Pipeline is still alpha at the moment, we do encourage you tofollow our getting started guide, give us feedback on the Jenkins Users mailing list or file bugs against the pipeline-model-definition component in JIRA.

The Pipeline Editor is a graphical user interface that gives Jenkins users the simplest way yet to get started with creating Pipelines in Jenkins. It will also save a lot of time for intermediate and advanced Jenkins users as a way to author Pipelines.

When you build your Pipeline in the Editor and click the save button, the editor will commit a new Jenkinsfile back to your repository in the form of the new Declarative Pipeline. When you want to edit again, Jenkins will read it from your repository exactly how you saw it previously.

The Pipeline Editor is a work in progress and should arrive in a beta release soon.

Thanks for reading our news from Jenkins World and be sure to check the blog for regular updates!

I’d also like to thank our amazing community for their feedback and support as we change the way software teams around the world use Jenkins. We couldn’t do this without you.

1) Classic Jenkins UI framework -Daniel Beck

In the first part of his talk, Daniel presented how Stapler, the web framework used in Jenkins, works, and how you can add to the set of URLs handled by Jenkins. In the second part he was talking about creating new views using Jelly and Groovy, and how to add new content to existing views.

Keywords:Stapler, Jelly, Groovy-defined UIs

2) Developing modern Jenkins UIs with Javascript -Tom Fennelly

Feel that Jenkins UI is a bit old? You are not alone. In addition to the old stack Jenkins offers a framework for writing UI components in Javascript with help of Node.js. Tom presented this new engine, which is being used in new Jenkins Web UI components like Jenkins installation wizard. He also provided several examples from the BlueOcean project he is working on.

Keywords: Node.js, ReactJS,Jenkins JS Builder,Jenkins Design Language,Blue Ocean

We are looking for speakers, who would be interested to share their experience about Jenkins best-practices, war stories and plugin development.

If you are interested to conduct a presentation, please contact meetup organizers using meetup.com “contact organizers” feature or via the events@lists.jenkins-ci.org mailing list.

Any way you look at it, last week’s Jenkins World Conference 2016 was a huge success.

In 2011, a few hundred users gathered in San Francisco for the first "Jenkins User Conference". Over successive years, this grew into several yearly regional Jenkins user conferences. This year, over 1,300 people came from around the world to "Jenkins World 2016", the first global event for the Jenkins community.

This year’s Jenkins World conference included:

Over the next week, I’ll be posting highlights from the event, including slides, videos, and links to other useful resources. Stay tuned!

To kick off Jenkins World, we had a full day "Contributor Summit". Jenkins is a distributed project with contributors from all over the globe. Conferences like this are perfect time to get contributors together face-to-face, to talk through current issues and upcoming plans for the the project. Some key topics discussed during this summit were:

The next day,Kohsuke gave a greatkeynote, showing how far the project as come this year and where it is headed. You can get the slideshere or see the full video below.

Overall, Jenkins World was a very enjoyable event. I’m sure everyone came away having learned a lot and made many new connections. I know I’m excited to see what the coming year brings for Jenkins and the Jenkins community.

Don’t forget that there are many ways to continue to build connections to the rest of the Jenkins community throughout the year, such as the Jenkins Online Meetup which hosts online events year-round. Or, see if there is aJenkins Area Meetup (JAM) near you. If there isn’t, take a look at the Jenkins Area Meetup page to see about starting one.

Thanks, and I hope to see you all and Jenkins World 2017!

The stage step is a primary building block in Pipeline, dividing the steps of a Pipeline into explicit units and helping to visualize the progress using the "Stage View" plugin or Blue Ocean. Beginning with version 2.2 of "Pipeline Stage Step" plugin, the stage step now requires a block argument, wrapping all steps within the defined stage. This makes the boundaries of where each stage begins and ends obvious and predictable. In addition, the concurrency argument of stage has now been removed to make this step more concise; responsibility for concurrency control has been delegated to the lock step.

Omitting the block from stage and using the concurrency argument are now deprecated in Pipeline. Pipelines using this syntax will continue to function but will produce a warning in the console log:

This message is only a reminder to update your Pipeline scripts; none of your Pipelines will stop working. If we reach a point where the old syntax is to be removed we will make an announcement prior to the change. We do, however, recommend that you update your existing Pipelines to utilize the new syntax.

note: Stage View and Blue Ocean will both work with either the old stage syntax or the new.

Rather than attempt to limit the number of concurrent builds of a job using the stage, we now rely on the "Lockable Resources" plugin and the lock step to control this. The lock step limits concurrency to a single build and it provides much greater flexibility in designating where the concurrency is limited.

The milestone step is the last piece of the puzzle to replace functionality originally intended for stage and adds even more control for handling concurrent builds of a job. The lock step limits the number of builds running concurrently in a section of your Pipeline while the milestone step ensures that older builds of a job will not overwrite a newer build.

Concurrent builds of the same job do not always run at the same rate. Depending on the network, the node used, compilation times, test times, etc. it is always possible for a newer build to complete faster than an older build. For example:

Rather than allowing Build 1 to continue and possibly overwrite the newer artifact produced in Build 2, you can use the milestone step to abort Build 1:

When using the input step or the lock step a backlog of concurrent builds can easily stack up, either waiting for user input or waiting for a resource to become free. The milestone step will automatically prune all older jobs that are waiting at these junctions.

Bookending an input step like this allows you to select a specific build to proceed and automatically abort all antecedent builds.

Similarly a pair of milestone steps used with a lock will discard all old builds waiting for a shared resource. In this example, inversePrecedence: true instructs the lock to begin most recent waiting build first, ensuring that the most recent code takes precedence.

Each of these steps can be used independently of the others to control one aspect of a Pipeline or they can be combined to provide powerful, fine-grained control of every aspect of multiple concurrent builds flowing through a Pipeline. Here is a very simple example utilizing all three:

For a more complete and complex example utilizing all these steps in a Pipeline check out the Jenkinsfile provided with the Docker image for demonstrating Pipeline. This is a working demo that can be quickly set up and run.

No matter the project, you need to gather and report test results. JUnit is one of the most widely supported formats for recording test results. For a scenarios where your tests are stable and your framework can produce JUnit output, this makes the JUnit plugin ideal for reporting results in Jenkins. It will consume results from a specified file or path, create a report, and if it finds test failures it will set the the job state to "unstable" or "failed".

There are also plenty of scenarios where the JUnit plugin is not enough. If your project has some failing tests that will take some time to fix, or if there are some flaky tests, the JUnit plugin’s simplistic view of test failures may be difficult to work with.

No problem, the Jenkins plugin model lets us replace the JUnit plugin functionality with similar functionality from another plugin and Jenkins Pipeline lets us do this in safe stepwise fashion where we can test and debug each of our changes.

In this article, I will show you how to replace the JUnit plugin with the xUnit plugin in Pipeline code to address a few common test reporting scenarios.

I’m going to use the "JS-Nightwatch.js" sample project from my previous post to demonstrate a couple common scenarios that the xUnit handles better. I already have the latestJUnit plugin andxUnit plugin installed on my Jenkins server.

I’ll be keeping my changes in link:my fork of the "JS-Nightwatch.js" sample project on GitHub, under the "blog/xunit" branch.

Here’s what the Jenkinsfile looked like at the end of that previous post and what the report page looks like after a few runs:

I’ll start by replacing JUnit with xUnit in my pipeline. I use the Snippet Generator to create the step with the right parameters. The main downside of using the xUnit plugin is that while it is Pipeline compatible, it still uses the more verbose step() syntax and has some very rough edges around that, too. I’ve filed JENKINS-37611 but in the meanwhile, we’ll work with what we have.

If I replace the junit step in my Jenkinsfile with that last step above, it produces a report and job result identical to the JUnit plugin but using the xUnit plugin. Easy!

Most projects don’t start off with automated tests passing or even running. They start with a people hacking and prototyping, and eventually they start to write tests. As new tests are written, having tests checked-in, running, and failing can be valuable information. With the xUnit plugin we can accept a baseline of failed cases and drive that number down over time.

I’ll start by changing the Jenkinsfile to fail jobs only if the number of failures is greater than an expected baseline, in this case four failures. When I run the job with this change, the reported numbers remain the same, but the job passes.

Next, I can also check that the plugin reports the job as failed if more failures occur. Since this is sample code, I’ll do this by adding another failing test and checking the job reports as failed.

In a real project, we’d make fixes over a number of commits bringing the number of failures down and adjusting our baseline. Since this is a sample, I’ll just make all tests pass and set the job failure threshold for failed and skipped cases to zero.

We’ve all known the frustration of having one flaky test that fails once every ten jobs. You want to keep it active so you can working isolating the source of the problem, but you also don’t want to destablize your CI pipeline or reject commits that are actually okay. You could move the test to a separate job that runs the "flaky" tests, but in my experience that just leads to a job that is always in a failed state and a pile of flaky tests no one looks at.

With the xUnit plugin, we can keep the this flaky test in main test suite but allow the our job to still pass.

I’ll start by adding a sample flaky test. After a few runs, we can see the test fails intermittently and causes the job to fail too.

I can almost hear my teammates screaming in frustration just looking at this report. To allow specific tests to be unstable but not others, I’m going to add a guard "suite completed" test to the suites that should be stable, and keep flaky test on it’s own. Then I’ll tell xUnit to allow for a number of failed tests, but no skipped ones. If any test fails other than the ones I allow to be flaky, it will also result in one or more skipped tests and will fail the build.

After a few more runs, you can see the flaky test is still being flaky, but it is no longer failing the build. Meanwhile, if another test fails, it will cause the "suite completed" test to be skipped, failing the job. If this were a real project, the test owner could instrument and eventually fix the test. When they were confident they had stabilized the test the could add a "suite completed" test after it to enforce it passing without changes to other tests or framework.

This post has shown how to migrate from the JUnit plugin to the xUnit plugin on an existing project in Jenkins pipeline. It also covered how to use the features of xUnit plugin to get more meaningful and effective Jenkins reporting behavior.

What I didn’t show was how many other formats xUnit supports - from CCPUnit to MSTest. You can also write your own XSL for result formats not on the known/supported list.

James Dumay took time out of his vacation to present Blue Ocean, a project that rethinks the user experience of Jenkins, modeling and presenting the process of software delivery by surfacing information that is important to development teams with as few clicks as possible, while still staying true to the extensibility that Jenkins always has had as a core value.

See recording HERE.

Andrey Falko from Salesforce shared how he and his Diagnostics team used Jenkins to deliver software securely and reliably to production within Salesforce.

See videos HERE andHERE.

This was a meetup with CA Technologies and includedMark Waite, maintainer of the Jenkins git plugin and a director at CA Technologies in Fort Collins.Tyler did a great presentation about Jenkins Pipeline and Blue Ocean and showed off how the community is using Blue Ocean to build Jenkins.

At this meetup, there were plenty of engaging discussions surrounding the Jenkins Certification and DevOps 2.1 Toolkit: Continuous Deployment with Jenkins and Docker Swarm. Guillem Sola shared his Jenkins certification experience HERE while Viktor Farcic presented his thoughts on the aspects of building, testing, deploying, and monitoring services inside Docker Swarm clusters and Jenkinshttps://www.youtube.com/watch?v=fs1ED_y5mUc.

October’s meetup was a joint effort with collaboration from Perú JUG, and Docker Lima. The first talk was an Introduction to Docker Ecosystem, second wasBuilding and Testing Apps with Docker andArquillian Cube and the last one wasCI/CD using Docker and Jenkins Pipelines.

We had a full house at the meetup. Now, everyone in the room has a Mr. Jenkins branded on their laptop :)

Special thanks to Mario Inga andHéctor Paz for their collaborations during the last meetups.

Now, let’s look at where those came from! We’re going to focus on garbage collection (GC) here and dig fast and deep to strike for gold; if you’re not familiar with GC fundamentals take a look at this source.

Because performance tuning is data driven, I’m going to use real-world data selected three very large Jenkins instances that I help support.

What we’re not going to do: Jenkins basics, or play with max heap. See the section "what should I do before tuning." This is for cases where we reallydo need a big heap and can’t easily split our Jenkins masters into smaller ones.

Symptom: Users report that the Jenkins instance periodically hangs, taking several seconds to handle normally fast requests. We may even see lockups or timeouts from systems communicating with the Jenkins master (build agents, etc). In long periods of heavy load, users may report Jenkins running slowly. Application monitoring shows that during lockups all or most of the CPU cores are fully loaded, but there’s not enough activity to justify it. Process and JStack dumps will reveal that the most active Java threads are doing garbage collection.

With Instance A, they had this problem. Their Jenkins Java arguments are very close to the default, aside from sizing the heap:

After enabling garbage collection (GC) logging we see the following rough stats:

.

Diving deeper, we get this chart of GC pause durations:

Key stats:

Explanations:

As you can see, young GC cycles very quickly clear away freshly-created garbage, but the deeper old-gen GC cycles run very slowly: 2-4 seconds. This is where our problems happen. The default Java garbage collection algorithm (ParallelGC) pauses everything when it has to collect garbage (often called a "stop the world pause"). During that period, Jenkins is fully halted: normally (with small heaps) these pauses are too brief to be an issue. With heaps of 4 GB or larger, the time required becomes long enough to be a problem: several seconds over short windows, and over a longer interval you occasionally see much longer pauses (tens of seconds, or minutes.)

This is where the user-visible hangs and lock-ups happen. It also adds significant latency to those build/deploy tasks. In periods of heavy load, the system was even experiencing hangs of 30+ seconds for a single full GC cycle. This was long enough to trigger network timeouts (or internal Jenkins thread timeouts) and cause even larger problems.

Fortunately there’s a solution: the concurrent low-pause garbage collection algorithms, Concurrent Mark Sweep (CMS) and Garbage First (G1). These attempt to do much of the garbage collection concurrently with application threads, resulting in much shorter pauses (at a slight cost in extra CPU use). We’re going to focus on G1, because it is slated to become the default in Java 9 and is the official recommendation for large heap sizes.

Let’s see what happens when someone uses G1 on a similarly-sized Jenkins master with Instance B (17 GB heap):

Their settings:

And the GC log analysis:

Key stats:

Okay, much better: some improvement may be expected from a 30% smaller heap, but not as much as we’ve seen. Most of the GC pauses are under well under 2 seconds, but we have 11 outliers - long Full GC pauses of 2-12 seconds. Those are troubling; we’ll take a deeper dive into their causes in a second. First, let’s look at the big picture and at how Jenkins behaves with G1 GC for a second instance.

G1 Garbage Collection with Instance C (24 GB heap):

Their settings:

Clearly they’ve done some garbage collection tuning and optimization. The AlwaysPreTouch pre-zeros allocated heap pages, rather than waiting until they’re first used. This is suggested especially for large heap sizes, because it trades slightly slower startup times for improved runtime performance. Note also that they pre-allocated the whole heap. This is a common optimization.

They also enabled StringDeduplication, a G1 option introduced in Java 8 Update 20 that transparently replaces identical character arrays with pointers to the original, reducing memory use (and improving cache performance). Think of it like String.intern() but it silently happens during garbage collection. This is a concurrent operation added on to normal GC cycles, so it doesn’t pause the application. We’ll look at its impacts later.

Looking at the basics:

Similar picture to Instance B, but it’s hidden by the sheer number of points (this is a longer period here, 1 month). Those same occasional Full GC outliers are present!

Key stats:

Overall fairly similar to Instance B: ~100 ms GC cycles, all the minor GC cycles are very fast. Object promotion rates sound similar.

Remember those random long pauses?

Let’s find out what caused them and how to get rid of them. Instance B had 11 super-long pause outliers. Let’s get some more detail, by opening GC Logs in GCViewer. This tool gives a tremendous amount of information. Too much, in fact —  I prefer to use GCEasy.io except where needed. Since GC logs do not contain compromising information (unlike heap dumps or some stack traces), web apps are a great tool for analysis.

What we care about are at the Full GC times in the middle (highlighted). See how much longer they are vs. the young and concurrent GC cycles up top (2 seconds or less)?

Now, I lied a bit earlier - sorry! For concurrent garbage collectors, there are actually 3 modes: young GC, concurrent GC, and full GC. Concurrent GC replaces the Full GC mode in Parallel GC with a faster concurrent operation that runs in parallel with the application. But in a few cases, we will are forced to fall back to a non-concurrent Full GC operation, which will use the serial (single-threaded) garbage collector. That means that even if we have 30+ CPU cores, only one is working. This is what is happening here, and on a large-heap, multicore system it is S L O W. How slow? 280 MB/s vs. 12487 MB/s for Instance B (for instance C, the difference is also about 50:1).

What triggers a full GC instead of concurrent:

How do we fix this?

For explicit GC:

Gotcha for people who’ve used CMS: if you have used CMS in the past, you may have used the option -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses — which does what it says. This option will silently fail in G1, meaning you still see the very long pauses from Full GC cycles as if it wasn’t set (no warning is generated). I have logged a JVM bug for this issue.

For the Metadata GC threshold:

Instance C also suffered the same problem with explicit GC calls, with almost all our outliers accounted for (230 out of 235) by slow, nonconcurrent Full GC cycles (all from explicit System.gc() calls, since they tuned metaspace):

Here’s what GC pause durations look like if we remove the log entries for the explicit System.gc() calls, assuming that they’ll blend in with the other concurrent GC pauses (not 100% accurate, but a good approximation):

Instance B:

The few long Full GC cycles at the start are from metaspace expansion — they can be removed by increasing initial Metaspace size, as noted above. The spikes? That’s when we’re about to resize the Java heap, and memory pressure is high. You can avoid this by setting the minimum/initial heap to at least half of the maximum, to limit resizing.

Stats:

Instance C:

Stats:

Okay, so we’ve tamed the long worst-case pauses!

Okay, now we’re in the home stretch! We’ve tamed the old-generation GC pauses with concurrent collection, but what about those longer young-generation pauses? Lets look at stats for the different phases and causes again in GCViewer.

Highlighted in yellow we see the culprit: the remark phase of G1 garbage collection. This stop-the-world phase ensures we’ve identified all live objects, and processes references ( more info).

Let’s look at a sample execution to get more info:

This turns out to be typical for the GC log: the longest pauses are spent in reference processing. This is not surprising because Jenkins internally uses references heavily for caching, especially weak references, and the default reference processing algorithm is single-threaded. Note that user (CPU) time matches real time, and it would be higher if we were using multiple cores.

So, we add the GC flag -XX:+ParallelRefProcEnabled which enables us to use the multiple cores more effectively.

Tuning young-generation GC further based on Instance C:

Back to GCViewer we go, to see what’s time consuming with the GC for Instance C.

That’s good, because most of the time is just sweeping out the trash (evacuation pause). But the 1.8 second pause looks odd. Let’s look at the raw GC log for the longest pause:

…​oh, well dang. Almost the entire time (1.792 s out of 1.820) is walking through the live objects and copying them. And wait, what about this line, showing the summary statistics:

Good grief, we flushed out 13 GB (!!!) of freshly-allocated garbage in one swoop and compacted the leftovers! No wonder it was so slow. I wonder how we accumulated so much…​

Oh, right…​ we set up for 24 GB of heap initially, and each minor GC clears most of the young generation. Okay, so we’ve set aside tons of space for trash to collect, which means longer but less frequent GC periods. This also gets the best performance from Jenkins memory caches which are using WeakReferences (survives until collected by GC) and SoftReferences (more long-lived). Those caches boost performance a lot.

We could take actions to prevent those rare longer pauses. The best ways are to limit total heap size or reduce the value of -XX:MaxGCPauseMillis=200 from its default (200). A more advanced way (if those don’t help enough) is to explicitly set the maximum size of the young generation smaller (say -XX:G1MaxNewSizePercent=45 instead of the default of 60). We could also throw more CPUs at the problem.

But if we look up, most pauses are around 100 ms (200 ms is the default value for MaxGCPauseMillis). For Jenkins on this hardware, this appears to workjust fine and a rare longer pause is OK as long as they don’t get too big. Also remember, if this happens often, G1 GC will try to autotune for lower pauses and more predictable performance.

We mentioned StringDeduplication was on with Instance C, what is the impact? This only triggers on Strings that have survived a few generations (most of our garbage does not), has limits on the CPU time it can use, and replaces duplicate references to their immutable backing character arrays.For more info, look here. So, we should be trading a little CPU time for improved memory efficiently (similarly to string interning).

At the beginning, this has a huge impact:

This peaks at an average of about ~90%:

After running for a month, less of an impact - many of the strings that can be deduplicated already are:

However it’s cheap to use: in average, each dedup cycle takes 8.8 ms and removes 2.4 kB of duplicates. The median takes 1.33 ms and removes 17.66 kB from the old generation. A small change per cycle, but in aggregate it adds up quickly — in periods of heavy load, this can save hundreds of megabytes of data. But that’s still small, relative to multi-GB heaps.

Conclusion: turn string deduplication on string deduplication is fairly cheap to use, and reduces the steady-state memory needed for Jenkins. That frees up more room for the young generation, and should overall reduce GC time by removing duplicate objects. I think it’s worth turning on.

Soft reference flushing: Jenkins uses soft references for caching build records and in pipeline FlowNodes. The only guarantee for these is that they will be removed instead of causing an OutOfMemoryError…​ however Java applications can slow to a crawl from memory pressure long before that happens. There’s an option that provides a hint to the JVM based on time & free memory, controlled by -XX:SoftRefLRUPolicyMSPerMB (default 1000). The SoftReferences become eligible for garbage collection after this many milliseconds have elapsed since last touch…​ per MB of unused heap (vs the maximum). The referenced objects don’t count towards that target. So, with 10 GB of heap free and the default 1000 ms setting, soft references stick around for ~2.8 hours (!).

If the system is continuously allocating more soft references, it may trigger heavy GC activity, rather than clearing out soft references. See the open bugJDK-6912889 for more details.

If Jenkins consumes excessive old generation memory, it may help to make soft references easier to flush by reducing -XX:SoftRefLRUPolicyMSPerMB from its default (1000) to something smaller (say 10-200). The catch is that SoftReferences are often used for objects that are relatively expensive to load, such lazy-loaded build records and pipeline FlowNode data.

G1 vs. CMS:

G1 was available on later releases of JRE 7, but unstable and slow. If you use it you absolutely must be using JRE 8, and the later the release the better (it’s gotten a lot of patches). Googling around will show horrible G1 vs CMS benchmarks from around 2014: these are probably best ignored, since the G1 implementation was still immature then. There’s probably a niche for CMS use still, especially on midsized heaps (1-3 GB) or where settings are already tuned. With appropriate tuning it can still perform generally well for Jenkins (which mostly generates short-lived garbage), but CMS eventually suffer from heap fragmentation and need a slow, non-concurrent Full GC to clear this. It also needs considerably more tuning than G1.

General GC tuning caveats:

No single setting is perfect for everybody. We avoid tweaking settings that we don’t have strong evidence for here, but there are of course many additional settings to tweak. One shouldn’t change them without evidence though, because it can cause unexpected side effects. The GC logs we enabled earlier will collect this evidence. The only setting that jumps out as a likely candidate for further tuning is G1 region size (too small and there are many humungous object allocations, which hurt performance). Running on smaller systems, I’ve seen evidence that regions shouldn’t be smaller than 4 MB because there are 1-2 MB objects allocated somewhat regularly — but it’s not enough to make solid guidance without more data.

If you’ve seen Stephen Connolly’s excellent Jenkins World talk, you know that most Jenkins instances can and should get by with 4 GB or less of allocated heap, even up to very large sizes. You will want to turn on GC logging (suggested above) and look at stats over a few weeks (rememberGCeasy.io). If you’re not seeing periodic longer pause times, you’re probably okay.

For this post we assume we’ve already done the basic performance work for Jenkins:

If not, we need to do that FIRST before looking at GC tuning, because those will have larger impacts.

We’ve gone from:

How:

There’s still a little unpredictability, but using appropriate settings gives a much more stable, responsive CI/CD server…​ even up to 20 GB heap sizes!

I’ve added -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=20 to our options above. This is covering a complex and usually infrequent case where G1 self-tuning can trigger bad performance for Jenkins — but that’s material for another post…​