Updated 2015-11-11 15:00 UTC: We have released Jenkins 1.638 and 1.625.2 which contain a fix for this vulnerability. See the security advisory for more information about these releases.

Updated 2015-11-06 03:55 UTC: Included a updated mitigation script which doesn't have a Jenkins boot race condition

Earlier today we received numerous reports about a previously undisclosed "zero day" critical remote code execution vulnerability and exploit in Jenkins core. Unfortunately the vulnerability was not disclosed to us ahead of its publication so we're still working on more thorough fix. In the meantime however, we wanted to inform you of the issue and provide a workaround which will help prevent this exploit from being used against public Jenkins installations, for future reference this issue is being tracked privately as SECURITY-218 in our issue tracker.

The attack is mounted through the Jenkins CLI subsystem, so the work-around is to remove/disable the CLI support inside of the running Jenkins server.

Using the following Groovy script you can disable the attack vector in your Jenkins installations by navigating to “Manage Jenkins” and then to “Script Console”, or just go to http://your-jenkins-installation/script. This only addresses the current running Jenkins process, in order to make the workaround persist between restarts of the Jenkins server, add the script below to $JENKINS_HOME/init.groovy.d/cli-shutdown.groovy (create the directory if necessary, and the file).

import jenkins.*;
import jenkins.model.*;
import hudson.model.*;

// disabled CLI access over TCP listener (separate port)
def p = AgentProtocol.all()
p.each { x ->
  if (x.name.contains("CLI")) p.remove(x)
}

// disable CLI access over /cli URL
def removal = { lst ->
  lst.each { x -> if (x.getClass().name.contains("CLIAction")) lst.remove(x) }
}
def j = Jenkins.instance;
removal(j.getExtensionList(RootAction.class))
removal(j.actions)

in order to make the workaround persist between restarts of the Jenkins server, add the script below to $JENKINS_HOME/init.groovy.d/cli-shutdown.groovy (create the directory if necessary, and the file).

The latest version of this script can be found in this GitHub repository.

As previously announced on the jenkinsci-advisories mailing list we’re preparing a security release for this upcoming Wednesday which will include patches for both the latest and LTS lines of Jenkins core. The Jenkins Security team is working to include a fix for this previously undisclosed exploit in or before this planned security release.

If you have questions about this exploit, join us in the #jenkins channel on Freenode or ask on the jenkinsci-users@ mailing list.

For security researchers and hobbyists, if you believe you have found a security vulnerability in Jenkins, we have some disclosure guidelines on this wiki page which will help us mitigate any discovered issues as quickly and safely as possible.

Be sure to subscribe to the jenkinsci-advisories mailing list (jenkinsci-advisories), it's the fastest way to get updates by the Jenkins Security team.

It is great to see the pick up of local activities through hosted JAMs. In October, the Jenkins community hosted Atlanta JAM and Bay Area JAM. Many thanks to our sponsors: Ericsson, CloudBees, Blazemeter, NetRoadShow.

Here's a summary of what was discussed:

• Atlanta JAM - Jenkins workflow and Docker to reduce friction in DevOps efforts.
• Bay Area JAM- Performance testing strategies, incorporating performance tests into Jenkins workflows and the metrics that matter most for troubleshooting and diagnosing issues.

A look forward to November and December:

• Hacksgiving
• Barcelona, Spain JAM
• Toulouse,France JAM

As usual, if you're interested in becoming an organizer or sponsor, here's how to get started. If you've heard of a great Jenkins talk out there, shoot us an email with speaker info to jenkinsci-jam@googlegroups.com so we can invite him/her to our next meetups.

We just released Jenkins 1.638 and 1.625.2 which contain important security fixes, including a fix for the zero-day vulnerability published on Friday. Please see the security advisory for more information.

Want to be kept up to date on Jenkins security releases, including advance notice on scheduled security updates? Subscribe to the jenkinsci-advisories mailing list!

Next week in the US we have a national holiday where, generally speaking, lots of turkey gets converted into left-over turkey sandwiches. For many software developers the Thanksgiving holiday also represents a lull in project schedules, freeing up some time to hack on pet projects or even contribute to open source projects.

Taking a cue from the Adopt a Plugin program that Daniel wrote about earlier this month, we thought it would be fun to organize a "virtual hackathon" to coincide with that gap in project schedules. Thus Hacksgiving 2015 was created!

We'll be hosting Hacksgiving Nov 23rd and Nov 24 from 7:00PST - 15:00PST (10:00EST - 18:00EST) and would love for you to join! (RSVP here)

You don't need to know Java to help! We will have documentation and design hacking going on as well.

We have a few goals for Hacksgiving:

1. Introduce new contributors to the process of writing code and/or documentation (documentation hacking details here).
2. Find some plugins which are up for adoption new maintainers.
3. Clean up or merge some existing plugins which need some care (listed here). ### Sessions to note

Here are some of the sessions scheduled that will be hosted by members of community that may interest you:

Day One

• 7:00PST/10:00EST (15-30min) - rtyler will host a welcome and introduction to contribution to the Jenkins project (walking through our contributors guide)
• 10:00PST/13:00EST (60min) - schristou will host a workshop titled "Introduction to plugin development for Jenkins"

Day Two

• 10:00PST/13:00EST (60min) - abayer will be hosting a "Plugin Developer Open Q&A" session, so bring your questions!

Hacksgiving is very unconference structured right now, so if you're interested in hosting a session please let us know in the #jenkins-community channel or by signing up for a session on the schedule

How to participate

Since this is a virtual hackathon, we'll be congregating and chatting in a couple of ways:

• On the #jenkinsIRC channel as per usual
• We'll be hosting sessions and tutorials via Google Hangouts, see the "hacker hangout* section on the wiki page up to date details
• Via the #hacksgiving hashtag on Twitter

You can also RSVP on our meetup page!

We hope you can join in the festivities!

Last week we hosted our first Hacksgiving event, a two-day virtual hackathon with a number of recorded sessions and plenty of pull requests submitted, I would say it was a success! I would like to thank everybody who took the time to watch, chat and present in the Hacker Hangout.

Now that everybody has had time to recover from the turkey and travel, we have some videos of the sessions sliced out and ready for publication.

In addition to the recorded sessions, there were a number of notes captured with useful links associated with practically each session. You can find those notes at the bottom of the Hacksgiving page.

Note: The following videos are all available in this YouTube playlist

Intro to the Jenkins project

This session was hosted by rtyler and meant to provide a cursory overview of where to get started with contributing to the Jenkins project

Intro to Plugin Development Workshop

This session was given both days of Hacksgiving by schristou and does a really great job of introducing the viewer to getting started with developing a Jenkins plugin with Java.

Workflow Q&A and Demo Session

This session was not originally scheduled, but some folks on the Jenkins IRC channel had some Workflow questions and Jesse Glick jumped into the Hacker Hangout to help us out!

Internationalization Live Coding / Q&A

Another impromptu session, this time with danielbeck hosting. In this session Daniel walks through a plugin he was working on for Hacksgiving and adds internationalization support while answering a few questions here and there.

Intro to the new static site

Kicking off day two of Hacksgiving, rtyler hosted a session on the new statically-generated Jenkins site. The new site will dramatically lower the barrier to entry for contribution to Jenkins documentation and blogs, by pushing everything through GitHub.

Plugin Developer Open Q&A

This was the last session of Hacksgiving, hosted by abayer and ended up being more like a casual discussion of the current status and future work in the plugin development ecosystem.

Note: This is a guest post by Kishore Bhatia. Kishore works for CloudBees, building custom frameworks with Open Source software and helping customers solve engineering problems around continuous delivery and DevOps at scale.

This year some great new Jenkins features came out of the butler’s goodie bag - amongst them, the most important one being the ability to realize continuous delivery pipeline as code! The features like Workflow Multibranch, pipeline-as-code (with a marker file that Jenkins looks for in your application’s SCM repository/branch, aptly named Jenkinsfile) are the foundations to making Jenkins super intelligent to automagically create workflows (rather, a CI/CD pipeline) to build your code and orchestrate the work required to drive your application from concept to delivery!

Overview

The Workflow Multibranch feature (provided by the workflow plugin) provides the following key abilities:

• Automatic Workflow (job) creation in Jenkins per new branch in the repo.
• Build specific to that child-branch and its unique scm change and build history.
• Automatic job pruning/deletion for branches deleted from the repository, according to the settings.
• Flexibility to individually configure branch properties, by overriding the parent properties, if required.

Jenkins pipeline-as-code (concept) enables you to maintain your CI/CD workflow logic in the project/application source code repo with no additional configuration to be maintained per branch in Jenkins.

The Workflow script to build/test/deploy your code is always synchronized with the rest of the source code you are working on.

To demonstrate the concept here - Let’s use a basic Java Web application project with a Maven pom.xml as shown in the structure below (this is using GitHub as the SCM but you can do this on SVN or Mercurial too).

This project has a marker file for Jenkins in the repo - Jenkinsfile.

So, what's a Jenkinsfile? The Jenkinsfile is essentially your Jenkins Workflow, a script, that defines the CI/CD pipeline logic for a project with steps to build/test/deploy etc. captured in various stages.

So for our sample Java web application, a basic Jenkinsfile could be something like -

<pre> node { // Mark the code checkout 'stage'.... stage 'Checkout'

// Checkout code from repository checkout scm

// Get the maven tool. // ** NOTE: This 'M3' maven tool must be configured // ** in the global configuration.
def mvnHome = tool 'M3'

// Mark the code build 'stage'…. stage 'Build' // Run the maven build sh "${mvnHome}/bin/mvn clean install" }</pre></code>

Just having this file in the source code repo root would mean that -

• Jenkins will automatically recognize this branch and create appropriate jobs by itself.
• Quick, 1-step code checkout using: “checkout scm” in your workflow
• Every time a new change is pushed to this branch, the branch is built and the commit status gets updated.
• When the branch is destroyed in the repository, or if Jenkinsfile is removed, the corresponding job gets destroyed from Jenkins automatically (You can retain these jobs and/or archive the builds for audit/compliance requirements using the retention property - Orphan Item strategy)

Note: there are various mechanisms to promote reuse of Workflow scripts, such as the Workflow Global Library.

Required Jenkins configuration

Make sure you’ve the latest Workflow and (v1.11 as of writing this blog) Workflow Multibranch plugins installed on your Jenkins instance

Also, ensure that other dependencies, like SCM plugins and build tools, are met:

• Either SVN/Git/Mercurial (depending on your SCM)
• GitHub Branch Source Plugin (optimized to use the GitHub API and improve performance)
• Maven build tool

Then create a new Multibranch Workflow Job with configuration as shown below - mainly selecting the Branch Sources (Git, in this example) and providing the branch/repo URL with credentials.

Branch sources (Git) - https://github.com/kishorebhatia/pipeline-as-code-demo (or a repo where you’ve cloned this source code with Jenkinsfile)

Leave all other properties default and Save.

You’ll observe that Jenkins would perform Branch Indexing on that “cd” job folder and start the workflow for the master branch, with an automatically created new job, named master, under the “cd” folder.

The workflow does a dummy step for application deploys to the environments in this sequence Staging -> Waits for manual approval -> PROD

Now, let’s create a new branch off of this master branch in your cloned git repo:

• $ git branch newBranch (create a newBranch)
• $ git checkout newBranch (switches to newBranch)
• $ git push --set-upstream origin newBranch (pushes newBranch)

You’ll observe that your Jenkins instance automatically picks up this newBranch and starts running the workflow (with the Jenkinsfile in this newBranch) to build/test/deploy the code.

Next, if you now delete this newBranch (git branch -D newBranch), Jenkins will automatically remove the orphan Workflow job for newBranch. You can retain these jobs even after the branches are deleted using the Orphaned Item Strategy property in the main "cd" job’s configuration.

So we observed the following benefits of this pipeline-as-code approach:

• Overall job definition is a script (Jenkinsfile)
• Calls your build tools and scripts for details
• The build script can be versioned alongside project sources
• Jenkins handles feature/experimental branches automatically
• Keep less configuration in $JENKINS_HOME

Dockerized Demo environment

You can also use the following docker image to run this demo with a preconfigured Jenkins environment and the sample job: jenkinsci/workflow-demo (i.e. docker pull jenkinsci/workflow-demo)

This docker container includes Jenkins with Workflow and Workflow Multibranch plugins, a local git repo with the aforementioned Java web application and Jetty to demonstrate a continuous delivery pipeline of this application deployed and tested across multiple environments in the pipeline with an approval gate before promoting to PROD (like QA, Staging and PROD).

There's a "cd" job pre-configured as a multibranch Workflow job.

Launch the docker demo as: docker run -p 8080:8080 -p 8081:8081 -p 9418:9418 -ti jenkinsci/workflow-demo

Now, you can access Jenkins on port 8080 and Jetty on port 8081 from localhost or the IP of your boot2docker/docker-machine environment.

The demo container has a local git repo so you can clone: git://localhost/repo. When creating new branches, each branch automatically creates a matching subproject in Jenkins and triggers the build for that branch. The workflow:

• Checks out source code from the same repository and commit as Jenkinsfile.
• Builds sources via Maven with unit testing.
• Runs two parallel integration tests that involve deploying the app to ephemeral server instances, which get thrown away when tests are done (this is done by using auto-deployment of Jetty)
• Once integration tests are successful, the webapp gets to the staging server at localhost:8081/staging (or your docker-machine/boot2docker instance IP)
• requires a human to Manually inspect the staging instance, and when ready, approves the deployment to the production server at http://localhost:8081/production/

References

• Developer blog by jglick introducing multibranch support
• workflow plugin tutorial
• workflow plugin presentations
• workflow plugin readme

We released Jenkins updates today that include important security fixes: 1.641 and 1.625.3. For detailed information about the security content of these updates, see the security advisory.

One of these fixes, SECURITY-95, results in possible problems in plugins such as Maven Plugin, Javadoc Plugin, and HTML Publisher Plugin, so make sure to read all about that in the wiki.

Please note that the update site may lag a bit behind. If you want to update as soon as possible, download the releases from our site.

A lot of people are using the Workflow plugin, but as with any scripting environment, users often have to start from scratch and learn the same lessons and shortcuts that other users have already learned. While there are blog posts from developers and users in various places, and some samples in the Workflow plugin documentation, more examples and tips and tricks are always, always useful. To help with that, we've created the workflow-examples repository on GitHub, as a place to store community-developed Workflow scripts that can help new users get started, show how to accomplish some non-trivial goals, and find tips and trick for taking your Workflow pipeline to the next level.

The repository has four directories:

• docs/ - documentation, guides, and more. Including a Best Practices document. We'd love to see more contributions to that doc, as well as any new ones that would be helpful to Workflow users!
• workflow-examples/ - general Workflow examples, showing how to use a given plugin with Workflow, quirks of the Workflow DSL syntax, and more.
• global-library-examples/ - examples of how to write code for the Workflow global library.
• jenkinsfile-examples/ - Sample Jenkinsfiles or other Workflow scripts from SCM .

During Hacksgiving some initial content was added, but not everything is covered yet, which is why I'm posting this - more is needed. We'd love to see your tips, examples, gotchas and more. If you've got Workflow scripts you'd like to contribute, please read the README and send a pull request. Thanks!

While we are gearing up for FOSDEM 2016 early next year in Brussels, I wanted to remind the Jenkins community about our Travel Grant Program. We're a little late on mentioning it, but the board has allocated the money to help Jenkins contributors travel to Brussels to participate in FOSDEM and the Jenkins Contributor Summit which we will be hosting the day after, Feb 1st, which we'll discuss more in a later blog post.

For the FOSDEM Travel Grant Program, we are able to cover up to $500 (USD) in expenses to help community members participate in FOSDEM.

If you're interested, please read the description of the program below. Please note that some of the details of the program are different from the linked grant program page

Regardless we hope to see you all at FOSDEM on January 30th and 31st, 2016, in Brussels!

Eligibility

All community members are eligible for support unless they've received a travel grant within the last year (based on the event's date). However, as we have very limited funds to support this program, we'll prefer applications by active contributors to the Jenkins project.

If you have other possible funding sources, please look to them first. This will allow more people to attend a Jenkins community event.

Application

The application process for FOSDEM, due to our poor timing, deviates from the traditional Travel Grant Program.

To apply for a travel grant, send an email with the following information to the Governance Board at jenkinsci-board@googlegroups.com before January 6th.

• Your name
• The event you'd like to attend
• The expected cost of travel (airfare, hotel, conference fees, etc.)
• A description of your contributions to the Jenkins project, such as:
• Plugins you developed
• Pull requests you authored
• Documentation you wrote
• Public presentations on Jenkins-related topics
• Why should we sponsor your trip?

Applicants Responsibilities

If you've been selected for a travel grant, we'll expect you to:

• Be available for a blog post about this program before the event.
• Help out at the Jenkins stand at FOSDEM
• If your schedule permits, we'd love to see you at the Jenkins 2.0 Contributor Summit the day after FOSDEM.

It should go without saying that we expect all Jenkins contributors representing the project at an event such as FOSDEM to act in a respectful and constructive manner. As we have not yet formally adopted our own Code of Conduct, we recommend reviewing the FOSDEM Code of Conduct.

After the trip, please submit a travel report to jenkinsci-dev@googlegroups.com mailing list. This report should include the following:

• What you accomplished at the event
• What you learned at the event
• Contacts you made
• Other useful information

We also expect you to submit your receipts via email to the person mentioned in the travel grant confirmation for review. We will reimburse actually incurred costs up to the 500 USD limit.

Thank you to Netflix for sponsoring the yummy burrito bar and offered up their brand new auditorium to host Jenkins Developers and Users Meetup group on Dec 16. We had 96 RSVPs which was impressive. Our speaker for the evening was Akshay Dayal, Software Engineer at Google. Akshay's session was about Scaling Jenkins - how and why Google decided to scale their existing Jenkins cluster (OSS) to meet their security/availability and failover requirements and how heavy automation played an important role in this effort.

The second talk was about how Google worked with Jenkins to read config data externally. Slides are listed below. The video will be posted on the meetup page) once it becomes available.

Slides for the talks are linked below:

• Scaling Jenkins
• External Project

Check out where Jenkins Area Meetups (JAMs) are located in the world. Don't see a JAM in your area? why not start your own, here's how.

Although December is a short month due to the holidays, there has been a good amount of local Jenkins activities that took place regardless of holiday obligations. Today and tomorrow I will be doing a series of posts to summarize December JAM World Tour. Special thanks to the JAM organizers and co-organizers who made it all happen in these cities:

• Lima, Peru
• St.Petersburg, Russia
• Toulouse, France
• Bay Area, CA

On December 9 Lima JAM hosted their first Jenkins meetup in Lima, Peru. There were attendance from various roles of DevOps: Dev, QA, and Ops. There was also a good mixture of different levels of Jenkins users, some were new and just starting to use Jenkins while others had extensive Jenkins experience.

The group has been invited by Docker and Ansible meetup organizers for a joint event in January to showcase technologies from Jenkins, Docker, and Ansible. Congrats to Lima JAM group.

Slides from the meetup can be foundhere. Additional shared resources used in the Lima JAM can be found here.

Check out where Jenkins Area Meetups (JAMs) are located in the world. Don't see a JAM in your area? why not start your own, here's how.

The first Jenkins meetup in Saint Petersburg, Russia took place on December 9th. The event has been organized with the help from Yandex and CloudBees.

In total there were about 80 attendees at the meetup. In addition tomeetup.com the event has been promoted by Yandex so we quickly reached capacity limit.

There were 3 talks conducted, speakers from Yandex, ZeroTurnaround and CloudBees. We discussed the current open-source project state, ongoing activities in the community, Jenkins-powered CD case studies from ZeroTurnaround and Jenkins plugin development approaches.

• Intro slides
• Who is Mr. Jenkins? Current State, common usage issues and trends in the community - video
• Jenkins beyond CI. ZeroTurnaround's experience - video
• When to write your own plugin and when not to - video
• Q&A Session video

Check out where Jenkins Area Meetups (JAMs) are located in the world. Don't see a JAM in your area? Why not start your own, here's how.

As I mentioned in yesterday's post, we're planning a "Contributor Summit" on February 1st, after FOSDEM 2016 (January 30th/31st), to focus on Jenkins 2.0. Since many of us are already planning, the Monday following the event turned out to be the ideal time to discuss 2.0.

Note: If you're not already familiar with some of the key proposals that were put forth, you can review them in the Jenkins 2.0 proposals summery page.

We've hosted one or two Contributor Summits in the past, and they're usually a day-long event where we try to gather a number of Jenkins core/plugin developers and active/power users to have detailed discussions around the theme of the summit. For this "Jenkins 2.0 Contributor Summit" we do not have a complete agenda yet, but we will post that to the Meetup event once it is fully prepared in the next couple weeks.

Suffice it to say, we'll be discussing a lot!

Venue and RSVP

The Contributor Summit will be hosted in a CloudBees office at: Rue des Colonies, 11, Brussels, Belgium. If you're already planning on attending FOSDEM, the office is near Grand Place and Cafe Delerium (where the Friday beer event is hosted).

The venue is of limited size, so if you're planning to join us, please RSVP to the Meetup event as soon as you're certain you will be able to attend. If you find yourself unable to attend, please remove yourself from the list to ensure that we can fit as many active contributors into the office as possible!

On December 15, the Toulouse JAM was co-hosted with the Toulouse JUG and Toulouse DevOps. Indeed it made sense since Jenkins is written in Java, makes use of Groovy code in many places (system groovy script, job dsl, workflow…), and it also made sense to co-organize with the local DevOps community since Jenkins is also a great tool to enable Continuous Integration, Continuous Delivery and automation in general. There were 103 RSVPs, with 80 to 90 people in attendance.

There were 3 talks planned for the evening:

• Job DSL Intro [fr], by Ghislain Mahieux
• Workflow plugin [fr], by Michaël Pailloncy (co-maintainer of the Build Trigger Badge plugin)
• Feedback on almost 10 years of CI and what's upcoming [fr], demo with Jenkins build scaling with Docker Swarm, by Baptiste Mathus

NOTE: presentations have been recorded (in french). They are still being processed, and once they are posted we will update this blog.

Photos can be found here