This past Thursday, September 24th, 2015, I presented a couple of prototypes of what I hope will be the future of the Jenkins GUI. Or perhaps more correctly, close enough to the future to start generating positive feedback from you the community that improving the Jenkins GUI is important and some pieces that I am showing are going in the right direction. If you have ~45 minutes to spare, I recommend the video (the narrator's voice is very soothing). If not, I offer the following as a reasonable summary.

Jenkins has a lot of strengths as tool. Its robust user community along with its thoughtful and extensible design are two of the most immediate. They are the two pillars that have made Jenkins the leader in the CD/CI space and the de facto choice for most of us looking to automate our build and test processes. But let's face it, by today's standards, the GUI doesn't really sing. I will even go so far as to say, I believe it is a platform liability at the moment, and even among we the Jenkins faithful, few of us look forward to using it.

In an effort to turn that tide, I traveled to this year's 3 main JUC events, in DC, London, and Santa Clara, pushing the idea that enhancement is possible and providing an evolving sketch of what that might look like. The three main areas of enhancement I have targeted for a first round of improvement are these:

• Item creation and configuration
• Plugin selection
• Workflow construction

Soon to follow, but not yet prototyped by me would be pieces dedicated to monitoring jobs in Jenkins as well as node and resource utilization and efficiency. Rightly or wrongly, I have started with the create and configure side of the GUI, as I see it as somewhat primary in a typical job creation scenario (you have to create a job before you can monitor it), but this second piece is no less important. Sadly, lips service is all I can offer you today, but more prototypes and video demonstrations are on the way.

Item Creation and Configuration in Jenkins

In most use cases, item creation means creating a freestyle job, so that is what I use as my base use case example. It is important to note, however, that most configuration in Jenkins happens through a shared set of GUI components. These components are a blend of Jelly files and Javascript and can be found in the .../main/resources/lib/form directory in the Jenkins source code. In operating on these pieces, I have the opportunity to effectively enhance broad areas of the Jenkins experience, including aspects of plugin use that share these components. This greatly increases the upside of the effort as well as the possible drama and side effects, which I will go into more detail on later.

As for the upside piece, the first bit of improvement I am looking to attain is breaking up the many 'toilet paper' style unbroken configuration lists sprinkled throughout Jenkins. The first example of this appears in item creation. On first installation, this issue is not immediately obvious, but if you have installed a variety of plugins or chosen to purchase CloudBee's Jenkins Enterprise product, you will find that Jenkins can have quite a few types of items to create. While they do have descriptive text, I still find them difficult to differentiate and almost impossible to casually scan. Thus, my first suggestion is to add some form of categorization to the item types. For this to function correctly, the GUI will need to be smart enough to apply the categories only when item counts are sufficient to justify them (if you only have 4 item creation types, it doesn't make sense to have 8 categories with which to sort them). But if you are a long time Jenkins user with many plugins you may also know it is possible to have more than a dozen item types. So if nothing else, an extension point that allowed for the categorization of item types seems helpful.

The configuration form itself, it also can become incredibly long with few landmarks or visual differentiation points. As a remedy, I propose calling out and clearly boxing each of the existing configuration sections and making sure that their names are as meaningful as possible. As an added step, I make the sections collapsible. This allows the user to jump to specific points in the form and tuck other areas out of the way. In some cases, we can make specific sections open by link context or even by user context.

Plugin Selection

Another essential piece of the Jenkins experience is plugin configuration. Today, if you are looking to add plugins to your Jenkins environment, you are almost certainly using Google to find a 3rd party review site, collecting the name of the plugin you want and then either linking to it on this website, or filtering for it in the Plugin Manager GUI.

Neither in the product nor on this website is there a particularly good resource for comparing plugins and evaluating which you might add.

Instead, I am looking to add something akin to an application store experience to both this website and the product UI. You should be able to group sort and compare plugins by a variety of criteria, including author, installation base, and user review. You should also have a set of general use categories that fits user needs and expectations, rather than the free ranging labels that plugin authors have arbitrarily applied today.

Workflow Script Builder

Finally, I have a GUI that allows for a sort of Drag-n-drop assembly of Workflows. A major tenant of the utility of Workflows as opposed to Freestyle jobs is that they can be completely separated from the Jenkins GUI and stored in a source repository. None-the-less, with absolutely no GUI, there is little to guide the user who is looking to get started without a upfront learning investment. As it turns out, a Workflow/Groovy script is pretty straight forward, but you don't really know that until you have made one. Also, Workflow allows for the orchestration of jobs across multiple nodes of hardware resources, making it a potentially involved little bit of configuration. Thus, my goal here is two fold. Allow the user to model a workflow quickly and easily and showcase a few of the more advanced features workflow enables. The result is this script builder. My hope is to host the prototype somewhere you all might be able to use it directly, but in the meantime, my hope is that my video pretty well explains how it works. Please take a look and post whatever comments you see fit.

...and really send along feedback...

So with all things community related, please, please, send back whatever feedback makes sense. I can be reached via Twitter @gusreiber.

Other places you can find me include, IRC (freenode/#jenkins) and Google+ (https://plus.google.com/+GusReiberUI). I would love to hear from you.

Questions and Answers from the talk:

1. How likely is it that any of these UI changes will make it into the core open source Jenkins? When would we start seeing them there?
   Most will be OSS. An exact schedule has not been determined, but most of it is still about a year away. Likely we will have an experimental wars for download along the way.
2. Is there anyway to determine which GUI attributes are contributed by which plugin?
   I take it that is a bit of a feature request? It came up at JUC West as well. Should be something that can be surfaced in the GUI. I agree, it would be helpful.
3. What is the difference between ANT and Jenkins?
   Ant is a good bit more bare-bones than Jenkins. In fact, you can add an Ant plugin to your Jenkins environment. You would typically use Ant to compile java source files. Jenkins orchestrates the fetching of the source files from some particular repository, the building of those files (often Jenkins uses Ant via its plugin to do this), running and reporting some suite of tests against that build, and then archiving or deploying the artifacts to wherever. Often times this requires navigating several computers with their own security constraints, so Jenkins helps you manage that as well.
4. What version of the Jenkins it is?
   This isn’t available today, but I am building against 1.621-SNAPSHOT currently, but will upgrade with Jenkins to the coming December LTS. I'm interested in seeing the list of 100 plugins that you mentioned (by Daniel?) Me too. :^) He and the community (which can be you if want to join IRC freenode.net/#jenkins and attend the hangouts and governance meetings: http://jenkins.ci.org )
5. For IRC, I assume the server is freenode.net?
   Yes.
6. Will there be any dashboard kind of feature for the build history in the new GUI?
   So far, I have been focusing on the create and configuration portion of the Jenkins UX as I see it as a barrier to entry for new users. The read/report/analyze half of the Jenkins UX I actually see as the portion with more long term value, as you tend to read more often than you write, so I am eager to jump in here as well. ....however, in its core today, Jenkins the tool seems to me to really want to see the world in the same context of flat XML files in folders as it actually persists its configuration data. To really make meaningful dashboards, it needs to be possible to query job configurations and build artifacts by a wide set of criteria that is not at all related to the folder in which the xml file happens to be stored. Also, some of the things you care about in the Jenkins universe are compute resources (masters/slaves/exactures). These are also not the same as config files in folders and need to be queryable as their own first class type of entity. ...so what I am saying with a lot of words is that I see the config piece as a somewhat more immediate and urgent fix. The broccoli of the meal, if you will. I will want to get that out as fast as possible to get it out of the way. The reporting piece is actually the wine. At the moment, we are giving you Bartles and Jaymes in paper cups. ...so a lot of work is still needed there.
7. Have you investigated Google Polymer as UI components for jenkins UI?
   I have not, but will now. I am actually quite a google fan-boy in much the way a lot of kids love Apple. (I also love Apple… being from Seattle, I even love MS). But, for the super near term, we are most focused on getting JQuery cleanly into core and Prototype.JS deprecated. Walk first, is my feeling.
8. Are there any tutorials on Jenkins workflow?
   Jesse Glick or KK are better people to ask about that, really. They are also on IRC: freenode.net/#jenkins. Daniel Beck as well, might be a good person to ask. My little workflow demo is still really just fiction. Will there be a 'Expand All' and 'Collapse All' buttons for the accordions in new configure GUI? (I would probably inject one if not by default) Yes. Also, they should be URL controllable so that they can be set by link or user context easily. Maybe they should also remember what you had open last? ...stuff to tinker with that really needs to be right.
9. What impact does the UI changes have on job configuration behind the scenes? Is configuration still stored in XML format?
   None. The post string stays the same and from then on, Jenkins is Jenkins.
10. Can the create item screen be configurable? At the moment, no, but ideally yes. It is still a big hand wave at the moment about how those categories are created, managed, and updated. The same categories ought to bubble back up when searching for the plugins to help relate what plugins generate what UI. I am hoping for guidance from the community. How will workflow fit in with new UI?
    In some respects, the new configuration page is about enhancing the more traditional freestyle job and not workflow. However, the last bit of my presentation with the script builder is exclusively about workflow. The plugin manager is about plugins, so it would apply to both.
11. How is a human notified for the wait for approval step in this workflow?
    So workflow approval can be done via the web GUI. But to get real notification, you would program that into your workflow Jenkins has a fairly large set of notification plugins. So you can use Jenkins to trigger email, or SMS, or HipChat, or Slack, or pretty much whatever. As these plugins are increasingly customized for workflow, you will get nice and nice workflow syntax for instantiating those actions. When my script builder is adopted, you would have a friendly button you could drag into the stage and it would notify you prior to the manual checkpoint.
12. Custom plugins still supported?
    Yes. Though there is supported and supported. The highest level of support for a plugin would be a custom DSL for workflow that would make for streamlined syntax in workflow for interacting with that plugin via Groovy. But existing plugins do not need that level of support to be used within a Jenkins file / Groovy script. Instead, the syntax for accessing the plugin is likely to be more complicated. ….some plugins are freestyle specific, in which case, they no longer make sense in workflow. ….Daniel Beck or Jesse Glick are probably better suited to answering this question, however...
13. Will there be an improvement in performance with docker builds, sonar scanning? From my experience sonar takes 20+ mins with jenkins plugin where as it takes 3 mins with maven plugin
    Is this times it is taking the GUI to render, or the actual build to run? I am not sure I am following the question exactly, but regardless, I am not well equipped to answer many questions about performance issues in Jenkins. I know of a fairly major performance issue specifically in the configuration form that I believe will be fixed in the new GUI, but that isn’t build performance, it is just form rendering performance.
14. I like the graphical configuration. Thanks. The scripting of a complex workflow looked a bit daunting.
    Cool. Yeah, my main and first goal is to get something out there that would allow folks to quickly sketch and deploy an actual working workflow that reasonably reflects an 80%ish use case. No GUI can ever be as fully flexible as a script, but I don’t think most people need the 95% case to get started and see the benefit of a versionable and robust config file format.
15. Will there be any effort to make the UI mobile friendly for the admin on the go?
    Absolutely. Especially on the TBD read/reporting end of the UI, but everything new needs to meet a reasonably high bar of device responsiveness. Today, the Jenkins GUI is just not responsive. Which is terrible.
16. As a plugin developer do I need to change implementing the ui source from jelly or groovy to some other language/technique or will it be compatible?
    So you will not NEED to change from whatever you are doing, except if you have built a plugin GUI that has custom script that either relies directly on behavior.js, hudson-behavior.js, or the particulars of the existing DOM structure (you do something in the client that requires your or some other input to be in a particular TABLE TR TD DOM traversal path). ...I believe 2 things are going to continue to happen at a faster and faster rate. New plugin authors are not going to want to write GUIs in Jelly and Prototype.js, but instead use some more modern client side MVC approaches like Angular, where the GUI interacts with a REST api instead of being a dom directly rendered from the server. It is a bit of a different mode of working than Jelly, and maybe slightly less direct, but it is a lot easier to find doc on how to do things with JQuery, Agile, Handlebars and the like, than it is to find doc on Jelly. And the responsiveness and breadth of gestures and controls in Jelly are already terribly behind what is now the main stream of web UI development. So I think plugin builders are, if they aren’t already, going to want better tools available to them. I also think that people are going to gravitate towards workflow or something similar. Since the UI for workflow is foremost a script, making a GUI for a plugin that works with it might be a fundamentally different beast. ...depending on what the plugin is trying to do… So again, new plugins or even upgrading existing plugins to work with workflow are likely want a new technology set, not just because the existing Jenkins GUI is changing, but because new plugins will want to do different and better stuff.
17. Are there connectors for other source control tools like CVS and Dimensions?
    I am not sure exactly which connector plugins are already supporting Workflow or how deeply that support goes. Because Jenkins has plugins that provide access to these SCMs, you can use workflow to go and fetch those source trees. A greater level of support for workflow from these plugins would mean a more elegant workflow syntax for that interaction. At the moment, my GUI script builder is still fiction. My plan would be to add GUI buttons for whatever are the most popular SCMs and I will attempt to mask the syntax regardless of its clumsiness. ….the way I am constructing my initial prototype, there is already a reasonably clear extension point for adding buttons that generate some chunk of Groovy syntax when it is dragged into a stage. So I will add the initial set based on community feedback and then the community can continue to add their own.
18. What are the compatibility issues existing plugin developers needs to be aware of?
    For plugins that interact with freestyle jobs, or really most job types that aren’t workflow, plugin developers should expect the page DOM structure to change. If for whatever reason, they find they are busting into some custom script to traverse the DOM to compare 1 setting to another, that will break. Also, hudson-behaviors.js itself has a number of functions in it that do DOM traversing, like “findFollowingTR”. In some cases the signatures of those functions might need to change and the DOM structure that they return might also change. If a plugin uses what were meant to have been internal functions, they are likely to break. Finally, the page geometry is going to change. This may seem so superficial and obvious that, who cares, but sometimes changing a column width translates into an important part of a GUI being hidden or otherwise inaccessible. That ends up being as critical a break as any other. ...so to combat these points of possible breakage, we are going to be looking for a handful somewhere between 20 and 100 plugins that we will want to test against. We haven’t made that list yet, let alone run any tests, so that is really a critical next step. For the plugin manager changes, I don’t see much if any of a braking issue, although I would like to add additional sorting and display power to the GUI, which means the GUI will need more metadata than currently exists, if the plugins want to take advantage of that new power in the GUI. This won’t break things, but plugin authors might want to go back to their plugins and fill in whatever the new bits of metadata end up being…. most likely they would be things like, richer descriptions, better category selections, and possibly icons.
19. I've not seen a lot of Jenkins but what I had I didn't really get, was awkward for all the reasons Gus mentioned. This looks brilliant. When can we have it?
    Tom and I, and now our junior pledge, Keith (not actually junior at all, just more fit than me), are busily typing as fast as we can as well as lobbying the community that our vision is more or less a correct one. We have a very interesting initial plugin selection GUI that might make this years final LTS (which I did not demo), which is none-the-less a nice step forward for Jenkins. In it will be a lot of the JS library bundling that will enable most of what I have shown in this demo. Our hope is that with each LTS we will be able to push out an additional piece of the GUI puzzle. Likely starting with the job create and configure GUI, which would be the mid year LTS. I am hoping that a year from now this will be how Jenkins looks and acts. ….in the meantime, we are grappling with how best to push preview releases so people can play with it and send me hate mail.
20. Is there any way to test front end of Jenkins plugins? And will that improve too?
    A major and almost blocking portion of this work used to be the custom and somewhat broken version of HTMLUnit that was in core, which greatly hampered including libraries other than Prototype in Jenkins and writing code using those libraries in some sort of testable way. Our new approach to rebuilding the Jelly controls which are the foundation of the Jenkins config page and in general are shared by all plugins that need to post data back to Jenkins, already have a testing strategy backed into our design. Those Jelly form controls are extensible in Jenkins today and would remain so. Our hope would be that any plugin adding custom controls would follow our same design and test pattern we are building in core. ….so that was a long answer, but the short answer YES! Today, building GUI parts into your Jenkins plugin is a bit of a mystery, where most people copy something they saw someone else did, hack it, and the only test is, well…. it worked for me. That is no good and a fundamental piece we are looking to change. ….still a long answer… Node.js and Jasmine are the specific tools we using.
21. What's the estimated rollout date for this workflow feature?
    The workflow feature is the newest concept I demonstrated, but in a lot of ways may also be the easiest to ship. As a script generator, exclusively, it could be hosted anywhere, and then you just paste your generated workflow script into the whatever existing Jenkins GUI better, submit into your source code. ….but at the moment, it isn’t actually on an official roadmap yet. Assuming the response to it remain positive, I would expect that to change fairly quickly.

Last week, the first Jenkins Area Meetup (JAM) took place in San Jose, CA on Wednesday, Sept 23. What a way to kick off the first JAM other than to have Docker, John Willis as our guest speaker. John talked about immutable infrastructure and its benefits and role of containers.

Kohsuke discussed Jenkins Workflow, the motivation behind the same and latest features of Jenkins Workflow like multi branch support followed by docker use cases. The highlight of the meetup was definitely Kohsuke breaking the news about Jenkins 2.0 and his vision and motivation behind it.

The next Bay Area JAM is slated for Oct 21. Be sure to check HERE for the agenda. We’d love to have you join us if you’re in the area. If you’re interested in speaking, or become a food & bev, venue, or recording sponsor please send email to the organizer or events@lists.jenkins-ci.org.

I hope many of you have had a chance to see the Jenkins 2.0 thread. I'm going to use the office hours next Wednesday to go through this proposal.

This is still primarily for developers in the project, as it's "just" a proposal with lots of details unspecified. It's more meant to help people understand where I'm coming from and what goals I have in mind for this effort.

As always, this will be on Hangout on air. The event page is here, and if you want to participate in the discussion, join here. Read-only viewers should use YouTube to watch, and you can still send questions in real time to IRC and I'll make sure to go through them.

Over 2,000 members of the Docker community attended Docker Hack Day events around the world. One of the forty-two Docker Hacks has some familiar names attached...

Nicolas De Loof and Yoann Dubreuil from Docker Rennes, who are also active in our community, waved the Jenkins flag in this event and produced Jenkins docker slaves plugin.

This plugin lets you run builds inside containers, and in that sense it's similar to the Docker plugin and the Docker custom build environment plugin. But internally it uses a quite interesting approach.

This fresh new implementation relies on a set of docker containers (aka ‘pod’) to setup a build executor, letting development team customize the build environment for their need without any constraint or prerequisite, and relying on docker containers to host test resources.

This project won the 3rd place in the Freestyle category of Docker Hack Day. Congratulations to Nicolas and Yoann on their win! Jenkins + Docker is a winning pair and this plugin will make a huge difference in your projects.

There's been some active discussions and planning around Jenkins Area Meetups (JAMs) specifically in the following cities:

• Rennes, France
• Seville, Spain
• Seattle, Washington
• Barcelona,Spain
• Raleigh, North Carolina
• Atlanta, Georgia

I wanted to gauge Jenkins interests in these cities, so let us know at jenkinsci-jam@googlegroups.com if you live in one of these areas, and if you would be interested in becoming a member or be involved in JAM one way or another!

Of course, if the city you live in currently does not have a JAM and you're interested in paying it forward, here's HOW YOU can become a JAM organizer.

For the past several years we've been attending FOSDEM, a massive free and open source event in Brussels, Belgium. In preparation for this upcoming FOSDEM (2016) event, we will be hosting an open planning meeting via Google Hangouts during this week's "Office Hours."

Agenda:

• Gauge who can participate and at what capacity.
• Pre-FOSDEM Contributor Summit
• After-hours meetup/happy hour
• Plans for a Jenkins stand (assuming we're accepted):
  • What demo materials
  • Schwag
  • Revised Jenkins Flyer

The FOSDEM 2016 wiki page is where we will be recording plans and tasks will be added to JIRA. If you cannot join us via the FOSDEM Office Hours Hangout, we will also be watching the #jenkins-community channel on the Freenode network if you cannot participate directly.

Please join us on this Google Hangout at 11:00 a.m. PDT this Wednesday (Oct 21)

Over the past few weeks there has been a vibrant discussion happening on thejenkinsci-dev@ mailing list as to what "Jenkins 2.0" means. While Jenkins does not currently adhere to semantic versioning, the change of a major version number does indicate a major milestone for the community.

Project founder, Kohsuke Kawaguchi presented his vision for Jenkins 2.0 in a office hours session, the slides for which can be found in this Google Presentation. Roughly paraphrasing Kohsuke's vision, 2.0 is primarily about making things better for the thousands of users out there.

This week, we'll be reviewing some key areas of the "Jenkins 2.0" proposal. Asking you, the user community, to provide feedback on these proposals, going from Jenkins internals to user interface.

Today's post involves a proposal to introduce a policy for API deprecation from community members Oliver Gondža and Daniel Beck. Extensibility is the heart of Jenkins, but over the past ten years we've not had a proper API deprecation policy other than "try not to break plugins, ever."

Daniel, expanding more on the problem wrote:

We have no backward compatibility policy besides "compatibility matters". With 1000+ plugins and basically the entire core being available to plugins, a lot of difficult or impossible to remove cruft has accumulated over the last ten years. This limits both what can be changed in core, and makes documentation difficult to use for plugin developers.

The two have put together a detailed proposal underJENKINS-31035 which suggests we:

limit the availability in APIs (classes, methods, fields, …) provided by core to a number of releases. Depending on the feature, this can range from a few months, to a few years (e.g. two years being about 100 releases of Jenkins and eight LTS baselines).

[...]

I highly encourage you to read the entire proposal on the issue tracker, where we are trying to collect feedback/history.

Providing Feedback

We're asking you to read the proposal inJENKINS-31035 and provide feedback if you have it.

If you have ever logged in to the issue tracker or thewiki, you have a "Jenkins user account" which means you'll be able to log into the issue tracker and vote for, or comment on the issue linked above.

(note: if you have forgotten your password, use the account app to reset it.)

We're going to review feedback, make any necessary adjustments and either approve or reject the proposal two weeks from today.

Stay tuned, and help make Jenkins 2.0 great!

As I mentioned in yesterday's post, there's been a lot of discussion recently about what "Jenkins 2.0" means. In a recent "Office Hours" session, Kohsuke Kawaguchi presented his vision for Jenkins 2.0 in a office hours session, the slides for which can be found in this Google Presentation. Roughly paraphrasing Kohsuke's vision, 2.0 is primarily about making things better for the thousands of users out there.

This week, we'll be reviewing some key areas of the "Jenkins 2.0" proposal. Asking you, the user community, to provide feedback on these proposals, going from Jenkins internals to user interface.

Thus far we've covered:

• Introducing a policy for API deprecation

Today's post involves a proposal originally from community member Jesse Glick who has proposed in JENKINS-29068 that Groovy be split out from the "core" Jenkins distribution. The linked issue expands on what the problem is here:

Currently Jenkins embeds a distribution of Groovy into "core" for a variety of scripting and management tasks. This version of Groovy is locked into core in such a way that users cannot upgrade Groovy independently from Jenkins itself. If the Jenkins-bundled version were upgraded to a different major version, it may break users' custom scripts as well as plugins that use Groovy due to API changes.

The proposal is relatively straight-forward and affects the many different users and use-cases for the embedded Groovy scripting support in Jenkins:

For ease of maintenance and modularity it would be useful to split Jenkins' use of Groovy into a library plugin; different clients could request 1.x and 2.x simultaneously by using different versions of the library, etc.

Stuff in core using Groovy that would need to either be put in this library (if infrastructure for other features) or put in another plugin depending on it (if first-class features themselves):

I selected this proposal to feature on this blog, despite its rather technical underpinnings, it will affect core developers, plugin developers, power and casual users alike. I encourage everybody to read through the proposal and its potential impact on the issue tracker.

Providing Feedback

We're asking you to read the proposal inJENKINS-29068 and provide feedback if you have it.

If you have ever logged in to the issue tracker or thewiki, you have a "Jenkins user account" which means you'll be able to log into the issue tracker and vote for, or comment on the issue linked above.

(note: if you have forgotten your password, use the account app to reset it.)

We're going to review feedback, make any necessary adjustments and either approve or reject the proposal two weeks from today.

Stay tuned for the rest of the week as we keep with our theme of going "from the inside out" and help us make Jenkins 2.0 great!

We have been featuring a few proposals this week for what "Jenkins 2.0" is going to include, today we're discussing my personal favorite, which I believe will have a tremendously positive impact for years to come (not to be too biased!): moving the "Pipeline as Code" support in Jenkins to the front and center.

Thus far in this blog series we have reviewed proposals covering:

• Introducing a policy for API deprecation
• Splitting Groovy out of 'core'

Today's proposal comes from project founder Kohsuke Kawaguchi titled "Pipeline as code front and center" and represents perhaps the most important and dramatic shift we hope to make in Jenkins 2.0.

This functionality has existed through the workflow plugin, which we have discussed at various Jenkins events before but if you're not aware of some of the power behind it, check out this presentation from Jesse Glick:

The proposal in JENKINS-31152 expands on the problem we aim to address:

The default interaction model with Jenkins has been very web UI driven, requiring users to manually create jobs, then manually fill in the details through a web browser. This requires large amounts of effort to create and manage jobs to test and build multiple projects and keeps the actual configuration of a job to build/test/deploy a project separate from the actual code being built/tested/deployed. This prevents users from applying their existing CI/CD best practices to the job configurations themselves.

To address this, Kohsuke is proposing that we :

Introduce a new subsystem in Jenkins that:

• lets you design a whole pipeline, not just a single linear set of tasks
• stores the said pipeline configuration as human-editable Jenkinsfile in your SCM
• makes it automatic to set up new pipelines when Jenkinsfile is added
• differentiates multiple branches in the same repository

This is the key new feature that positions Jenkins for continuous delivery use cases and other more complex automations of today.

Kohsuke's proposal is largely about bringing together a lot of already existing pieces together to provide a very compelling experience for new and existing users alike. I hope it is clear now why this proposal is so exciting to me.

Providing Feedback

We're asking you to read the proposal inJENKINS-31152, which itself have some additional tickets linked under it, and provide feedback if you have it.

If you have ever logged in to the issue tracker or thewiki, you have a "Jenkins user account" which means you'll be able to log into the issue tracker and vote for, or comment on the issue linked above.

(note: if you have forgotten your password, use the account app to reset it.)

We're going to review feedback, make any necessary adjustments and either approve or reject the proposal two weeks from today.

Stay tuned for a couple more posts covering proposals to improve the Jenkins interface and user experience!

We have been featuring a few proposals this week for what "Jenkins 2.0" is going to include. Today we'll be diving into the most noticeable changes being proposed for Jenkins 2.0: the User Experience (UX) improvements

Thus far in this blog series we have reviewed proposals covering:

• Introducing a policy for API deprecation
• Splitting Groovy out of 'core'
• Pipeline as code front and center

The UX improvements being proposed aren't necessarily as uniform as the proposals from earlier in the week but represent a large amount of prototype and exploratory work done by folks like Tom Fennelly, Gus Reiber and a few others. Those following the dev list may have already seen some of these proposals in some of the "mega threads" that we have had discussing potential UI/UX improvements previously.

The improvements proposed for 2.0 can be found under JENKINS-31156. The most promising proposal under this issue is to update the plugin manager experience.

Another very important proposal for 2.0 worth mentioning is the proposal to update UI work well on mobile devices.

Providing Feedback

We're asking you to read the issues linked from JENKINS-31156 and comment and vote on those issues accordingly.

If you have ever logged in to the issue tracker or thewiki, you have a "Jenkins user account" which means you'll be able to log into the issue tracker and vote for, or comment on the issue linked above.

(note: if you have forgotten your password, use the account app to reset it.)

We're going to review feedback, make any necessary adjustments and either approve or reject the proposal two weeks from today.

Stay tuned for tomorrow's post covering the remainder of the proposed user experience changes!

This week we have featured a number of proposals for what we would like to see in "Jenkins 2.0", the vision of which is to make Jenkins users more efficient, productive and happy. We started with some more internally facing changes and have slowly progressed from the "inside-out" to today's topic: improving the out of the box user experience. That is to say, the experience that a brand-new Jenkins user has when getting started with the server.

Just to recap, so far we've reviewed:

• Introducing a policy for API deprecation
• Splitting Groovy out of 'core'
• Pipeline as code front and center
• User experience improvements (part one)

The subject of today's proposal is captured in JENKINS-31157, which, like yesterday's proposal, contains a few issues linked from it with more details.

At a high level, the problem aiming to be solved is:

When a new user installs Jenkins, they are greeted with the main, empty, dashboard which suggests that they "create jobs." This makes no mention of plugins or the configuration options that are relevant to helping the user make Jenkins match their needs.

In past and current versions of Jenkins, if you know what you're looking for it's relatively easy to move around the interface. If you've never used Jenkins before, it can be very challenging to find your way around or even know what it is possible to do with Jenkins.

The proposed changes aim to address this initial confusion:

Instead of changing the post-install defaults, which may not properly represent the user's needs, the first-time user experience should help guide the user through configuration and plugin installation quickly so they can use Jenkins for their needs. Effectively it should be as easy as possible for a user to arrive at a good configuration for their usage.

Jenkins contributor Tom Fennelly, who has led this discussion on the mailing lists in the past, has posted a good prototype screencast of what some of this might entail:

Providing Feedback

We're asking you to read the issues linked from JENKINS-31157 and comment and vote on those issues accordingly.

If you have ever logged in to the issue tracker or thewiki, you have a "Jenkins user account" which means you'll be able to log into the issue tracker and vote for, or comment on the issue linked above.

(note: if you have forgotten your password, use the account app to reset it.)

We're going to review feedback, make any necessary adjustments and either approve or reject the proposal two weeks from today.

This concludes this week's blog series highlighting some of the Jenkins 2.0 proposals we felt were important to discuss with the broader Jenkins user audience. Many of these, and other minor proposals, can be found on the Jenkins 2.0 wiki page.

With more than a thousand public plugins in the Jenkins community now, it should come as no surprise that some of them are no longer actively maintained. Plugin authors move on when they change jobs, or lose interest in the plugin, and that's fine. Plugins are hosted on the Jenkins project infrastructure after all, and when a maintainer moves on, others can continue their work.

The major problem of course is that it's often difficult to tell whether a plugin is still maintained (and there's just not a lot going on), or whether its maintainer has lost interest. Most plugins don't get released every few weeks, or even every few months, and still do their job just fine.

To connect plugins that aren't actively maintained with potential maintainers, we recently implemented the "Adopt-a-plugin" initiative: We built a list of plugins that are up for "adoption", and display a prominent message on the plugins' wiki pages. Anyone interested in taking over as a plugin maintainer can then contact us, and we'll set you up.

Are you interested in becoming a plugin maintainer? Maybe one of your favorite plugins isn't actively maintained right now. Check out the Adopt a Plugin wiki page for more details on this program, and a list of plugins that would benefit from your help.

Preceding some of last week's Jenkins 2.0 discussions, there had been some threads on whether we should move Jenkins to require Java 8. The introduction of Java 8 last year brought performance improvements and highly desirable API changes, which make developing Java-based applications (arguably) much easier than before. The release was followed earlier this year by the end-of-life announcement for Java 7; the writing is on the wall: upgrade to Java 8.

I wanted to answer the question "does it even make sense to force an upgrade to Java 8?" There are plenty of technical discussions that we can have in the community on whether or not this is the right approach, but my goal was to try and measure the current Jenkins install base for Java 8 preparedness.

While we do not currently (to my knowledge) collect Java runtime versions in our anonymous usage statistics, we do have access logs from our mirror redirector which might provide some insight.

With some access logs data, I went through the millions of requests made to Jenkins infrastructure in 2015 and started filtering out the user agent which made those requests.

NOTE: This data is totally not scientific and is only meant to provide a coarse insight into what versions of Java access Jenkins web infrastructure.

When Jenkins hits the mirror network, it's not overriding the default user agent from the Java runtime, so many of the user agents for the HTTP request are something like Java/1.7.0_75. This indicates that the request came from a Java Runtime version 1.7.0 (update 75).

Looking at the major JVM versions making (non-unique) requests to Jenkins infrastructure we have:

• 1.8.0: 21,278,960
• 1.7.0: 27,340,214
• 1.6.0: 4,148,833

This breaks down across various updates as well, which is also particularly interesting to me because many of these Java versions have long since had security advisories posted against them.

As I mentioned before, this is not a rigorous analysis of the access log data and is also not filtered by unique IP addresses. What I found most interesting though is that the Java 8 upgrade numbers are actually fairly strong, which I didn't expect. I expect that piece of the pie will continue to grow. Hopefully so much so that we're able to move over to Java 8 before the end of 2016!

It is great to see the pick up of local activities through hosted JAMs. In October, the Jenkins community hosted Atlanta JAM and Bay Area JAM. Many thanks to our sponsors: Ericsson, CloudBees, Blazemeter, NetRoadShow.

Here's a summary of what was discussed:

• Atlanta JAM - Jenkins workflow and Docker to reduce friction in DevOps efforts.
• Bay Area JAM- Performance testing strategies, incorporating performance tests into Jenkins workflows and the metrics that matter most for troubleshooting and diagnosing issues.

A look forward to November and December:

• Hacksgiving
• Barcelona, Spain JAM
• Toulouse,France JAM

As usual, if you're interested in becoming an organizer or sponsor, here's how to get started. If you've heard of a great Jenkins talk out there, shoot us an email with speaker info to jenkinsci-jam@googlegroups.com so we can invite him/her to our next meetups.

Updated 2015-11-06 03:55 UTC: Included a updated mitigation script which doesn't have a Jenkins boot race condition

Earlier today we received numerous reports about a previously undisclosed "zero day" critical remote code execution vulnerability and exploit in Jenkins core. Unfortunately the vulnerability was not disclosed to us ahead of its publication so we're still working on more thorough fix. In the meantime however, we wanted to inform you of the issue and provide a workaround which will help prevent this exploit from being used against public Jenkins installations, for future reference this issue is being tracked privately as SECURITY-218 in our issue tracker.

The attack is mounted through the Jenkins CLI subsystem, so the work-around is to remove/disable the CLI support inside of the running Jenkins server.

Using the following Groovy script you can disable the attack vector in your Jenkins installations by navigating to “Manage Jenkins” and then to “Script Console”, or just go to http://your-jenkins-installation/script. This only addresses the current running Jenkins process, in order to make the workaround persist between restarts of the Jenkins server, add the script below to $JENKINS_HOME/init.groovy.d/cli-shutdown.groovy (create the directory if necessary, and the file).

import jenkins.*;
import jenkins.model.*;
import hudson.model.*;

// disabled CLI access over TCP listener (separate port)
def p = AgentProtocol.all()
p.each { x ->
  if (x.name.contains("CLI")) p.remove(x)
}

// disable CLI access over /cli URL
def removal = { lst ->
  lst.each { x -> if (x.getClass().name.contains("CLIAction")) lst.remove(x) }
}
def j = Jenkins.instance;
removal(j.getExtensionList(RootAction.class))
removal(j.actions)

in order to make the workaround persist between restarts of the Jenkins server, add the script below to $JENKINS_HOME/init.groovy.d/cli-shutdown.groovy (create the directory if necessary, and the file).

The latest version of this script can be found in this GitHub repository.

As previously announced on the jenkinsci-advisories mailing list we’re preparing a security release for this upcoming Wednesday which will include patches for both the latest and LTS lines of Jenkins core. The Jenkins Security team is working to include a fix for this previously undisclosed exploit in or before this planned security release.

If you have questions about this exploit, join us in the #jenkins channel on Freenode or ask on the jenkinsci-users@ mailing list.

For security researchers and hobbyists, if you believe you have found a security vulnerability in Jenkins, we have some disclosure guidelines on this wiki page which will help us mitigate any discovered issues as quickly and safely as possible.

Be sure to subscribe to the jenkinsci-advisories mailing list, it's the fastest way to get updates by the Jenkins Security team.

We just released Jenkins 1.638 and 1.625.2 which contain important security fixes, including a fix for the zero-day vulnerability published on Friday. Please see the security advisory for more information.

Want to be kept up to date on Jenkins security releases, including advance notice on scheduled security updates? Subscribe to the jenkinsci-advisories mailing list!

Next week in the US we have a national holiday where, generally speaking, lots of turkey gets converted into left-over turkey sandwiches. For many software developers the Thanksgiving holiday also represents a lull in project schedules, freeing up some time to hack on pet projects or even contribute to open source projects.

Taking a cue from the Adopt a Plugin program that Daniel wrote about earlier this month, we thought it would be fun to organize a "virtual hackathon" to coincide with that gap in project schedules. Thus Hacksgiving 2015 was created!

We'll be hosting Hacksgiving Nov 23rd and Nov 24 from 7:00PST - 15:00PST (10:00EST - 18:00EST) and would love for you to join! (RSVP here)

You don't need to know Java to help! We will have documentation and design hacking going on as well.

We have a few goals for Hacksgiving:

1. Introduce new contributors to the process of writing code and/or documentation (documentation hacking details here).
2. Find some plugins which are up for adoption new maintainers.
3. Clean up or merge some existing plugins which need some care (listed here).

Sessions to note

Here are some of the sessions scheduled that will be hosted by members of community that may interest you:

Day One

• 7:00PST/10:00EST (15-30min) - rtyler will host a welcome and introduction to contribution to the Jenkins project (walking through our contributors guide)
• 10:00PST/13:00EST (60min) - schristou will host a workshop titled "Introduction to plugin development for Jenkins"

Day Two

• 10:00PST/13:00EST (60min) - abayer will be hosting a "Plugin Developer Open Q&A" session, so bring your questions!

Hacksgiving is very unconference structured right now, so if you're interested in hosting a session please let us know in the #jenkins-community channel or by signing up for a session on the schedule

How to participate

Since this is a virtual hackathon, we'll be congregating and chatting in a couple of ways:

• On the #jenkinsIRC channel as per usual
• We'll be hosting sessions and tutorials via Google Hangouts, see the "hacker hangout* section on the wiki page up to date details
• Via the #hacksgiving hashtag on Twitter

You can also RSVP on our meetup page!

We hope you can join in the festivities!

Last week we hosted our first Hacksgiving event, a two-day virtual hackathon with a number of recorded sessions and plenty of pull requests submitted, I would say it was a success! I would like to thank everybody who took the time to watch, chat and present in the Hacker Hangout.

Now that everybody has had time to recover from the turkey and travel, we have some videos of the sessions sliced out and ready for publication.

In addition to the recorded sessions, there were a number of notes captured with useful links associated with practically each session. You can find those notes at the bottom of the Hacksgiving page.

Note: The following videos are all available in this YouTube playlist

Intro to the Jenkins project

This session was hosted by rtyler and meant to provide a cursory overview of where to get started with contributing to the Jenkins project

Intro to Plugin Development Workshop

This session was given both days of Hacksgiving by schristou and does a really great job of introducing the viewer to getting started with developing a Jenkins plugin with Java.

Workflow Q&A and Demo Session

This session was not originally scheduled, but some folks on the Jenkins IRC channel had some Workflow questions and Jesse Glick jumped into the Hacker Hangout to help us out!

Internationalization Live Coding / Q&A

Another impromptu session, this time with danielbeck hosting. In this session Daniel walks through a plugin he was working on for Hacksgiving and adds internationalization support while answering a few questions here and there.

Intro to the new static site

Kicking off day two of Hacksgiving, rtyler hosted a session on the new statically-generated Jenkins site. The new site will dramatically lower the barrier to entry for contribution to Jenkins documentation and blogs, by pushing everything through GitHub.

Plugin Developer Open Q&A

This was the last session of Hacksgiving, hosted by abayer and ended up being more like a casual discussion of the current status and future work in the plugin development ecosystem.

Note: This is a guest post by Kishore Bhatia. Kishore works for CloudBees, building custom frameworks with Open Source software and helping customers solve engineering problems around continuous delivery and DevOps at scale.

This year some great new Jenkins features came out of the butler’s goodie bag - amongst them, the most important one being the ability to realize continuous delivery pipeline as code! The features like Workflow Multibranch, pipeline-as-code (with a marker file that Jenkins looks for in your application’s SCM repository/branch, aptly named Jenkinsfile) are the foundations to making Jenkins super intelligent to automagically create workflows (rather, a CI/CD pipeline) to build your code and orchestrate the work required to drive your application from concept to delivery!

Overview

The Workflow Multibranch feature (provided by the workflow plugin) provides the following key abilities:

• Automatic Workflow (job) creation in Jenkins per new branch in the repo.
• Build specific to that child-branch and its unique scm change and build history.
• Automatic job pruning/deletion for branches deleted from the repository, according to the settings.
• Flexibility to individually configure branch properties, by overriding the parent properties, if required.

Jenkins pipeline-as-code (concept) enables you to maintain your CI/CD workflow logic in the project/application source code repo with no additional configuration to be maintained per branch in Jenkins.

The Workflow script to build/test/deploy your code is always synchronized with the rest of the source code you are working on.

To demonstrate the concept here - Let’s use a basic Java Web application project with a Maven pom.xml as shown in the structure below (this is using GitHub as the SCM but you can do this on SVN or Mercurial too).

This project has a marker file for Jenkins in the repo - Jenkinsfile.

So, what's a Jenkinsfile? The Jenkinsfile is essentially your Jenkins Workflow, a script, that defines the CI/CD pipeline logic for a project with steps to build/test/deploy etc. captured in various stages.

So for our sample Java web application, a basic Jenkinsfile could be something like -

node {
   // Mark the code checkout 'stage'....
   stage 'Checkout'

   // Checkout code from repository
   checkout scm

   // Get the maven tool.
   // ** NOTE: This 'M3' maven tool must be configured
   // **       in the global configuration.           
   def mvnHome = tool 'M3'

   // Mark the code build 'stage'....
   stage 'Build'
   // Run the maven build
   sh "${mvnHome}/bin/mvn clean install"
}

Just having this file in the source code repo root would mean that -

• Jenkins will automatically recognize this branch and create appropriate jobs by itself.
• Quick, 1-step code checkout using: “checkout scm” in your workflow
• Every time a new change is pushed to this branch, the branch is built and the commit status gets updated.
• When the branch is destroyed in the repository, or if Jenkinsfile is removed, the corresponding job gets destroyed from Jenkins automatically (You can retain these jobs and/or archive the builds for audit/compliance requirements using the retention property - Orphan Item strategy)

Note: there are various mechanisms to promote reuse of Workflow scripts, such as the Workflow Global Library.

Required Jenkins configuration

Make sure you’ve the latest Workflow and (v1.11 as of writing this blog) Workflow Multibranch plugins installed on your Jenkins instance

Also, ensure that other dependencies, like SCM plugins and build tools, are met:

• Either SVN/Git/Mercurial (depending on your SCM)
• GitHub Branch Source Plugin (optimized to use the GitHub API and improve performance)
• Maven build tool

Then create a new Multibranch Workflow Job with configuration as shown below - mainly selecting the Branch Sources (Git, in this example) and providing the branch/repo URL with credentials.

Branch sources (Git) - https://github.com/kishorebhatia/pipeline-as-code-demo (or a repo where you’ve cloned this source code with Jenkinsfile)

Leave all other properties default and Save.

You’ll observe that Jenkins would perform Branch Indexing on that “cd” job folder and start the workflow for the master branch, with an automatically created new job, named master, under the “cd” folder.

The workflow does a dummy step for application deploys to the environments in this sequence Staging -> Waits for manual approval ->PROD

Now, let’s create a new branch off of this master branch in your cloned git repo:

• $ git branch newBranch (create a newBranch)
• $ git checkout newBranch (switches to newBranch)
• $ git push --set-upstream origin newBranch (pushes newBranch)

You’ll observe that your Jenkins instance automatically picks up this newBranch and starts running the workflow (with the Jenkinsfile in this newBranch) to build/test/deploy the code.

Next, if you now delete this newBranch (git branch -D newBranch), Jenkins will automatically remove the orphan Workflow job for newBranch. You can retain these jobs even after the branches are deleted using the Orphaned Item Strategy property in the main "cd" job’s configuration.

So we observed the following benefits of this pipeline-as-code approach:

• Overall job definition is a script (Jenkinsfile)
  • Calls your build tools and scripts for details
• The build script can be versioned alongside project sources
  • Jenkins handles feature/experimental branches automatically
• Keep less configuration in $JENKINS_HOME

Dockerized Demo environment

You can also use the following docker image to run this demo with a preconfigured Jenkins environment and the sample job: jenkinsci/workflow-demo (i.e. docker pull jenkinsci/workflow-demo)

This docker container includes Jenkins with Workflow and Workflow Multibranch plugins, a local git repo with the aforementioned Java web application and Jetty to demonstrate a continuous delivery pipeline of this application deployed and tested across multiple environments in the pipeline with an approval gate before promoting to PROD (like QA, Staging and PROD).

There's a "cd" job pre-configured as a multibranch Workflow job.

Launch the docker demo as: docker run -p 8080:8080 -p 8081:8081 -p 9418:9418 -ti jenkinsci/workflow-demo

Now, you can access Jenkins on port 8080 and Jetty on port 8081 from localhost or the IP of your boot2docker/docker-machine environment.

The demo container has a local git repo so you can clone: git://localhost/repo. When creating new branches, each branch automatically creates a matching subproject in Jenkins and triggers the build for that branch. The workflow:

• Checks out source code from the same repository and commit as Jenkinsfile.
• Builds sources via Maven with unit testing.
• Runs two parallel integration tests that involve deploying the app to ephemeral server instances, which get thrown away when tests are done (this is done by using auto-deployment of Jetty)
• Once integration tests are successful, the webapp gets to the staging server at localhost:8081/staging (or your docker-machine/boot2docker instance IP)
• requires a human to Manually inspect the staging instance, and when ready, approves the deployment to the production server at http://localhost:8081/production/

References

• Developer blog by jglick introducing multibranch support
• workflow plugin tutorial
• workflow plugin presentations
• workflow plugin readme

We released Jenkins updates today that include important security fixes: 1.641 and 1.625.3. For detailed information about the security content of these updates, see the security advisory.

One of these fixes, SECURITY-95, results in possible problems in plugins such as Maven Plugin, Javadoc Plugin, and HTML Publisher Plugin, so make sure to read all about that in the wiki.

Please note that the update site may lag a bit behind. If you want to update as soon as possible, download the releases from our site.