There are plenty of exciting developments happening in the Jenkins community. The summit will feature a 'State of the Project' update including updates from the Jenkins officers. We will also have updates on the 'Big 5' projects in active development: * Jenkins Essentials * Jenkins X * Configuration as Code * Jenkins Pipeline * Cloud Native Jenkins Architecture

Plus we will feature a Google Summer of Code update, and more!

The agenda is shaping up well and here is the outline so far.

The BoF (birds-of-a-feather) session will be an opportunity for in depth discussions, hacking or learning more about any of the big 5. Bring your laptop, come prepared with questions and ideas, and be ready for some hacking too if you want. Join in, hear the latest and get involved in any project during the BoF sessions. If you want to share anything there will be an opportunity to do a 5-min ignite talk at the end. Attending is free, and no DevOps World | Jenkins World ticket is needed, but RSVP if you are going to attend to help us plan. See you there!

If you follow the developer mailing list, you may have seen the discussion about introducing SIGs in the Jenkins project. The SIG model has been proposed byR. Tyler Croy, and it largely follows the successfulKubernetes SIG model. The objective of these SIGs is to make the community more transparent to contributors and to offer venues for specific discussions. The idea of SIGs and how to create them is documented inJEP-4. JEP-4 is still in Draft state, but a few SIGs have been already created using that process:Platform SIG, GSoC SIG and, finally,Cloud Native SIG.

SIGs are a big opportunity to the Jenkins project, offering a new way to onboard contributors who are interested only in particular aspects of Jenkins. With SIGs they can subscribe to particular topics without following the entire Developer mailing list which can become pretty buzzy nowadays. It also offers company contributors a clear way how to join community and participate in specific areas. This is great for larger projects which cannot be done by a single contributor. Like JEPs, SIGs help focus and coordinate efforts.

And, back to major efforts…​ Lack of resources among core contributors was one of the reasons why we did not deliver on Pluggable Storage stories back in 2016. I believe that SIGs can help fix that in Jenkins, making it easier to find groups with the same interests and reach out to them in order to organize activity. Regular meetings are also helpful to get such efforts moving.

Points above are the main reasons why I joined the Cloud Native SIG. Similarly, that’s why I decided to create a Platform SIG to deliver on major efforts like Java 10+ support in Jenkins. I hope that more SIGs get created soon so that contributors could focus on areas of their interest.

In the original proposal Carlos Sanchez, the Cloud Native SIG chair, has described the purpose of the SIG well. There has been great progress this year in cloud-native-minded projects like Jenkins X and Jenkins Essentials, but the current Jenkins architecture does not offer particular features which could be utilized there: Pluggable Storage, High Availability, etc. There are ways to achieve it using Jenkins plugins and some infrastructure tweaks, but it is far from the out-of-the-box experience. It complicates Jenkins management and slows down development of new cloud-native solutions for Jenkins.

So, what do I expect from the SIG?

The SIG agenda is largely defined by the SIG participants. If you are interested to discuss particular topics, just propose them in the SIG mailing list. As the current SIG page describes, there are several areas defined as initial topics:Artifact Storage,Log Storage,Configuration Storage

All these topics are related to the Pluggable Storage Area, and the end goal for them is to ensure that all data is externalized so that replication becomes possible. In addition to the mentioned data types, discussed at the Jenkins World 2016 summit, we will need to externalize other data types:Item and Run storage,Fingerprints,Test and coverage results, etc. There is some foundation work being done for that. For example, Shenyu Zheng is working on aCode Coverage API plugin which would allow to unify the code coverage storage formats in Jenkins.

Once the Pluggable Storage stories are done the next steps are true High Availability, rolling or canary upgrades and zero downtime. At that point other foundation stories like Remoting over Kafka by Pham Vu Tuan might be integrated into the Cloud Native architecture to make Jenkins more robust against outages within the cluster. It will take some time to get to this state, but it can be done incrementally.

Let me briefly summarize current state of the 3 focuses listed in the Cloud Native SIG.

Special Interest Groups are a new format for collaboration and disucssion in the Jenkins community. Although we have had some work groups before (Infrastructure, Configuration-as-Code, etc.), introduction of SIGs sets a new bar in terms of the project transparency and consistency. Major architecture changes in Jenkins are needed to ensure its future in the new environments, and SIGs will help to boost visibility and participation around these changes.

If you want to know more about the Cloud Native SIG, all resources are listed on the SIG’s page on jenkins.io. If you want to participate in the SIG’s activities, just do the following:

I am also working on organizing a face-to-face Cloud Native SIG meeting at theJenkins Contributor Summit, which will happen on September 17 duringDevOps World | Jenkins World in San Francisco. If you come to DevOps World | Jenkins World, please feel free to join us at the contributor summit or to meet us at the community booth. Together with Jesse and Carlos we are also going to present some bits of our work at theA Cloud Native Jenkins talk.

Stay tuned for more updates and demos on the Cloud-Native Jenkins fronts!

When we talk about Jenkins here, we’re referring to “Jenkins 2” - a name we use to generally refer to the 2.0 and beyond versions of Jenkins. Jenkins 2 offers a powerful evolution of Jenkins over prior versions. In particular, it provides full integration for “pipeline-as-code” (PAC). PAC refers to being able to write your pipeline in a scripting language, much like source code for any program. The code you write becomes the program that defines your pipeline. It is also the code that gets executed when your pipeline is initiated. Listing 1 shows a simple example pipeline. Notice that this is very different from the classic way of creating pipelines in Jenkins. Here you are writing code - rather than the more traditional approaches, such as filling in web forms to configure a Freestyle job.

Listing 1: Example Jenkins 2 pipeline

The language that we write the Jenkins pipeline code in is a Domain-Specific Language (DSL). You can think of it as the “programming language” for Jenkins pipelines. There are two variants of it. The style we saw in figure 1 is called “scripted syntax”. It is a mixture of elements from the Groovy programming language and special Jenkins “steps”. The Jenkins steps are provided by the plugins that are installed in the current system. A built-in tool called the Snippet Generator provides a wizard interface to allow users to pick the step and options they want. Then, the user can click on a button to have Jenkins automatically generate the correct DSL code in the large text box (figure 1). The DSL code can be copied from there and pasted into the pipeline script.

A second type of syntax is called “declarative syntax.” We won’t go into detail on it here. But it is a much more structured syntax that focuses on having users declare what they want in a pipeline, rather than writing the logic to make it happen.

In addition to the steps that are provided by plugins, additional functionality for pipelines can be provided by global variables. The simplest way to think of a global variable is as an object with methods that can be invoked on it. Several of these are built in to Jenkins, such as the Docker global variable. Others can be created by users as part of the structure of a shared source code repository called a “shared pipeline library.”

To get a list of the global variables that are currently available to your Jenkins instance, you can go to the Snippet Generator screen. Immediately below the box for the generated pipeline script is a section titled Global Variables. There, within the small print, is a link to get to the actual section (figure 2).

Clicking on that link takes us to a list of currently available Global Variables. If you have the Docker Pipeline Plugin installed, you will see one at the top for Docker. (Figure 3).

Broadly, the docker global variable includes methods that can be applied to the Docker application, Docker images, and Docker containers.

We’ll focus first on a couple of the Docker image methods as shown in figure 4.

There are multiple ways you can use these methods to create a new image. Listing 2 shows a basic example of assigning and pulling an image using the image method.

Listing 2: Assigning a image to a variable and pulling it down.

This can also be done in a single statement as shown in listing 3.

Listing 3: Shorthand version of previous call.

You can also download a Dockerfile and build an image based on it.(See listing 4.)

Listing 4: Pipeline code to download a Dockerfile and build an image from it.

Figure 5 shows the actual output from running that “Build image” stage. Note that the docker.build step was translated into an actual Docker build command.

Another powerful method available for the Docker global variable is the inside method. When executed, this method will do the following:

Mounting the workspace means that the Jenkins workspace will appear as a volume inside the container. And it will have the same file path. So, things running in the container will have direct access to the same location. However, this can only be done if the container is running on the same underlying system - such that it can directly access the path.

In terms of executing the build steps, the inside method acts as a scoping method. This means that the environment it sets up is in effect for any statement that happens within its scope (within the block under it bounded by {}). The practical application here is that any pipeline “sh” steps (a call to the shell to execute something) are automatically run in the container. Behind the scenes, this is done by wrapping the calls with “docker exec”.

When executed, the calls with the global variable are translated (by Jenkins) into actual Docker call invocations. Listing 5 shows an example of using this in a script, along with the output from the first invocation of the “inside” method. You can see in the output the docker commands that are generated from the inside method call.

Listing 5: Example inside method usage.

Once completed, the inside step will stop the container, get rid of the storage, and create a record that this image was used for the build. That record facilitates image traceability, updates, etc.

As you can see, the combination of using the Docker “global variable” and its “inside” method provide a simple and powerful way to spin up and work with containers in your pipeline. In addition, since you are not having to make the direct Docker calls, you can invoke steps like sh within the scope of the inside method, and have them executed by Docker transparently.

As we mentioned, this is only one of several ways you can interact with Docker in your pipeline code. To learn about the other methods and get hands-on practice, join me at DevOps World/Jenkins World in San Francisco or Nice for the workshop "Creating a Deployment Pipeline with Jenkins 2". Hope to see you there!

There are plenty of exciting developments happening in the Jenkins community. The summit will feature a 'State of the Project' update including updates from the Jenkins officers. We will also have updates on the 'Big 5' projects in active development:

Plus we will feature a Google Summer of Code update, Special Interest Group updates and more!

The agenda is shaping up well and here is the outline so far.

The BoF (birds-of-a-feather) session will be an opportunity for in depth discussions, hacking or learning more about any of the big 5. Bring your laptop, come prepared with questions and ideas, and be ready for some hacking too if you want. Join in, hear the latest and get involved in any project during the BoF sessions. If you want to share anything there will be an opportunity to do a 5-min ignite talk at the end. Attending is free, and no DevOps World | Jenkins World ticket is needed, but RSVP if you are going to attend to help us plan. See you there!

JCasC allows us to apply a set of YAML files on our Jenkins master at startup or on-demand via the web UI. Those configuration files are very concise and human readable compared to verbose XML files the Jenkins uses to actually store configuration. The files also have user-friendly naming conventions making it easy for administrators to configure all Jenkins components.

Here’s an example:

As you can see, you don’t need long explanation to understand how this YAML file will setup your Jenkins master.

The most immediate benefit of JCasC is reproducibility. An administrator can now bootstrap a new Jenkins master with the exact same configuration with a trivial setup. This allows them to create a test instance and check the impact of plugin upgrades in a sandboxed environment. This also lets them be more confident with failover and disaster recovery scenarios.

Further benefits come when administrators start managing their Jenkins’ YAML configuration files in source control, like they do with Terraform configuration. Doing so gives them auditing and reversibility of their Jenkins master configuration. Theycan establish a sane configuration change workflow that runs a test Jenkins instance and ensures configuration is healthy before actually applying any change to their production Jenkins master.

Last but not least, with ability to quickly setup Jenkins masters and control them from a set of shared YAML configuration files, administrators can now offer per-team Jenkins instances, with more flexibility on installed plugins. A Master becomes more or less a transient piece of infrastructure for your team, as long as they also manage build definition with Jenkinsfiles.

With Configuration-as-Code we can stop having to treat our Jenkins master like a pet we need to pamper, and start managing Jenkins masters as cattle you can replace without effort nor impacts. Welcome in the “as-code” world.

You can read more about the Jenkins Configuration-as-Code plugin on the project’s github repository. To chat with the community and contributors join ourgitter channel, or come see us in person at link:Jenkins World to discuss the JCasC project and its future!

Also don’t miss next post from the Configuration-as-Code series, where we’ll look at how JCasC works with sensitive data like passwords and other credentials.

Since Jenkins is all about running short workloads, you should aim for the Terminating scope of quota. But how do you specify workloads in Jenkins so that correct scope is used?

If you were to do this in Kubernetes, you have to specify .spec.activeDeadlineSeconds. The same field can also be specified by the Kubernetes plugin when you are specifying a Pod Template.

Same configuration is available in the Jenkinsfile as well if you don’t like static configurations.

This was just a small sample of features of the Kubernetes plugin in Jenkins. For more, be sure to check out ourtalk where we share more of how you can utilise Kubernetes with Jenkins!

Our project has been an amazing success over the past 10+ years, thanks to you all. What started as my hobby project became a huge community that boasts thousands of contributors and millions of users. When I think about what enabled this amazing journey, I can think of several magic sauces:

The way we set up our community meant that collectively we were able to work toward solving certain kinds of problems locally and organically, such as Android application development, new UX, more expressive pipeline description language, …​

But at the same time, the incremental, autonomous nature of our community made us demonstrably unable to solve certain kinds of problems. And after 10+ years, these unsolved problems are getting more pronounced, and they are taking a toll — segments of users correctly feel that the community doesn’t get them, because we have shown an inability to address some of their greatest difficulties in using Jenkins. And I know some of those problems, such as service instability, matter to all of us.

In a way, we are stuck in a local optimum, and that is a dangerous place to be when there is growing competition from all sides. So we must solve these problems to ensure our continued relevance and popularity in the space.

Solving those problems starts with correctly understanding them, so let’s look at those.

In the past, my frustration and regret is that we couldn’t take on an effort of this magnitude. But that is NO MORE! As CTO of CloudBees, I’m excited that these challenges are important enough for CloudBees now that we want to solve these efforts within the Jenkins project.

I’ve been talking to many of you, and there are a number of existing efforts going on that touch this space already. From there, the vision emerged is that we organize around two key efforts:

This document lays out the key directions and approaches in a broad stroke, which I discussed with a number of you in the past. Hopefully, this gives you the big picture of how I envision where to move Jenkins forward, not just as the creator of Jenkins but as the CTO of CloudBees, who employs a number of key contributors to the Jenkins project.

The Warnings plugin provides the following features when added as a post build action (or step) to a job:

In the next sections, I’ll show the new and enhanced features in more detail.

I hope these new features are useful for everyone! Please download or install this new release and test it in your jobs:

If you find a problem, incorrect phrase, typo, etc. please report a bug in Jira (or even better: file a PR in GitHub).

This has been a brief overview of the new features of the Warnings plugin in Jenkins. For more, be sure to check out mytalk at "DevOps World | Jenkins World" where I show more details of the Warnings plugin!

Pluggable storage is one of the major areas we have been working on over the last few months. There are a number of parallel stories which are summarized onthis page. There has been significant progress in the areas of artifact storage, build logging and configuration storage. A number of Jenkins Enhancement Proposals were submitted and accepted, and there are plugin releases and prototypes for these stories.

During our talk we will discuss the current status of these stories and future plans. In particular, we will cover the following areas and reference implementations:

There are existing plugins for the areas above, but there is a difference in approach we have taken. Instead of creating new custom steps we extend Jenkins architecture in a way that the storage becomes transparent to users. For example, with Artifact Manager for S3 Plugin common Archive Artifacts steps work transparently with Remote storage, as well as Jenkins Pipeline’s stash()/unstash() steps.

The reference implementations intentionally use different technologies so that we cover more scenarios. We regularly discuss the implementations in the Cloud Native SIG, and we would appreciate your feedback.

Want something new? Several days ago Kohsuke Kawaguchi, the creator of Jenkins, posted theJenkins: Shifting Gears article to summarize the plan for Jenkins evolution. Cloud Native Jenkins is a critical part of this plan, and it is not “just Jenkins X”. There are various architectural changes in Jenkins required to make this vision happen, and we plan to work on these changes in the Cloud Native SIG.

In our presentation, we will talk about our experiment with ephemeral Jenkins and single-shot masters. In this story we are creating a headless single-shot master which starts in a container, executes a Pipeline build and pushes all the results to remote storage so that the container can just be deleted after completion. Such a master bundles plugins and self-configuration logic using “Configuration as Code”, so that it can start executing Pipelines in just a few seconds. Once packaged, it can be invoked from CLI as simply as…​

or, in Kubernetes:

Such a single-shot master could also be made a part of a Cloud Native Jenkins system. Standard event handlers like Prow can invoke the builds on webhooks and report results back, so that the single-shot master can be used to build pull requests or to run Continuous Delivery flows. Extra agents could also be connected to the master on-demand, using the Kubernetes plugin or sidecar containers.

In order to make this experiment possible, we used a toolchain based on Docker,Jenkinsfile Runner,Configuration as Code Plugin (JCasC), and a links:https://github.com/jenkinsci/custom-war-packager[Custom WAR Packager tool] which glues all the things together.

Custom WAR Packager is a new tool which takes various configurations (YAML specification defining core version, list of plugins, system properties, Groovy Hooks, JCasC YAMLs)…​ and then bundles everything as a ready-to-fly WAR file or Docker image. Starting from version 1.2, Custom WAR Packager also supports packaging Jenkinsfile Runner images as an experimental feature. I will do a separate blogpost about this new tool later, but there is already some documentation a number of demos in the project’s repo.

Yes, we will have a demo! We will show a single-shot master running with Pluggable storage implementations for AWS environments (Amazon S3, AWS CloudWatch, EKS, etc.), which executes Jenkins Pipelines for Maven projects and provisions agents in Kubernetes on-demand.

The demo has to be published yes, but you can already find a more simple Jenkinsfile Runner demohere.

The upcoming DevOps World | Jenkins World conferences are heavily packed with talks related to Cloud Native Jenkins, including war stories and presentations on projects like Jenkins X and Jenkins Evergreen. It is a great chance to get more information about using Jenkins in cloud environments.

If you are a Jenkins contributor or just want to become a contributor, also join the Contributor Summit (Sep 17 in US and Oct 23 in Nice) or visit the Jenkins community booth in the Exhibition hall. At the Contributor Summit on Sep 17 we will also have a face-to-face Cloud Native SIG meeting. Feel free to contribute to the agenda here.

With DevOps World | Jenkins World San Francisco right around the corner, CloudBees is excited to sponsor the 2018 DevOps and Jenkins Community Survey. We want to capture the details of your DevOps experience in order to provide valuable insights to the Jenkins Community and beyond. Our community is stronger together - and this look at our collective experience will reveal the big picture and shine a light on key trends. This year, as the Jenkins project continues to evolve with Jenkins X , Configuration as Code and more, your input is more critical than ever.

Let’s look at what we learned in 2016 and 2017:

In 2016 we found that:

In 2017 respondents reported that:

What will you and the community tell us this year? Are more people practicing DevOps? Is Kubernetes the leader in container orchestration? Is pipeline the standard for creating workflows? Take the survey and let’s find out!

As always, your personal information (name, email address and company) will NOT be used by CloudBees for sales or marketing and the survey results will be made publicly available to the Jenkins Community. We will also be publishing a blog series analyzing trends over the last 5 years and offering predictions on the evolution of DevOps. If you’re curious about what insights your input will provide, see the results of last year’s 2017 survey.

As an added incentive to participate, CloudBees will enter participants into a drawing for a free pass to DevOps World | Jenkins World 2019 (1st prize, $1,199.00 value) or a $100 Amazon gift card (2nd prize)!

The survey will close at the end of October so grab a cup of coffee get started. We promise the survey will be done before your latte is.

Take me to the survey.

There are laws that govern prize giveaways and eligibility; CloudBees has compiled all those fancy terms and conditions here.