Quantcast
Channel: Jenkins Blog
Viewing all 1087 articles
Browse latest View live

Jenkins World 2018: Call for Papers is Open

$
0
0

This is a guest post by Alyssa Tong, who runs the Jenkins Area Meetup program and is also responsible for Marketing & Community Programs at CloudBees, Inc.

Jenkins World 2018

Happy 2018! The Jenkins World train is ready to take off once again. As usual, the sign of festivities looming begins with the Call for Papers. Those who attended Jenkins World 2017 know that Jenkins World 2018 is coming back to San Francisco. But what they did not know is that Jenkins World will also be coming to Europe. You read that right, Jenkins World is taking place in two locations in 2018:

  • Jenkins World USA | San Francisco | September 16 - 19, 2018

  • Jenkins World Europe | October - date and location TBA

To encourage open collaboration and stimulate discussions that will help advance Jenkins adoption and drive it forward, we invite Jenkins users, developers and industry experts to submit a speaking proposal to Jenkins World San Francisco and or Europe. Submissions for both locations are being accepted now and will close on March 18, 2018 @ 11:59PM Pacific.

Where do I go to submit my proposal?

Submissions for both Jenkins World USA and Europe are accepted at:

Can I make proposal(s) to both conferences?

Yes, you can! Once you’ve created an account on the CFP website you will be given the option to make submission(s) to one conference or both conferences.

When is the deadline Jenkins World USA?

Sunday March 18, 2018 @ 11:59PM Pacific

When is the deadline for Jenkins World Europe?

Sunday March 18, 2018 @ 11:59PM Pacific

Important Dates:

  • CFP Opens: January 17, 2018

  • CFP Closes: March 18, 2018 @ 11:59pm Pacific

  • CFP Notifications: April

  • Agenda Announcement: April

  • Event Dates:

    • Jenkins World USA | San Francisco | September 16 - 19, 2018

    • Jenkins World Europe | October - exact date TBA


Overhaul of Manage Jenkins page

$
0
0

Overview

Recently some UI improvements around the Manage Jenkins page have been introduced. The visual changes are very subtle but behind them, there are interesting benefits.

Some of the goals that we have tried to achieve:

  • Applying a semantic HTML

  • Removing the <table> tag usage for implementing layouts and content structures. Read this article if you want to know reasons and/or arguments.

  • Small re-styling focused on spacing, margins, composition, etc..

  • Accessibility

In order to provide a quick overview of the visual changes, let’s take a look at these screenshots.

System tray with administrative messages (before)

JENKINS 43786 3 before

System tray with administrative messages (after)

JENKINS 43786 2

Manage Jenkins page (before)

JENKINS 43786 4 before

Manage Jenkins page (after)

JENKINS 43786 1

Information about how this change can affect the current implementations of Administrative Monitors can be found in the following section

For core developers

Let’s use a real example for showing how this proposal works.

This is the original UI implementation of HudsonHomeDiskUsageMonitor.java:

<?jelly escape-by-default='true'?><j:jellyxmlns:j="jelly:core"xmlns:st="jelly:stapler"xmlns:d="jelly:define"xmlns:l="/lib/layout"xmlns:t="/lib/hudson"xmlns:f="/lib/form"><divclass="warning"><formmethod="post"action="${rootURL}/${it.url}/act"name="${it.id}"><divstyle="float:right"><f:submitname="yes"value="${%Tell me more}"/><f:submitname="no"value="${%Dismiss}"/></div>
      ${%blurb(app.rootDir)}</form></div></j:jelly>

And this is the proposed change:

<?jelly escape-by-default='true'?><j:jellyxmlns:j="jelly:core"xmlns:st="jelly:stapler"xmlns:d="jelly:define"xmlns:l="/lib/layout"xmlns:t="/lib/hudson"xmlns:f="/lib/form"><divclass="alert alert-warning"><formmethod="post"action="${rootURL}/${it.url}/act"name="${it.id}"><f:submitname="yes"value="${%Tell me more}"/><f:submitname="no"value="${%Dismiss}"/></form>
  ${%blurb(app.rootDir)}</div></j:jelly>

Some highlights:

  • No more ad hoc UI compositions

  • No more custom CSS classes when Jenkins project is already using Bootstrap for many different things

  • Based on Bootstrap Alert

All administrative monitors defined in Jenkins core have been adapted as part of this proposal.

For plugin developers

No changes are really needed, but we do recommend you to adapt your plugins to this proposal so Jenkins users have a better user experience.

Taking into account that you want to keep backward compatibility, you will need some changes.

In your implementation of Administrative Monitor, add this helper method:

/**
 * This method can be removed when the baseline is updated to 2.103
 *
 * @return If this version of the plugin is running on a Jenkins version where JENKINS-43786 is included.
 */@Restricted(DoNotUse.class)publicboolean isTheNewDesignAvailable() {if (Jenkins.getVersion().isNewerThan(new VersionNumber("2.103"))) {returntrue;
    }returnfalse;
}

In your view (a.k.a. Jelly file or Groovy file):

<?jelly escape-by-default='true'?><j:jellyxmlns:j="jelly:core"><j:iftest="${!it.isTheNewDesignAvailable}"><divclass="warning"><p>SSH Host Key Verifiers are not configured for all SSH slaves on this Jenkins instance. This could leave these slaves open to man-in-the-middle attacks. <ahref="${rootURL}/computer/">Update your slave configuration</a> to resolve this.</p></div></j:if><j:iftest="${it.isTheNewDesignAvailable}"><divclass="alert alert-warning">
    SSH Host Key Verifiers are not configured for all SSH slaves on this Jenkins instance. This could leave these slaves open to man-in-the-middle attacks. <ahref="${rootURL}/computer/">Update your slave configuration</a> to resolve this.</div></j:if></j:jelly>

If you don’t want to keep a strict backward compatibility, the impact is minimal. In fact, you can see an example on GitHub Plugin.

Some helpful references:

Do not hesitate to ping me if you decide to adapt your Administrative Monitors.

You are invited to the Post-FOSDEM 2018 Jenkins Hackfest

$
0
0

On the first weekend in February, numerous free and open source developers from around the world will travel to Brussels, Belgium, for arguably the largest event of its kind:FOSDEM. Among the thousands of hackers in attendance will be a number of Jenkins contributors.

On the Monday after FOSDEM,you are invited to join a group of those contributors for a full day of hacking on Jenkins. Folks of all experience levels are welcome; there will be sessions for everyone from seasoned hackers to new contributors.

All-day Jenkins Hackfest

The Hackfest will start at 9:30am on Monday with general introductions and gathering of potential topics/projects. Bring suggestions for topics that interest you, or just come and choose from the topics others suggest. There will be plenty of topics from which to choose. All the topic suggestions will all be added to a backlog and we will identify and cluster around popular topics. Then we’ll divide into smaller groups and work on individual topics in timeboxed sessions.

Meals, snacks, and beverages will be provided throughout the day, wrapping up with dinner around 5pm.

Something for everyone

Hackfests like this one are a great opportunity for contributors of all levels to get invovled, learn from each other, and work together on interesting and high impact areas of the project.

Some long-time contributors already know what areas they’ll work on and are looking for people interested in joining them. Mark Waite (maintainer of the Jenkins Git and Git Client plugins) and Christian Halstrick (SAP) will be spending the day improving the way Git client plugin uses JGit. R. Tyler Croy (Jenkins community concierge) and Olivier Vernin (Jenkins infrastructure engineer) will work on infrastructure improvements.

Others contributors, such as Jesse Glick and Andrew Bayer (recipients of the "A Small Matter of Programming" award), will arrive without a set plan. They will, of course, have some topics to propose, so you might get a chance to work with them. Or if you have an area you’d like to work on, they and many other experts will be on hand for discussion and code review.

This is also a great opportunity for new contributors to join the project. Baptiste Mathus, long time contributor and all-around nice guy, will host a "New Contributor Hackergarten" covering the basics of contributing to Jenkins and submitting fixes via GitHub Pull requests. Even those with minimal coding experience can contribute by improving documentation and making typo fixes via this same process.

Fun!

More than anything else, Hackfests like this are great fun. No matter what your level of exerience, there will be plenty to do and great people with whom to do it. Reserve a space byjoining the meetup here. Then bring your own laptop and passion for improving Jenkins.

Details

  • Date: Monday, February 5, 2018

  • Time: 9:30 AM to 5:00 PM

  • Location: BeCentral sprl/bvba Cantersteen 12 1000 Brussel Belgium Room: Studio C (1st floor)

  • RSVP Required

Meals, snacks, and beverages will be provided. Bring your own computer.

Ask for "Jenkins" at the front desk if you get lost.

Security updates for multiple Jenkins plugins

Security updates for Jenkins core

$
0
0

We just released security updates to Jenkins, versions 2.107 and 2.89.4, that fix multiple security vulnerabilities.

For an overview of what was fixed, see the security advisory. For an overview on the possible impact of these changes on upgrading Jenkins LTS, see our LTS upgrade guide.

While the severity score works out as medium for all the vulnerabilities, we strongly recommend that anyone operating publicly accessible Jenkins instances update as soon as possible, as their secrets on disk might be at risk by SECURITY-705.

Subscribe to the jenkinsci-advisories mailing list to receive important notifications related to Jenkins security.

Jenkins is accepted to Google Summer Of Code 2018, join us!

$
0
0

Jenkins GSoC

We are happy to announce that Jenkins project has been accepted toGoogle Summer of Code 2018. This year we invite students and mentors to join the Jenkins community and work together on various initiatives: core, plugins, development tools and infrastructure.

Our mentors have already created some project ideas. For example, you are welcome to work on the new Configuration-as-Code initiative or to help creating standard API for Code Coverage plugins. If you like test automation, there is a proposal toJenkins Acceptance Test Harness. Over the next weeks we plan to extend this list by new even more project ideas.

All information about the Jenkins GSoC is available on its subproject page.

I am a student. How do I apply?

See Information for students for application guidelines.

First step is to join discussions in the mailing lists in order to introduce yourself, establish connections with the community and potential mentors. The application period starts on March 12 and ends on March 27, but don’t let it misguide you! Use time before application to discuss project ideas/proposals with mentors and to process their feedback. To create a better proposal, we also recommend to study Jenkins and to do some contributions in the area of your project proposal.

Not satisfied by the current project ideas? You can propose your own idea in the developer mailing list.

I want to be a mentor. Is it too late?

It’s not! If you are passionate about open-source and Jenkins, we invite you to join the mentors team. You can either propose a new project idea or join an existing one. See Call for Mentors and Information for mentors for details.

This year mentorship does NOT require strong expertise in Jenkins development. The objective is to guide students and to get involved into the Jenkins community. GSoC org admins will help to find advisers if special expertise is required.

Important dates

  • Mar 05 - deadline for new GSoC project idea proposals

  • Mar 12 - student application period starts

  • Mar 27 - deadline for student applications

  • Apr 23 - accepted projects announced, community bonding starts

  • May 14 - coding period starts

  • Aug 06 - end of the coding period

See the GSoC Timeline for more info. In the Jenkins project we will also organize special events during and after GSoC (e.g. at Jenkins world).

P.S: I am going to create a special GSoC 2018 edition of swags for mentors and students. Do not miss them ;)

Project Cheetah - Faster, Leaner Pipeline That Can Keep Up With Demand

$
0
0

This is a guest post by Sam Van Oort, Software Engineer at CloudBees, and contributor to the Jenkins project, and maintainer of the Pipeline Plugins.

Since it launched, Pipeline has had a bit of a Dr. Jekyll and Mr. Hyde performance problem. In certain circumstances, Pipeline can turn from a mild-mannered CI/CD assistant into a monster. It will happily eat storage read/write capacity like popcorn without caring about the other concerns of our friendly butler. When combined with other additional factors, this can result in real-world stability problems. For example, combining slow storage with a spike in running Pipelines has brought down production Jenkins at more than one organization. Similarly, users see issues if a busy master gets hit with an extra source of stress; past culprits have been heavy automated (ab)use of Jenkins APIs, now-solved user lookup bugs, backup jobs, and plugins run crazy that load excessive numbers of builds. Symptoms ranged from visible slowdowns in the UI to unresponsive jobs and "hung" masters.

Now I’m not saying this to scare people or to criticize the technology we’ve built. Implementing Pipeline scalability best practices coupled with SSD storage keeps Jenkins in a happy place. We just need context on the weaknesses to see why it’s important to address them.

Introducing "Project Cheetah"

Today we’re announcing the first major results of "Project Cheetah", our long-running effort to address these challenges and improve Pipeline scalability. More broadly, Cheetah aims to help in 3 places:

  • Small-scale containers: Pipeline needs to run leanly in resource-constrained containers, to enable easy scale-out without consuming excessive resources on shared container hosts.

  • Enterprise systems: Pipeline needs to effectively serve high-scale Jenkins instances that are central to many large companies.

  • General case: run Pipelines a bit more quickly on average, and allow users to get much-stronger performance in worst-case scenarios.

These changes are implemented across many of the Pipeline plugins.

Yes, but what does it DO?

Project Cheetah offers several things, but the most important is Durability Settings for all Pipelines, and especially the Performance-Optimized setting. This setting avoids several potentially unexpected performance "surprises" that may strike users. In the general case, it greatly reduces the disk IO needs for Pipeline. How much? Below is a graph of storage utilization with legacy Pipeline versions (think early 2017) and with the latest version using the Performance-Optimized mode. These are tested on an AWS instance backed by an EBS volume provisioned with 300 IOPs.

Before and After:

io utilization

As you can see, storage utilization goes down by a lot. While the exact number will vary, across the benchmark testcases this results in Pipeline throughput of 2x to 6x the previous before becoming IO-bound. This also increases stability of Jenkins masters because they will tolerate unexpected load.

This comes with a major drop in CPU IOWait as well:

cpu iowait

And of course the rate at which data is written to disk and number of writes/s is also reduced:

diskio

For enterprise users, timing stats often show 10-20% of normal builds is serializing the Program and writing the record of steps run ("FlowNodes") - the performance optimized durability setting will cut this to almost nothing (for standard pipelines, 1/100 or less) - so builds will complete faster, especially complex ones.

Please see the Pipeline Scalability documentation for deeper information on the new Durability Settings, how to use them, and which plugin versions are required to gain these features.

Also, users may see a reduction in hung Pipelines because new test utilities made it possible to identify and correct a variety of bugs.

How Do I Set Speed/Durability Settings?

There are 3 ways to configure the durability setting:

1. Globally, you can choose a global default durability setting:

Under "Manage Jenkins" > "Configure System", labelled "Pipeline Speed/Durability Settings". You can override these with the more specific settings below.

global durability

2. Each Pipeline can get a custom Durability Setting:

This is one of the job properties located at the top of the job configuration, labelled "Custom Pipeline Speed/Durability Level." This overrides the global setting. Or, use a "properties" step - the setting will apply to the NEXT run after the step is executed (same result).

pipeline durability highlighted

Jenkinsfile (Declarative Pipeline)
pipeline {
    agent any
    stages {
        stage('Example') {
            steps {
                echo 'Hello World'
            }
        }
    }
    options {
        durabilityHint('PERFORMANCE_OPTIMIZED')
    }
}

3. Multibranch Projects can use a new BranchProperty to customize the Durability Setting.

Under the SCM you can configure a custom Branch Property Strategy and add a property for Custom Pipeline Speed/Durability Level. This overrides the global Durability Setting and will apply to each branch at the next run. You can also use a "properties" step to override the setting, but remember that you may have to run the step again to undo this.

multibranch durability

Durability settings will take effect with the next applicable Pipeline run, not immediately. The setting will be displayed in the log.

durability in log highlighted

There is a slight durability trade-off for using the Performance-Optimized mode — the appropriate section of the Pipeline Scalability documentation has the specifics. For most uses we do not expect this to be important, but there are a few specific cases where users may wish to use a slower/higher-durability setting. The Best Practices are documented.

We recommend using Performance-Optimized by default, but because it does represent a slight behavioral change the initial "Cheetah" plugin releases defaults to maintain previous behavior. We expect to switch this default in the future with appropriate notice once people have a chance to get used to the new settings.

Will Performance-Optimized Mode Help Me?

  • Yes, if your Jenkins instance uses NFS, magnetic storage, runs many Pipelines at once, or shows high iowait (above 5%)

  • Yes, if you are running Pipelines with many steps (more than several hundred).

  • Yes, if your Pipeline stores large files or complex data to variables in the script, keeps that variable in scope for future use, and then runs steps. This sounds oddly specific but happens more than you’d expect.

    • For example: readFile step with a large XML/JSON file, or using configuration information from parsing such a file with One of the Utility Steps.

    • Another common pattern is a "summary" object containing data from many branches (logs, results, or statistics). Often this is visible because you’ll be adding to it often via an add/append or Map.put() operations.

    • Large arrays of data or Maps of configuration information are another common example of this situation.

  • No, if your Pipelines spend almost all their time waiting for a few shell/batch steps to finish. This ISN’T a magic "go fast" button for everything!

  • No, if Pipelines are writing massive amounts of data to logs (logging is unchanged).

  • No, if you are not using Pipelines, or your system is loaded down by other factors.

  • No, if you don’t enable higher-performance modes for pipelines. See above for how!

Other Goodies

  • Users can now set an optional job property so that individual Pipelines fail cleanly rather than resuming upon restarting the master. This is useful for niche cases where some Pipelines are considered disposable and users would value a clean restart over Pipeline durability.

  • We’ve reduced classloading and reflection quite significantly, which improves scaling and reduces CPU use:

classloading

  • Script Security (as of version 1.41) has gotten optimizations to reduce the performance overhead of Sandbox mode and eliminate lock contention so Pipeline multithreads better.

  • Pipeline Step data uses up less space on disk (regardless of the durability setting) - this should be 30% smaller. Assume it’s a few MB per 1000 steps - but for every build after the change.

  • Even in the low-performance/high-durability modes, some redundant writes have been removed, which decreases the number of writes by 10-20%.

How Did You Do It?

That’s probably material for another blog post or Jenkins World talk.

The short answer is: first we built a tool to simulate a full production environment and provide detailed metrics collection at scale. Then we profiled Jenkins to identify bottlenecks and attacked them. Rinse and repeat.

What Next?

The next big change, which I’m calling Cheetah Part 2 is to address Pipeline’s logging. For every Step run, Pipeline writes one or more small log files. These log files are then copied into the build log content, but are retained to make it possible to easily fetch logs for each step.

This copying process means every log line is written twice, greatly reducing perforance, and writing to many small files is orders of magnitude slower than appending to one big log file.

We’re going to remove this duplication and data fragmentation and use a more efficient mechanism to find per-step logs. This should further improve the ability to run Pipelines on NFS mounts and hard-drive-backed storage, and should significantly improve performance at scale.

Besides this, there’s a variety of different tactical improvements to improve scaling behavior and reduce resource needs.

The Project Cheetah work doesn’t free users to completely ignorePipeline scaling best practices and previous suggestions. Nor does it eliminate the need for efficient GC settings. But this and other enhancements from the last year can significantly improve the storage situation for most users and reduce the penalties for worst-case behaviors. When you add all the pieces together, the result is a faster, leaner, more reliable Pipeline experience.

5 tips for improving your Jenkins World 2018 talk proposals

$
0
0

Jenkins World

You’ve marked your calendar to alert you weeks in advance of the CFP deadline. You’ve spent your nights and weekends to skillfully craft your abstract and you are convinced this is THE one. You confidently clicked the submit button only to find out weeks later you’ve received an automated declined email. On top of feeling disappointed, you probably wondered why your proposal was turned down.

As Jenkins World CFP is still open, we hope to demystify the selection process by offering a few tips to help increase the chance of your proposal getting accepted.

Tip 1: Make your title concise and attention-grabbing

There’s only a few seconds to make an impression with your proposal. Your title and abstract should be succinct and attention grabbing. Think about what makes your Jenkins proposal exceptional over others. What value will it bring to attendees?

Tip 2: Know your audience

Jenkins World audiences want to learn and adopt the newest development practices, process, languages and frameworks. They are highly technical and are interested in deep technical talks that are original and offers valuable information.

Tip 3: Transformational user stories

Share your unique Jenkins project experiences, challenges, strategies, lessons learned, and outcome. What can you share with your audience that will facilitate knowledge transfer and development of new and innovative techniques?

Tip 4: Topic should be relevant and applicable

Consider how to make the topic more broadly applicable to the attendees, such as with a hands-on demo explaining the concept. By showcasing innovative methods that draw out relevant and compelling knowledge to the audience, everybody is likely to more thoroughly absord the content of the talk. Also, who doesn’t love a cool demo?

Tip 5: Got Tips & Tricks?

Jenkins is used tons of different ways. Sharing novel approaches, tips, or tricks for increased efficiency and proficiency with Jenkins is almost always well received!

Important Dates:

  • CFP Closes: March 18, 2018 @ 11:59pm Pacific

  • CFP Notifications: April

  • Agenda Announcement: April

  • Event Dates:

    • Jenkins World USA | San Francisco | September 16 - 19, 2018

    • Jenkins World Europe | Nice, France | October 23 - 25, 2018

Register for Jenkins World 2018 with the code JWATONG for a 20% discount off your pass.

Good Luck!


Browser Market Share on jenkins.io

$
0
0

Over the last year, several efforts were done on jenkins.io likesecurity advisories or documentation and I wanted to understand the impact it had on its traffic.

I had a look to the Google Analytics account used for jenkins.io and it was interesting to discover which browsers are used by Jenkins visitors and how this compares with other websites. So I decided to analyze one year of data from January 2017 to January 2018. Then I selectedstatcounter as an external data source in order to compare results from jenkins.io.

First, let’s talk about numbers:

  • During that period, 3,496,245 users across the world visited this website.

  • This is a growth of 52.69% visitors compared to the year before.

  • 92.28% of them used a desktop, 6.63% were on mobile and only 1.10% on a tablet.

jenkins browser 2017 2018

Table 1. Browser Usage
BrowserPercentage

Chrome

69.82%

Firefox

14.75%

Safari

6.44%

IE

5.84%

Edge

1.59%

We clearly see that 91% of Jenkins.io visitors only use three browsers. Chrome is by far the leader followed by Firefox and Safari.

Obviously those numbers taken alone don’t mean anything, so let’s try to put them in perspective with another data source.

  • Netscape 4.8 was still used by 15 visitors over the year.

  • 6 visitors used their Nintendo DS.

StatCounter

Let’s now compare our previous results with the same time period from statcounter who argues to collect data from 2million websites. Statcounter allow us to display browsers stats based on 4 different categories, All platform, Desktop, Tablet or Mobile. + For this comparison, I am interested by All platform and of course Desktop as 92.28% of Jenkins.io visitors use a desktop.

Table 2. Browser Market Share Across All Platforms
BrowserPercentage

Chrome

54.13%

Safari

14.50%

UC Browser

8.25%

Firefox

6.15%

IE

3.89%

Excepted Chrome who still leads the ranking with "only" 54%, all the others actors are totally shuffled. Safari moved to the 2nd place with 14.50% UC Browser appeared in the top 3 with 8.25% where only 0.10% of Jenkins.io visitors were using UC browser. Firefox moved to the 4th places with only 6.15%. It’s interesting how this "All platform" browser market share from statcounter doesn’t reflect Jenkins.io visitors' habits. (Source)

Now at a first glance, results look more similar to the Jenkins.io visitors; excepted for Chrome and Firefox which shed some percentages in favor of Internet Explorer and Edge. This promotes IE to the 3rd place with 8.86% of usage, and downgrade Safari to the 4th place with 6,44%. Compared to Jenkins.io results, Edge is still at the 5th position. It’s very interesting to see how this ranking is so different from the "All platform" one but follows the same pattern coming from Jenkins.io visitors. This highlights how browser choices are strongly influenced by the platform used and contrary to a general trend, Jenkins.io visitors remain desktop users. This absence of mobile users probably explains why some browsers are so under represented within Jenkins visitors. (Source)

Table 3. Browser Market Share on the Desktop
BrowserUsage

Chrome

63.59%

Firefox

13.75%

IE

8.86%

Safari

5.50%

Edge

4.35%

Conclusion

These observations should be carefully interpreted as it may be overestimated or underestimated (depending on various factors) and doesn’t take into consideration local specificities.

Jenkins.io visitors are a majority of desktop users. They are aligned with the world trends when we talk about desktop browsers however only a minority of them come from a mobile. This is probably because jenkins.io is not (yet) mobile friendly and obsiously this would be a nice improvement to have.

All contributions are welcomed and especially for mobile user experience.

Security hardening: Jenkins LTS 2.107.1 switches XStream / Remoting blacklists to whitelists (JEP-200)

$
0
0

This is a post about a major change in Jenkins, which is available starting from Jenkins 2.102 and Jenkins LTS 2.107.1. This is a change with a serious risk of regressions in plugins. If you are a Jenkins administrator, please read this blogpost andupgrade guidelinesBEFORE upgrading.

I would like to provide some heads-up about theJEP-200 change, which is included into the new Jenkins LTS 2.107.x baseline.

Background

For many years Jenkins used to specifically blacklist certain classes and packages according to known or suspected exploits. This approach has been proven unsustainable due to the risk of deserialization attacks via unknown classes from 3rd-party components, after theSECURITY-429/CVE-2017-1000353 fix in 2.46.2 it was decided to replace blacklists by more restrictive whitelists. In October 2017 Jesse Glick proposed a Jenkins Enhancement Proposal, which finally got accepted as JEP-200.

The change implies a risk of regressions in plugins serializing non-whitelisted Java-internal and 3rd-party classes, and that’s why it is so important to follow the upgrade guidelines for this release.

Current state

JEP-200 was first integrated in Jenkins 2.102 (released in January 2018), and it has got a lot of testing since that. See this blogpost for the original announcement.

Over the last two months we received more than 75 issues from users of Jenkins weekly releases. All these issues have been triaged, and we have released most of the fixes. More than 50 plugins were fixed in total, and many more plugins were updated in order to enable compatibility testing. A significant part of the discovered regressions were caused by real defects which were causing performance and stability issues in plugins. Thanks a lot to all the Jenkins contributors and plugin maintainers who helped deliver timely changes for this effort!

Over last 6 weeks Jenkins weekly releases had positive community ratings, the overall JEP-200 adoption reached ~12% of all Jenkins installations on March 01. All major plugins have been also tested directly or verified in the wild on weekly releases. So we are confident that the change is ready to be released in LTS.

On the other hand, we continue to receive JEP-200 regression reports. They are mostly caused by niche plugins which are not widely used in weekly releases, and unfortunately not all fixes have been released yet (see the Wiki page for up-to-date info). We anticipate more regressions to be reported after the LTS release and broader adoption.

In order to simplify the upgrade to the new LTS baseline, I have prepared some helpful materials together with Liam Newman and Jesse Glick. Below you can find the embedded slide deck and video, or scroll down to see the key information in the text form.

Video:

For Jenkins administrators

Upgrading to a core with JEP-200 requires a special update procedure, which is described below.

Upgrading Jenkins

  1. JEP-200 is not the only major change in 2.107.1, please read the full upgrade guide carefully

  2. If you have a way of testing the upgrade before applying it to production, do it

  3. Back up your instance so you have any easy way of rolling back

  4. Update all affected plugins. See this Wiki page for the list of affected plugins, fix statuses and workarounds

  5. Apply workarounds for non-released patches if needed (see below)

  6. Update to the new version of the Jenkins core

Using backups and staging servers is good advice before any upgrade but especially this one, given the relatively high risk of regression. Due to the nature of the changes, some plugins may refuse to load after the upgrade and cause your Jenkins service to fail to start.

After the upgrade

To the extent that advance testing of the impact of this change on popular plugins has been completed, most users (and even plugin developers) should not notice any difference. Still, it is highly advised to monitor your system after the upgrade, especially the following:

  • Jenkins System log (especially during the startup)

  • Job/Build logs

If you do encounter a log message referencing the https://jenkins.io/redirect/class-filter/ URL, most likely it is a JEP-200 regression. Example:

some.pkg.and.ClassName in file:/var/lib/jenkins/plugins/some-plugin-name/WEB-INF/lib/some-library-1.2.jar might be dangerous, so rejecting; see https://jenkins.io/redirect/class-filter/

If you see this kind of message, we highly recommend reporting it so that it can be investigated and probably fixed quickly.

Reporting JEP-200 issues

Please report any issues you encounter matching the above pattern in theJenkins issue tracker, under the appropriate plugin component. Before reporting please check whether this issue has already been reported.

  1. Add the JEP-200 label

  2. Include the stacktrace you see in the log

  3. If possible, include complete steps to reproduce the problem from scratch

You can find examples of previously reported issues using this query.

Jenkins developers will evaluate issues and strive to offer a fix in the form of a core and/or plugin update. Right after the feature release there will be a special team triaging the reports with high priority See JEP-200 Maintenance plan for more info.

For more details and current status, seePlugins affected by fix for JEP-200.

Applying workarounds

Assuming you see no particular reason to think that the class in question has dangerous deserialization semantics, which is rare, it is possible to work around the problem in your own installation as a temporary expedient. Note the class name(s) mentioned in the JEP-200 log messages, and run Jenkins with the hudson.remoting.ClassFilter startup option, e.g.:

java -Dhudson.remoting.ClassFilter=some.pkg.and.ClassName,some.pkg.and.OtherClassName -jar jenkins.war ...

This workaround process may require several iterations, because classes whitelisted in the workaround may also include fields with types requiring whitelisting.

For plugin developers

If you are a plugin developer, please see the original JEP-200 announcement. That blog post provides guidelines for testing and fixing plugin compatibility after the JEP-200 changes. The presentation above also provides some information about what needs to be tested.

Jenkins community account password audit

$
0
0

Their system looks similar to how publishing of plugins works in the Jenkins project:

  • Accounts are protected by passwords chosen by users.

  • Individual contributors have permission to release the components they maintain.

  • The components they release are used by millions of developers around the world to deliver their software.

In other words, weak passwords are a problem for us just as much as for NPM, and what happened to them could happen to us.

To address this problem, the Jenkins security and infra teams have recently collaborated on a password audit. The audit covered all accounts with permissions to upload plugins and components, and on accounts with other levels of privileged infrastructure access. We ran brute force tools on salted password hashes of those accounts looking for "weak" passwords — passwords present in a set of publicly available password lists we chose for this audit.

We checked the password of every qualifying account for every unsafe password rather than trying to match them to any previous password leaks' email/password pairs. Users with weak account passwords were notified via email a few weeks ago and were asked to change their password to something stronger.

We performed the same checks over the previous weekend, but this time we only checked the passwords of accounts whose passwords were deemed weak during our first check. We then invalidated the password of any account whose password was still not considered "strong" (i.e. their password was unchanged or had been changed to another weak password). Users of those accounts will need to request a password reset before signing in again.

We plan to implement further safeguards, including improving the account management app at https://account.jenkins.io to reject weak passwords. If you’re interested in helping the security team make Jenkins more secure, let us know on the jenkinsci-dev mailing list, or request to join the security team.

Introducing Jenkins X: a CI/CD solution for modern cloud applications on Kubernetes

$
0
0

We are excited to share and invite the community to join us on a project we’ve been thinking about over the last few months called Jenkins X which extends the Jenkins ecosystem to solve the problem of automating CI/CD in the cloud.

Background

The last few years have seen massive changes in the software industry:

  • use of immutable container images for distributing software which are smaller, easier to work with and lead to cheaper infrastructure costs than VMs alone (approx 20% less on average)

  • Kubernetes has become the defacto way of installing, upgrading, operating and managing containers at scale on any public or hybrid cloud

    • 2018 is the year all the major public clouds, operating system vendors and PaaS offerings support Kubernetes natively

    • we now have an open source industry standard for distributing, installing and managing applications on any cloud!

  • increased adoption of microservices and cloud native applications leading to massive increase in the number of components which require CI/CD along with increased release frequency

  • improvements in DevOps practices coming from the community such as the State of DevOps Report which show the approach of high performing teams

    • increasingly many businesses now realise that to compete you have to deliver value quickly via software

    • teams need to become high performing if the business is to succeed

All of this adds up to an increased demand for teams to have a solution for cloud native CI/CD with lots of automation!

Introducing Jenkins X

Jenkins X LogoJenkins X is a project which rethinks how developers should interact with CI/CD in the cloud with a focus on making development teams productive through automation, tooling and DevOps best practices.

Jenkins X is open source and we invite you to give us feedback and to contribute to the project.

Whats the big deal?

For many years Jenkins has been capable of doing pretty much anything in the CI/CD space; the challenge has always been figuring out how to get the right plugins, configuration and code to work together in your Jenkinsfile.

For me the big deal about Jenkins X is as a developer you can type one command jx create or jx import and get your source code, git repository and application created, automatically built and deployed to Kubernetes on each Pull Request or git push with full CI/CD complete with Environments and Promotion via GitOps!

Developers and teams don’t have to spend time figuring out how to package software as docker images, create the Kubernetes YAML to run their application on kubernetes, create Preview environments or even learn how to implement CI/CD pipelines with declarative pipeline-as-code Jenkinsfiles. It’s all automated for you out of the box! So you can focus instead on delivering value!

At the same time, Jenkins X doesn’t hide anything. If you do want to hack the Dockerfile, Jenkinsfile or Helm charts for your apps or their environments then go right ahead - those are all available versioned in git with the rest of your source code with full CI/CD on it all. GitOps FTW!

Jenkins X automates CI/CD and DevOps best practices for you - so you can become a faster performing team! Let your butler do more work for you!

Demo

Here’s a demonstration of Jenkins X running on GKE:


You can check out more demos here.

Jenkins X Features

Now lets walk through the features of Jenkins X that we showed in the demo:

Automated CI/CD Pipelines

  • get a Pipeline automatically setup for you that implements best practice CI/CD features:

    • creates a Jenkinsfile for defining the CI/CD pipelines through declarative pipeline-as-code

    • creates a Dockerfile for packaging the application up as an immutable container image (for applications which generate images)

    • creates a Helm chart for deploying and running your application on Kubernetes

  • ensures your code is in a git repository (e.g. GitHub) with the necessary webhooks to trigger the Jenkins CI/CD pipelines on push events

  • triggers the first release pipeline to promote your application to your teams Staging Environment

Then on each Pull Request:

  • a CI pipeline is triggered to build your application and run all the tests ensuring you keep the master branch in a ready to release state

  • your Pull Request is deployed to a Preview Environment (more on this later)

When a Pull Request is merged to the master branch the Release pipeline is triggered to create a new release:

  • a new semantic version number is generated

  • the source code is modified for the new version (e.g. pom.xml files get their <version> elements modified) and then tagged in git

  • new versioned artifacts are published including:

    • docker image, helm chart and any language specific artifacts (e.g. pom.xml and jar files for Java, npm packages for node or binaries for go etc)

  • the new version is promoted to Environments (more on this later)

Environment Promotion via GitOps

In Jenkins X each team gets their own environments. The default environments are Staging and Production but teams can create as many environments as they wish and call them whatever they prefer.

An Environment is a place to deploy code and each Environment maps to a separate namespace in Kubernetes so they are isolated from each other and can be managed independently.

We use something called GitOps to manage environments and perform promotion. This means that:

  • Each environment gets its own git repository to store all the environment specific configuration together with a list of all the applications and their version and configuration.

  • Promotion of new versions of applications to an environment results in:

    • a Pull Request is created for the configuration change that triggers the CI pipeline tests on the Environment along with code review and approval

    • once the Pull Request is merged the release pipeline for the environment which updates the applications running in that environment by applying the helm chart metadata from the git repository.

Environments can be configured to either promote automatically as part of a release pipeline or they can use manual promotion.

The defaults today are for the Staging environment to use automatic promotion; so all merges to master are automatically promoted to Staging. Then the Production environment is configured to use manual promotion; so you choose when do promote.

However it is easy to change the configuration of how many environments you need and how they are configured via the jx create environment and jx edit environment commands

Preview Environments

Jenkins X lets you create Preview Environments for Pull Requests. Typically this happens automatically in the Pull Request Pipelines when a Pull Request is submitted but you can also perform this manually yourself via the jx preview command.

The following happens when a Preview Environment is created:

  • a new Environment of kind Preview is created along with a kubernetes namespace which show up the jx get environments command along with the jx environment and jx namespace commands so you can see which preview environments are active and switch into them to look around

  • the Pull Request is built as a preview docker image and chart and deployed into the preview environment

  • a comment is added to the Pull Request to let your team know the preview application is ready for testing with a link to open the application. So in one click your team members can try out the preview!

Issue Comment

This is particularly useful if you are working on a web application or REST endpoint; it lets your team interact with the running Pull Request to help folks approve changes.

Feedback

If the commit comments reference issues (e.g. via the text fixes #123) then Jenkins X pipelines will generate release notes like those of the jx releases.

Also, as the version associated with those new commits is promoted to Staging or Production, you will get automated comments on each fixed issue that the issue is now available for review in the corresponding environment along with a link to the release notes and a link to the app running in that environment. e.g.

Issue Comment

Getting started

Hopefully you now want to give Jenkins X a try. One of the great features of Jenkins is that it’s super easy to get started: install Java, download a war and run via java -jar jenkins.war.

With Jenkins X we’ve tried to follow a similarly simple experience. One complication is that Jenkins X has more moving pieces than a single JVM; it also needs a Kubernetes cluster :)

First you need to download and install the jx command line tool so its on your PATH.

Then you need to run a single command to create a new Kubernetes cluster and install Jenkins X (in this example, on GKE).

jx create cluster gke

Today we support creating Kubernetes clusters and installing Jenkins X on Amazon (AWS), Google (GKE), Microsoft Azure, and even locally using minikube. We plan to support AWS EKS soon.

At the time of this writing the easiest cloud to get started with is Google’s GKE so we recommend you start there unless you already use AWS or Azure. Amazon and Microsoft are working hard to make Kubernetes clusters as easy to create and manage as they are on GKE.

All the public clouds have a free tier so you should be able to spin up a Kubernetes cluster and install Jenkins X for a few hours then tear it down and it should be cheaper than a cup of coffee (probably free!). Just remember to tear down the cluster when you are done!


If you really don’t want to use the public cloud, you can install Jenkins X on an existing kubernetes cluster (if it has RBAC enabled!). Or, if you can install and run minikube, then you should be able to install Jenkins X on it as well.

Relationship between Jenkins and Jenkins X

Jenkins is the core CI/CD engine within Jenkins X. So Jenkins X is built on the massive shoulders of Jenkins and its awesome community.

We are proposing Jenkins X as a sub project within the Jenkins foundation as Jenkins X has a different focus: automating CI/CD for the cloud using Jenkins plus other open source tools like Kubernetes, Helm, Git, Nexus/Artifactory etc.

Over time we are hoping Jenkins X can help drive some changes in Jenkins itself to become more cloud native, which will benefit the wider Jenkins community in addition to Jenkins X.

Please join us!

So I hope the above has given you a feel for the vision of where we are heading with Jenkins X and to show where we are today. The project is still very young, we have lots to do and we are looking for more input on where to go next and what to focus on. We’re also working on high level roadmap.

To make Jenkins X a success we’d love you to get involved, try it out and give us feedback in the community! We love contributions whether its email, chat, issues or even better Pull Requests ;).

If you’re thinking of contributing here’s some ideas:

To help get faster feedback we are using Jenkins X as the CI/CD platform to develop Jenkins X itself. For example Jenkins X creates all the releases and release notes. We’ll talk more about UpdateBot in a future blog post but you can see all the automated pull requests generated in the various Jenkins X pipelines via UpdateBot pushing version changes from upstream dependencies into downstream repositories.

Note that the Jenkins community tends to use IRC for chat and the Kubernetes community uses Slack, so Jenkins X has rooms for both IRC and slack depending on which chat technology you prefer - as the Jenkins X community will be working closely with both the Jenkins community and the various Kubernetes communities (Kubernetes, Helm, Skaffold, Istio et al).

One of the most rewarding things about open source is being able to learn from others in the community. So I’m hoping that even if you are not yet ready to use Kubernetes in your day job or are not yet interested in automating your Continuous Delivery - that you’ll at least consider taking a look at Jenkins X, if for no other reason than to help you learn more about all these new ideas, technologies and approaches!

Thanks for listening and I’m looking forward to seeing you in the community.

Evolving Mission of Jenkins

$
0
0

Lately, perhaps subtle but exciting changes are starting to happen in the Jenkins project.

The past few weeks have seen the birth of two new initiatives in Jenkins:Jenkins Essentials andJenkins X. Each is exciting in its own right, and I encourage interested parties to take a look at their goals and missions and participate in them. But in this post, I want to discuss why together these two dots form an important arc, which actually started in the introduction of Jenkins 2 and continued with Blue Ocean.

In Jenkins 2, we changed Jenkins so that it starts with richer functionality and more sensible security setup, among other things. This was the first step in a new direction for Jenkins. We changed our focus from “we’ll write plugins and you figure out the rest” to “we’ll write plugins, we’ll assemble them, and we’ll help you be more productive.”

Blue Ocean was another step on this journey. We focused on important continuous delivery use cases in Jenkins, and aimed to provide a great user-experience for those use cases. Aside from obvious productivity boost for users, it also decidedly blended together feature areas that are internally provided by a whole bunch of different plugins, but users see much less seam between them.

Jenkins Essentials, which R Tyler Croy proposed in recent weeks, is another step forward. That project aims to take an even bigger responsibility in keeping people’s Jenkins instances up and running. Like Blue Ocean, Jenkins Essentials focuses on delivering a comprehensive Jenkins user experience rather than a collection unrelated of plugins which users have to figure out how to wire together. It also creates an exciting vehicle for contributors, in which we can develop and deliver features quite differently, and more rapidly, than how we deliver them today.

Jenkins X, which was proposed by James Strachan a few weeks after Jenkins Essentials, is the latest point on this same arc. Jenkins X brings a different aspect to building a solution — it focuses on a specific vertical area, namely Kubernetes application development, and drastically simplifies the software development in that domain by bringing together Jenkins, a whole bunch of plugins, and an opinionated best practice of how one should use Kubernetes.

Collectively, the arc that these efforts form aims to solve the most important and consistent concerns for Jenkins users — ease of use, plugin complexity, fear of upgrade, etc.

In the early days of Jenkins, it was up to each and every Jenkins admin to find the right way to assemble pieces into a solution for their organizations, but this hard work remained largely private. Now, these newer projects are bringing this back into the community. They are making Jenkins more valuable to existing users, and more approachable and useful to a whole new set of users who are not currently using Jenkins.

From that perspective, I hope more projects like them will follow, pushing us beyond “just writing plugins”, taking even bigger steps to make users productive. This is a little bit like how I watched Eclipse evolve from just a Java IDE to an umbrella of projects.

Exciting times!

Jenkins Essentials: five minutes, five clicks

$
0
0

In his presentation at the 2017 Jenkins World Contributor Summit,Kohsuke challenged us to continue the work started with Jenkins 2 of making Jenkins easier to install and easier to use. "A user should be successful with Jenkins in under five minutes and five clicks." At that same Contributor Summit, a few of us discussed the idea of a distribution which had "batteries included", whichAndrew proudly named "Jenkins Essentials." At the time I was certainly not as excited about the project as I am now, I thought to myself "we built a Setup Wizard in Jenkins 2, nobody needs a Setup Wizard++."

Jenkins Essentials

As Kohsuke and I continued to discuss the idea, more and more ideas came up. Towards the end of 2017 the picture became much clearer: Jenkins Essentials would be a comprehensive, low-maintenance distribution to help new and existing users be successful with Jenkins, without needing to be Jenkinsexperts. This will of course not replace the existing distribution of Jenkins core and its plugins, which allow many of us large amounts of flexibility, but rather it will make Jenkins easier for users who don’t want to "build it themselves."

The more I thought about it, the more excited about the idea I became: Jenkins Essentials could open the door to new improvements and features in Jenkins which had been left in the "idea and design" phase going back almost two years! Really, I checked, some of the concepts adopted into the design of Jenkins Essentials were first conceived of in early 2016!

Kohsuke briefly discussed the project inhis previous blog post but in post I wanted to expand on what Jenkins Essentials is, and ourprogress has been in its development.

What’s in Jenkins Essentials

A few months ago I preparedthis presentation for theFOSDEM 2018 Jenkins Contributor Summit, which outlines the following "pillars" or Jenkins Essentials, which are also described inJEP-300:

Automatically Updated Distribution

In order to provide an easier-to-use and easier-to-manage Jenkins environment, Jenkins Essentials will be distributed as an automatically self-updating distribution, containing Jenkins core and a version-locked set of plugins considered "essential." Rather than attempting to mirror the existing Weekly and LTS release lines for core, plus some plugin version matrix, Jenkins Essentials will update in a manner similar to Google Chrome.

This automatically updating distribution will mean that Jenkins Essentials will require significantly less overhead to manage, receiving improvements and bug fixes without any user involvement. From the user perspective, their Jenkins will appear to automatically improve over time.

There is really interesting work being pioneered byBaptiste Mathus withJEP-302 to ensure that these automatic upgrades can be performed safely.

Automatic Sane Defaults

Providing a core along with "essential" plugins is a good first step to helping Jenkins users successfully automate their CI/CD workloads, but requires additional "smoothing" over some of the numerous options and configurations plugins. Jenkins Essentials will perform some amount of "automatic environment-based self-configuration."

For example, clicking a "Launch Stack" button from the Download page would launch an AWS-flavored Jenkins Essentials which, out of the box attempts to set up AWS-specific configuration with S3 and EC2 services.

Connected

In order to provide a more seamless experience for end-users, and ensure that Jenkins project developers receive useful error and usage telemetry to drive further improvements in Jenkins, Jenkins Essentials must necessarily be viewed as a "Connected" application. This means some yet-to-be-specified number of server-side applications to coordinate updates, receive and process telemetry, broker 3rd-party service authentications, relay webhooks, etc.

Obvious Path to User Success

The final pillar in Jenkins Essentials, is to ensure that Jenkins provides an obvious path for a user to configure and use it successfully. This largely entails in-application documentation, examples, and disabling legacy functionality within the application. All with the end goal of preventing users from inadvertently choosing legacy, or poorly supported, options when configuring their CI/CD workloads.

Progress thus far

Suffice it to say, Jenkins Essentials is a hugely ambitious project! We have been making steady progress however, as you can see in thejenkins-infra/evergreen repository on GitHub. We have been adamantly following theJenkins Enhancement Proposal process, and have been making sure our designs and implementations are clear as we build them. Thus far we’ve written designs and implemented:

Unfortunately we don’t yet have the first parts of the Automatically Updated Distribution working, which means you cannot download Jenkins Essentials today and get started with it. We’re still building the Jenkins-side and server-side components necessary to make the full feedback loop operate, without which we would not be able to safely deliver new upgrades to Jenkins Essentials installations.

If you’re interested in getting involved, you can check out our Gitter channel or ourJira issues board.

Jenkins Essentials is just one major initiative going on in the Jenkins project this year, so I hope you’re as excited as I am for the future of Jenkins!

The new things arriving in Declarative Pipeline!

$
0
0

Last week we released the latest version of Declarative Pipelines, version 1.2.8. With that out, we thought now would be a good time to introduce you to the new features and options that have been added to Declarative since the beginning of 2018. These are all available now in the Update Center, with version 1.2.8.

Accessing the new Declarative Directive Generator

Declarative Directive Generator

This is something we’re really happy about - if you go to the "Pipeline Syntax" link from your Pipeline’s page in Jenkins, you’ll see a couple new links on the left, including "Declarative Directive Generator". The Directive Generator is much like the Snippet Generator that’s been in Pipeline for a couple years now, but where the Snippet Generator is just for filling out a form for a step and generating the Pipeline code that configuration maps to, the Directive Generator is built to help you write your Declarative Pipeline directives, like agent, options, stage, and more!

This is the first release to include the Directive Generator, and it’s definitely going to see more polish going forward, but we think it should be quite helpful for you already. We’ll be putting up another blog post looking at the Directive Generator in more detail in the near future.

New when conditions

We’ve added a number of new when conditions, providing you more control over whether your stages get executed.

  • equals - Compares two values - strings, variables, numbers, booleans - and returns true if they’re equal. I’m honestly not sure how we missed adding this earlier! You can do "not equals" comparisons using the not { equals ... } combination too.

  • changeRequest - In its simplest form, this will return true if this Pipeline is building a change request, such as a GitHub pull request. You can also do more detailed checks against the change request, allowing you to ask "is this a change request against the master branch?" and much more.

  • buildingTag - A simple condition that just checks if the Pipeline is running against a tag in SCM, rather than a branch or a specific commit reference.

  • tag - A more detailed equivalent of buildingTag, allowing you to check against the tag name itself.

In addition, we’ve added a new option to when: beforeAgent. This allows you to specify that the when conditions should be evaluated before entering theagent for the stage, rather than the normal behavior of evaluating when conditions after entering the agent. When beforeAgent true is specified, you will not have access to the agent's workspace, but you can avoid unnecessary SCM checkouts and waiting for a valid agent to be available. This can speed up your Pipeline’s execution in some cases.

Using the new Declarative Directive Generator

New post conditions

The changed condition has always been a bit confusing, and to be honest, it wasn’t our best work. changed will fire any time the current run’s status is different than the previous run’s status - whether the current run is healthier than the previous one, or the other way around. That’s…​not actually very useful. So now we’ve added two new post conditions that should provide you with a lot more value than changed has.

  • fixed - This will check to see if the current run is successful, and if the previous run was either failed or unstable.

  • regression - This will check to see if the current run’s status is worse than the previous run’s status. So if the previous run was successful, and the current run is unstable, this will fire and its block of steps will execute. It will also run if the previous run was unstable, and the current run is a failure, etc.

New options

The options directive in Declarative can contain a number of different kinds of configuration: traditional Jenkins job properties, like buildDiscarder, wrapper steps to execute the entire Pipeline within, like timeout, and Declarative-specific options that can switch from some default behaviors of Declarative execution. We’ve added two new Declarative-specific options in the last few releases.

  • checkoutToSubdirectory - Allows you to override the location that the automatic SCM checkout will use. Using checkoutToSubdirectory("foo"), your Pipeline will checkout your repository to "$WORKSPACE/foo", rather than the default of "$WORKSPACE".

  • newContainerPerStage - If you’re using a top-level docker or dockerfileagent, and want to ensure that each of your stages run in a fresh container of the same image, you can use this option. Any stage without its ownagent specified will run in a new container using the image you’ve specified or built, on the same computer and with access to the same workspace.

Stage options

Sometimes, you may only want to disable automatic checkout of your repository, using the skipDefaultCheckout(true) option, for one specific stage in your Pipeline. Or perhaps you want to have a timeout that covers an entirestage, including time spent waiting for a valid agent, post condition execution, or the new input directive for stages (see further down for more details on that!). To make those things possible, we’ve added a new options direction to stage. You can use a subset of the top-level options content in a stage's options - wrapper steps, and Declarative-specific options that are marked as legal in a stage.

Input

You’ve always been able to run the input step inside a stage's steps block, but we’ve found that approach can lose out on some of the value that the input step provides.

To help with that, we’ve added a new input directive to stage, with the same parameters as the input step. When you use thestageinput directive rather than using the step directly, any parameters you’ve specified for the input will be made available in the stage's environment, meaning you can reference parameters from the input in when conditions, or in environment variables.

Jenkinsfile (Declarative Pipeline)
pipeline {
    agent any
    stages {
        stage('Example') {
            input {
                message "Should we continue?"
                ok "Yes, we should."
                submitter "alice,bob"
                parameters {
                    string(name: 'PERSON', defaultValue: 'Mr Jenkins', description: 'Who should I say hello to?')
                }
            }
            steps {
                echo "Hello, ${PERSON}, nice to meet you."
            }
        }
    }
}

Also, the input directive is evaluated before you enter any agent specified on this stage, so if you are using a top-level agent none and each stage has its own agent specified, you can avoid consuming an executor while waiting for the input to be submitted.

Lastly, you can use timeout in the stageoptions, as mentioned above, to time-out the input if too much time has passed without a response.


I hope you find these new features and options for Declarative Pipelines helpful, and I look forward to the rest of 2018 as we continue to invest and improve in Jenkins Pipeline!


Opinionated Kubernetes and Jenkins X

$
0
0

Irecently wrote about how all the cloud platforms are all in Kubernetes and so are developers. It is an exciting time, but the problem for many is that this is a huge blank sheet of paper for how to build and deploy applications. A white space, a void, a limitless canvas of possibilities. Insert metaphors here.

The problem, as you may guess, is that few people really like or are able to start with a blank canvas. I know I prefer to start with something working and iterate towards a solution, or be given some rails to stay on (again with the metaphors).

That’s where the Jenkins X project comes in. Jenkins X is a Kubernetes-native continuous integration and continuous delivery platform for developing cloud native applications that was recently introduced as aJenkins Enhancement Proposal, sponsored by James Strachan.

There is a lot to take in but at it’s heart, this is an open source opinionated way to do continuous delivery with Kubernetes, natively, without necessarily having to learn all the things I talked about in my last blog post. I shall attempt to explain what this is all about and why it matters to developers. As someone said on the jenkins-dev mailing list “We have the two glued together with baling wire and twine” - Jenkins X aims to simplify how to work with continuous delivery and Kubernetes.

First and most importantly, let’s see the logo:

Jenkins X Logo

You can see the nautical theme leaking through (and Kubernetes). Whilst it is called Jenkins X, it is about quite a lot more than Jenkins.

Jenkins X makes decisions for you

Jenkins X presents itself to you initially as a handy sleek command line (a native binary you can install called jx - the debate is on as to how pronounce it). Let’s take a tour (sail?):

> jx import my-app

If you have an existing project, this will detect what type of project it is, build a pipeline for you (and a bunch of Kubernetes things, like Helm Charts), add it to your project and set it up in GitHub, WebHooks and all, build the project (run the pipeline) and deploy a version to a “staging” environment.

If it looks ok, you can promote it to production:

> jx promote --env production --version 1.0.1 my-app

If something went wrong in production, you can roll back an app to any version (the version numbers are made for you):

> jx promote --env production --version 1.0.0 my-app> jx get apps # list versions

An environment is a well-established concept for web developers using continuous delivery: out of the box Jenkins X makes three of them for you (dev, staging and production), but you can make as many as you like. Environments have rules around how things are promoted into them (and they also have their own extensible pipelines, but you can just use them as-is to start).

You can also create a Spring Boot microservice app:

> jx create spring

Answer a few questions and it will set everything up for you.

Any changes you make to your app automatically are built, and if the build looks good, they go through to the staging environment. WebHooks are setup for you (if you are using GitHub) to smooth this over.

For those looking at starting from pre-made apps, there are "quickstarts":

> jx create quickstart

They are based on a growing set of starter apps, in a variety of languages and tech stacks.

Review apps for your changes: Each pull request is built/tested, and a “review app” is made available in a temporary environment. That means each proposed change, before it goes to the default branch (master), has an environment made (temporary) that it can be tried out in. In GitHub, this shows up as a comment in the pull request:

Comment in a GitHub Pull Request

Project type detection

As you can see, so far there is no editing or manual creation of pipelines, or scripting or setup, just importing or creating your app and go. This is powered byDraft “packs” (a handy project that came out of Azure). What you end up with is a Jenkinsfile in your project repository. You may want to edit it one day, or you may be happy with it as is! Jenkins is famous for being un-opinionated in what you do, but Jenkins X has strong opinions (but you can extend and customise).

image::/images/jenkins-x/draft-logo.png[Draft Logo, width=300]

Deploying or promoting to environments

Deploying happens via pipelines behind the scenes - when a change is pushed, or a version promoted. You don’t need to directly interact with Kubernetes if you don’t need to. A tool called Helm does the heavy lifting: Helm is used to package and perform installations and upgrade for your apps.

Helm Logo

There is a little more magic going on here with environments, which you don’t see at first. Each environment, for a team, is represented by a Git repository behind the scenes. Configuration as code is a well-established best practice these days, so why not use it to track deployments and initiate deployments. I also mentioned in my previous post how declarative Kubernetes is: it is perfect for keeping all config in a repository, of the desired system state.

Each promotion is actually a pull request to a per-environment repository. This repository is made and managed for you (and kept outside of the main application code repository), you don’t have to look at it, but you can extend things there should you need to. Specific environment repositories may have different access rules, or be controlled by a different team (perhaps even deploy to a different cluster). Some have coined the term for this as “GitOps.” I first came across this concept on a WeaveWorks blog.

I’ll try and explain this one with a diagram:

GitOps Workflow

The pipeline is actually split in the middle. On the left is the more familiar continuous integration pipeline. This works on pull requests, pre-release version of things and is all about testing(automated and manual review). The source of truth for this is the configuration in the applications repository: branches, pull requests and so on.

The right-hand side is the continuous delivery pipeline. This kicks in when the application is ready to be updated with a new release. This is the “GitOps” repo behind the scenes that controls the state of things in Kubernetes. A promotion on this side is a pull request, and then a merge, from the staging repository to the production repository.

Installing Jenkins X

The jx command line has a jx install command that installs it into a Kubernetes cluster.

The best experience initially is using Google’s excellent GKE service:

> jx create cluster gke

This will ask a few questions, and go and set it all up for you in a cluster set aside for Jenkins X (recommended). Jenkins X runs entirely as services on top of a Kubernetes cluster.

> jx install

Is designed to work with a Kubernetes cluster (if it already exists, recommendation is to have a cluster put aside for Jenkins X if possible). Amazon EKS support is coming (mostly it is around testing), that service is in beta/early access so it is still a work in progress, as is Microsoft Azures excellent AKS service.

So where is Jenkins?

Good question, thanks for asking. Well, it is behind the scenes. As you have seen, there was no direct interaction with Jenkins, but it is there, running the pipelines for continuous integration and continuous delivery of the respective repositories, and orchestrating things with Kubernetes.

If you run jx get pipelines you can see URLs to the various pipelines that have been setup for you are part of interacting with Jenkins X.

By the way,James Strachan has written an extensive blog on jenkins.io that really explores the Jenkins X project in-depth. Once you finish reading this blog, take a stroll on over there and read James'. He also provides several ways you can get involved in the project.

What else can I do with the command line?

Lots, the jx command line has built in help:

jx open

  • open apps, services or pipelines in your browser

jx activity

  • explains how things got to where they are, a history

jx get environments

  • list environments

jx get apps

  • show the state of applications, what versions are in what environments.

What’s next

There is a whole lot more to this, and lots more moving parts and services that are set up for you that are very useful, but it is best to head over to jenkins-x.io and have a look.

This project is obviously in early stages (it is stll a Draft JEP after all) and there is lots happening. Check out the Jenkins X community if you want to chat on slack, IRC, issues or email. Also, read theJenkins Enhancement Proposal doc.

Security updates for Jenkins core

$
0
0

We just released security updates to Jenkins, versions 2.116 and 2.107.2, that fix two security vulnerabilities.

For an overview of what was fixed, see the security advisory.

Subscribe to the jenkinsci-advisories mailing list to receive important notifications related to Jenkins security.

Jenkins X making awesome progress after 24 days

$
0
0

Its been 24 days since weannounced Jenkins X, a CI/CD solution for modern cloud applications on Kubernetes. I’m truly blown away by the response and feedback from the community - thank you!

We’ve also had lots of folks report they’ve successfully used Jenkins X on a number of clouds including GKE, AWS and AKS along with on premise clusters which is great to hear!

Here’s a brief overview of the changes in the last 24 days from the Roadmap:

  • we now fully support GitHub and GitHub enterprise. BitBucket cloud and gitea is almost there too. Hopefully BitBucketServer and Gitlab are not too far away either. For more detail seesupporting different git servers

  • For issue tracking we support GitHub, GitHub Enterprise and JIRA. For more detail seesupporting issue trackers

  • Gradle support is now available from jx create spring or by importing gradle apps

  • Go, Node and Rust build packs are now available with more planned

  • New addons for anchore and kubeless

Also we’ve made it a little bit easier to keep your jx binary up to date continuously. Just type one of the following:

  • link:http://jenkins-x.io/commands/jx_version/[jx version] will prompt you if there is a new version available and if prompted, it will upgrade itself

  • link:http://jenkins-x.io/commands/jx_upgrade_cli/[jx upgrade cli] will upgrade the jx binary if its available orlink:http://jenkins-x.io/commands/jx_upgrade_platform/[jx upgrade platform] for the platform

For more detail on the changes over the last 24 days with metrics please see thechangelog generated by Jenkins X

We’d love to hear your feedback what you think of Jenkins X and theRoadmap - pleasejoin the community.

Jenkins X Explained Part 1 - an integrated CI/CD solution for Kubernetes

$
0
0

Jenkins X is an opinionated platform for providing CI / CD on top ofKubernetes. We’ve chosen a set of core applications that we install and wire together so things work out-of-the-box, providing a turn key experience. This blog aims to build on previous introductions to Jenkins X and provide a deeper insight to what you get when you install Jenkins X.

key

So what happens? After downloading the jx CLI you will now be able to create clusters with public cloud providers or install onto an existing Kubernetes cluster.

cloud Providers

This command will create a cluster on your cloud provider of choice.

> jx create cluster

Alternatively you can bring your own Kubernetes cluster and install Jenkins X on it:

> jx install

That said, we’ve found that creating a new cluster on a public cloud such as GKE is a lot way easier to start as we can be sure of the state of the cluster. For example we know that storage, networking and loadbalancers will be working as expected. Creating a cluster on GKE takes only a few minutes so it’s a great way to try things out as well as run your enterprise workloads.

For now lets assume we are using GKE. When jx create cluster has finished you will see some output in the terminal that also includes the default admin password to use when logging into the core applications below. There is a flag —-default-admin-password you can use to set this password yourself.

Accessing applications

We automatically install an Nginx ingress controller running with an external loadbalancer pointing at it’s Kubernetes service. We also generate all the Kubernetes Ingress rules using a golang library called "exposecontroller". This runs as a Kubernetes Job triggered by aHelm hook once any application is installed to the cluster. 

Using "exposecontroller" means we can control all the ingress rules for an environment using a single set of configurations, rather than each application needing to know how to expose the kubernetes service to the outside world. This also means we can easily switch between HTTP and HTTPS plus support intregration with projects likecert-manager for auto generation of signed TLS certificates.

Environments

One important point to make is Jenkins X aims to use terminology that developers are familiar with. That’s not to say we are changing Kubernetes fundamentals, it’s more that if you don’t know Kubernetes concepts then we aim to help you still adopt the cloud technology and pull back the curtain as you gain confidence and experience. To that point, a core part of Jenkins X are "environments". An environment can have one or more applications running in it. In Kubernetes term an "environment" maps to the concept of a "namespace" in code.

The installation by default created three environments, this is customisable but by default we have a "dev", a "staging" and a "production environment". To list, select, or switch between these environments run:

> jx env

Jenkins X core applications

In the "dev" environment we have installed a number of core applications we believe are required at a minimum to start folks off with CI/CD on Kubernetes. We can easily add to these core apps using Jenkins X addons but for now lets focus on the core apps. Jenkins X comes with configuration that wires these services together, meaning everything works together straight away. This dramatically reduces the time to get started with Kubernetes as all the passwords, environment variables and config files are all setup up to work with each other.

  1. Jenkins — provides both CI and CD automation. There is an effort to decompose Jenkins over time to become more cloud native and make use of Kubernetes concepts around CRDs, storage and scaling for example.

  2. Nexus — acts as a dependency cache for Nodejs and Java applications to dramatically improve build times. After an initial build of a SpringBoot application the build time is reduced from 12 mins to 4. We have not yet but intend to demonstrate swapping this with Artifactory soon.

  3. Docker Registry — an in cluster docker registry where our pipelines push application images, we will soon switch to using native cloud provider registries such as Google Container Registry, Azure Container Registry or Amazon Elastic Container Registry (ECR) for example.

  4. Chartmuseum — a registry for publishing Helm charts

  5. Monocular — a UI used for discovering and running Helm charts

Helm

We learned a lot in our early days with fabric8 on Kubernetes and there were some projects from the ecosystem that either weren’t around or (at the time) didn’t work with OpenShift, therefore we were restricted when making some design decisions. A couple of years on and now with Jenkins X we were able to look at other OSS projects that have been flourishing, so I was very happy to start looking at Helm. Helm is a package manager for Kubernetes and allows easy installation and upgrades of applications.

It was pretty clear that for Jenkins to evolve and include deployments to the cloud we should embrace Helm and provide an opinionated experience that helps teams and developers. The core applications mentioned above means Jenkins X provides an out of the box integrated CI/CD solution for Helm.

We know that helm has limitations but with the work onHelm 3, the focus of the Kubernetes sig-apps group, the Kubernetes community and investment we see from key organisations such as Microsoft, we feel Helm is currently the best way to install and upgrade applications on Kubernetes.

GitOps

We mentioned earlier that we setup three environments by default. What this means is for the staging and production environments we created:

  1. Kubernetes namespace

  2. An environment resource (CustomResourceDefinition) in the dev environment which includes details of how applications are promoted to it and includes various team settings.

  3. A git repository that we store what applications and their versions should be present in that environment. These are stored in a Helm requirements.yaml file

  4. A Jenkins Pipeline job: explained in more detail below

CI/CD for Environments

Having a Jenkins Pipeline Job for each environment means that Pull Requests to the git repo trigger a CI job. For now that job performs basic validation but in the future will include ‘gates’ to ensure a change to that environment has passed expected checks such as QA tasks, gain enough approvals from the correct people, etc - YES CI for environments!!

Once CI checks have passed the new application or version change can be merged. Only users that have karma can merge the Pull Request and therefore we get RBAC plus traceability for our environment deployments.

This means every application manifest, their version and configuration including storage requirements, resource needs and secrets for your environments are stored in Git repositories. Given a disaster recovery scenario this is exactly what you want.

Did I just say secrets in Git? Yes! We will be providing a nicer experience to helps folks get set up but we ourselves encrypt our secrets and store them in Git, then decrypt them when we come to install and upgrade.

We do all this with the help of a Helm wrapper called helm secrets. I’m working on a followup blog post with examples, better explanations and how to guides + add better integration with JX in the coming weeks.


Fancy getting involved?

We mainly hangout in the jenkins-x Kubernetes slack channels and for tips on being more involved with Jenkins X take a look at our contributing docs

If you’ve not already seen it here’s a video showing the create cluster explained in this blog.

Faster sailing on Blue Ocean 1.5.0

$
0
0

Hello, I am Jenn, the new Product Manager for Blue Ocean and Jenkins Pipeline at CloudBees. I am based out of the Seattle area and am excited to be working on Jenkins. :D We released version 1.5.0 of the Blue Ocean plugin late last week. If you’re using Blue Ocean, you’ll want to grab this update since it includes many improvements and bug fixes!

New Features

Blue Ocean now includes a user interface update to show the downstream jobs launched with the build step (link: JENKINS-38339)

Visualizing Downstream Jobs

With Blue Ocean 1.5.0, users can now Reorder Steps in the Blue Ocean Pipeline Editor simply by dragging and dropping steps to reorder them in the list of steps. (JENKINS-38323)

The "Artifacts" tab also now supports pagination, which allows developers to paginate through the Artifacts tab. Previously, this list was cut off at 100 entries. (JENKINS-43588)

Improvements

We were able to include two performance improvements in 1.5.0 which reduce theDashboard loading time in Blue Ocean:

Support for viewing output for failed runs with no stages is also included in this release. For developers who have no stages/steps defined in their pipelines, they can now see the full log output for failed runs. This update helps with Pipeline debugging in Jenkins. (JENKINS-48074)

Further improving the log output for Pipeline Runs, 1.5.0 also improves viewing of long log output lines with wrapping. Previously, a single, long line of output in the log wouldn’t be fully visible in the log window. (JENKINS-49036)

Fixes

One notable bug fix we addressed in this release was that input steps inpost directives would not properly prompt for input. By fixingJENKINS-49297 users of Declarative Pipeline with Blue Ocean can include input steps in their post directives.


The full detailed change log can be viewed on theBlue Ocean plugin page

Update Your Plugin

Plugin updates in Jenkins are available in the Plugin Manager Update Center. This page includes instructions for using the UI and CLI to update your plugins: https://jenkins.io/doc/book/managing/plugins/.

If you are using the Blue Ocean UI, click Administration in the page’s header to open Plugin Manager.

Installing the primary Blue Ocean plugin will update its dependencies as well.

Providing Feedback

Viewing all 1087 articles
Browse latest View live