It seems a bit unreal, but the last JUC agenda for 2015 is now online. Jenkins User Conference U.S. West is the last JUC of the year running from September 2-3 in Santa Clara, CA. So, if you haven't attended JUC yet this year, this is your chance!
Register with a friend to take advantage of the community's 2-for-1 deal and get two tickets for the price of one.
Which talk are you looking forward to most? Check out the agenda and tweet your choice to @jenkinsconf!
There's only a month left until JUC U.S. West on September 2-3! If you're still on the fence, check out the recaps of JUC Europe talks recently posted to the CloudBees blog. These should give you an idea about the kinds of talks you can expect at a Jenkins User Conference:
If you're interested in the upcoming Jenkins UI overhaul, make sure to attend Gus and Tom's talk about it. Don't want to wait until JUC to learn more about this? Follow the discussion on the developers mailing list and contribute through early testing.
This JUC will again have an Ask The Experts booth with several Jenkins experts and developers available there throughout the event. If you want to discuss Workflow with Jesse, or pitch your UI ideas to Gus, this is where you'll be able to do that.
As you may have noticed, our wiki and issue tracker were unavailable from Thursday to Sunday last week. What happened?
We host parts of our infrastructure at the Open Source Lab at Oregon State (OSUOSL), including the databases for these two services. So far, there's no post mortem by OSUOSL (they expect to post one later this week), so we need to piece together what we know.
The databases for the wiki and issue tracker became inaccessible around midnight/12 AM Thu/Fri night (all times UTC). Due to the large number and size of databases on that server, pulling from backups, restoring from backup and replaying the binlogs took them quite a while. During that time, we put up a maintenance screen on the wiki (and messed up the one for Jira, so there were connection timeouts instead).
The databases were back around 3 AM on Sunday. We disabled the maintenance screens around 6 PM later that day.
While this was a rather lengthy outage, it could have been much worse. We lost none of the data, after all. We thank the OSUOSL team for their efforts getting everything back up over the weekend!
A week ago we reported that Uday is looking at organizing a regular Jenkins meet-up in Silicon Valley. This has made a progress since then, and this evening we'll get together to figure out logistics for the first meet-up:
The agenda is:
Uday and I will be there, and Uday told me that he heard from another guy who will join us. If you are around and is willing to come over, we'd love to see you. If you are interested, I'd also encourage you to join the Jenkins events list, where a discussion is happening.
Consider this as a shout out to mobile app developers: You are invited! For the first time, there’s a mobile track at this year’s Jenkins User Conference to discuss the best ways to extend CI/CD to mobile application testing.
As agile practices take hold, enterprises are expecting more collaboration between dev and test teams. Dev teams are doing more testing while QA teams are becoming more skilled at coding. This is happening now, and as a result open-source test automation frameworks like Selenium and Appium are flourishing. At the same time, CI/CD adoption is growing. This is happening more so for web development rather than mobile. It’s no secret that incorporating mobile test automation and CI comes with challenges. Mobile UI testing on real devices is still a manual process for many organizations. Manual testing is perhaps a path of least resistance, but it also commits teams to the longest delivery path. Some argue they lack the environment, resources or skilled people to create test automation. While the argument rages, its clear other teams are solving the challenge. Teams are prioritizing the requirement to build a test framework and aligning disparate tools into an effective CI workflow. Recent webinars with Paychex and RaboBank demonstrate CI/CD best practices can effectively extend to mobile app programs using real devices. Particularly when the lab is moved to the cloud and teams can focus on building robust test automation suites.
But overall, the transition to an agile SDLC for mobile apps is happening too slowly. Yet the mobile market demands constant updates. An essential part of an agile SDLC is utilizing automated testing and continuous integration. To test builds using a CI server requires automation which is key to agile development in a fast-paced mobile world because it allows testing to be done by developers early in the lifecycle.
Extending CI to mobile programs is easy with Perfecto Mobile’s support for open source frameworks such as Selenium Remote Web Driver, Appium and Calabash where existing CI plugins are available. Support for commercial tools like HP UFT is also available. With the Perfecto Mobile Jenkins Plugin, you can perform automated functional testing every build. The result is obvious, discover defects earlier, deliver faster feedback and increase release frequency and, ultimately, have better performing apps.
Learn more about extending your CI practice to mobile projects in our upcoming JUC mobile session: “Fast Feedback: Jenkins and Functional Mobile App Testing Without Pulling Your Hair Out.” The session will share suggested coding practices along with planning guidance on maximizing the quality coverage during daily, nightly and weekly builds.
The Jenkins User Conference US West takes place in Santa Clara, CA on Sep 2-3, 2015.
Stop by the Perfecto Mobile booth and share your story.
This post is by Carlo Cadet, Director of Product Marketing at Perfecto Mobile. If you have your ticket to JUC U.S. West, you can attend his talk "Fast Feedback: Jenkins + Functional & Non Functional Mobile App Testing, Without pulling your Hair out!" on Day 1.
Still need your ticket to JUC? If you register with a friend you can get 2 tickets for the price of 1! Register here for a JUC U.S. West, the last JUC of the year!
Thank you to our sponsors for the 2015 Jenkins User Conference World Tour:
Jesse Glick will host the next office hour this Wednesday, 11 AM PDT on Workflow.
Workflow has been Jesse's project for the last year or so. If you don't know what Workflow is, check out these talks about it from past JUCes:
This will be a developer-focused session on integrating with Workflow. He'll discuss things like how to make sure your plugin can be used as part of workflows, and best practices for extending the workflow DSL. There's already been a session on Workflow in January, but Jesse hasn't been idle, and there's new stuff to share.
Participate in the Hangout on Air or watch live on YouTube.
I recently wrote about the two day outage of our wiki and issue tracker:
While this was a rather lengthy outage, it could have been much worse. We lost none of the data, after all.
OSUOSL have since published their post mortem. I was really wrong about not losing any data:
A further complication was that our backups were pointed at mysql2, which was out-of-date with mysql1, due to the initial synchronization failures. Fortunately, we had the binary logs from the 17th through the 30th. This means that though most data could be restored, some data from between the 15th and the 17th was lost.
For our issue tracker, that means that issues JENKINS-29432 to JENKINS-29468 were lost, as well as comments posted from about July 15 12:20 PM to July 17 2 AM (UTC). We know this thanks to the jenkinsci-issues mailing list where the lost issues and comments can be looked up for reposting.
We unfortunately don't have such a record from our wiki.
After a great event on the East Coast in June, now over to the West Coast for another exciting Jenkins User Conference! I'll be there for JUC West on September 2-3 with the XebiaLabs team, and am looking forward to talking to the Jenkins users, partners, developers and community members that will be coming together.
At JUC East, I talked about the importance of Automated Testing in your Continuous Delivery pipeline, and I was really pleased by the number of interesting discussions and comments that came about as a result.
For JUC West, I'll be taking a broader view, and will talk about building an "Enterprise Continuous Delivery Machine" around Jenkins. I'm going to focus on the challenge of identifying and choosing solutions for the many "adjacent problem spaces" to Continuous Integration that you run into when trying to move to Continuous Delivery: artifact management, feature tracking, environment provisioning, deployment automation, test management, pipeline orchestration, production feedback and more.
We'll discuss some of the options available for each category, with a special focus on app deployment, test result management and pipeline orchestration. We'll also present a couple of real-world Continuous Delivery Machine architectures, and analyze some of the motivations for each organization's choices.
Most of our users use XebiaLabs tools/products in combination with Jenkins to build out their Continuous Delivery stack. If you’re scaling out your Jenkins usage too, stop by the XebiaLabs booth to see if you can pick up some tips and to say hello.
Look forward to seeing you at the event, or check the slides or recording we will post after the event. Hope to see you there!
This post is by Andrew Phillips, VP, Product Management at XebiaLabs. If you have your ticket to JUC U.S. West, you can attend his talk "Sometimes Even the Best Butler Needs a Footman: Building an Enterprise Continuous Delivery Machine Around Jenkins" on Day 1.
Still need your ticket to JUC? If you register with a friend you can get 2 tickets for the price of 1! Register here for a JUC U.S. West, the last JUC of the year!
Thank you to our sponsors for the 2015 Jenkins User Conference World Tour:
The Jenkins Newsletter is out a bit early this quarter. If you are not signed up to receive it via email, check out Volume 9 here.
You will be connected to all sorts of Jenkins resources from Jenkins trainingsessions, to some Jenkins User Conference news, to how Jenkins works with Kubernetes and Docker.
I hope that you enjoy this issue! Please let me know what content you find to be the most useful, reach out to me with content that you would like to see in the next issue, and feel free to tell me how I can improve the Jenkins Newsletter: Continuous Information. You can reach out to me at continuous-information@cloudbees.com. Thanks!
Nicolas De Loof will host an office hournext Wednesday 11 AM PDT on integrating Kubernetes with Jenkins. Kubernetes is an open-source project by Google that provides a platform for managing Docker containers as a cluster.
During this session, Nicolas will introduce Kubernetes, explain how it can benefit Jenkins and demonstrate the Kubernetes Plugin. Then he will discuss the design of the Kubernetes plugin and plans he has for future improvements.
Participate in the Hangout on Air or watch live on YouTube.
My greatest surprise at JUC 2014 in Boston was how many enterprise Jenkins CI users had developed plugins for their own use. I had not pictured enterprise release managers as plugin developers. Here at Black Duck Software, we developed Jenkins plugins for four years running. Fabrice Solami, a sales engineer, wanted to do more than automate our code scanning tool via a shell script step in the Jenkins job. He wrote a first plugin that added a build step to run the tool and configure the job more comfortably. The plugin became quickly popular, and when customers asked for it to also support maven builds and run on slaves, it was time for help from the engineering team, particularly the integration team I lead.
Over the years we developed four more plugins and overhauled the original one with the user community (aka paying customers) growing to >75 organizations, mostly large or really large development organizations. In the process, we received lots of feedback and discovered some Jenkins features we feel are essential for good plugin design for the enterprise. Let me share these insights so that you can consider them in your plugin development.
Our plugins connect to our web applications and need authentication to utilize our SDK. The first plugins used username and password fields in every job configuration. That made tedious configuration work and stores the passwords in clear text in the configuration files on disk. Ouch!
We did wise up and started using the credentials plugin to manage username/passwords centrally and securely. It even allows setting authorization roles in such a way that the maintainer of a job can use the credentials without seeing the password. With that in place, our plugins are fit for banks and insurance companies and any other security-conscious organization.
Did you know that Jenkins talks REST? We found it to be an easy way to create and update jobs. It is a really handy tool. The REST API is easy to script for all sorts of external interactions.
However, plugins need to do a little effort to support it on their part; yet it is almost trivial to do. So from our experience it should not be left out.
We wrote a small Java program that reads, creates, updates job configurations, and can trigger job runs. It reads the jobs and commits them to a file for easy mass editing and updates the jobs afterwards.
Our internal use case is to manage regression tests. We have medium-sized lists of jobs that run regression tests. With this tooling we can create a new set of jobs for a given plugin that runs against a new target server, that is, a server version under QA. It all happens in less than 15 minutes.
We also made this part of our migration from our first plugin to its successor with all the enterprise capabilities, but incompatible configuration. Using the REST API and some more Java programs we can create a csv file / Excel spreadsheet with jobs that are configured with the previous plugin. The user can filter the list with the spreadsheet application as needed, and then use the resulting list as input to the batch upgrade tool. This makes the upgrade a gradual affair and not a tedious exercise in UI configuration changes.
If you are developing plugins for in-house use, you have the option to install/update those through file upload. However, in an enterprise you likely have multiple Jenkins servers for different divisions, development groups, or regions. The notification of updates becomes tedious at best. Wouldn’t it be nice to run your own update site, so that your plugin(s) become discoverable in the “Available” tab of the “Manage plugins” screen? Wouldn’t it be a dream if available new versions show up automatically in the “Updates” tab, including Jenkins version compatibility check?
UpdateSites Manager plugin by IKEDA Yasuyuki is the answer to your prayers. It is easy to install, and the process to create and publish an update site is not too complicated and can become part of your Jenkins job building/releasing the plugin.
In my presentation at JUC 2015 West, I’ll share more details on how this makes a difference and how you can use these techniques to make your plugins enterprise-grade. As a bonus, I’ll show you how to get a free vulnerability report for your Maven or Gradle builds.
This post is by Kaj Kandler, Integration Manager atBlack Duck Software, Inc. If you have your ticket to JUC U.S. West, you can attend his talk "Making Plugins that are Enterprise Ready" on Day 1.
Still need your ticket to JUC? If you register with a friend you can get 2 tickets for the price of 1! Register here for a JUC U.S. West, the last JUC of the year!
Thank you to our sponsors for the 2015 Jenkins User Conference World Tour:
We're currently setting up a program to support community members' travel to Jenkins community events. Our goal is to enable more members of the community to meet each other and exchange ideas in person.
We're still hashing out the details, but it'll be available to every Jenkins community member. Apply, telling us what Jenkins-related event you'd like to attend and how awesome you are, and we may support your travel with up to 500 USD. For details on how this will work, see the current draft of the travel grant program.
The first person to be supported in this way is Pradeepto Bhattacharya from Pune, India. He was a speaker at this year's JUC Europe in London, and will give twotalks at JUC US West next week—and we help him get there! He asked us a few weeks back whether the Jenkins project could support his trip to the US. We came to the conclusion that this would be a good idea—so good in fact, that we decided to build a regular program from it.
Are you planning to attend a JUC or similar event, but worry about the cost of travel? We may be able to help you out!
There’s a lot of discussion lately around Cloud Native. If this term is new to you, I suggest a quick read of Cloud Native: What it Means and Why it Matters? From my perspective, Cloud Native offers tremendous benefit to enterprise companies, startups and developers looking to add value quickly or capture market share. Cloud Native platforms, such as Cloud Foundry, provide a number of features to reduce the effort of developing software and operating it on or off premise. A few notable features include load balancing, application routing, cluster scheduling, and containerisation. Cloud Native also offers a significant advancement for building integrated pipelines to deliver software. Before we discuss these advancements, let’s consider the role of the container.
One of the most influential components of Cloud Native is the container. At this point, containers are fairly ubiquitous and most developers have experimented or used containers. For instance, if you've pushed an application to Cloud Foundry or Pivotal Web Services, you’ve used an container without knowing it.
Initially containers were a place to automate the deployment and execution of your code, but over time customization became necessary to handle specific use cases. As a result, container creation now occurs earlier in the development and build phase. As applications are packaged within binaries and containers, validation of the application and container configuration needs to be validated before leaving the developer’s laptop. So what does this mean for the continuous delivery (CD) pipeline?
Developers will tell you their role has expanded over the years as agile methodologies have changed the way software is engineered. Techniques like Test Driven Development (TDD) and CD pipelines encourage software teams to deliver higher quality code in every build. Of course, a good CD pipeline starts at the developer’s laptop. Building and testing the start of a pipeline requires the correct tools while preserving the developer’s choice of container.
The diagram below demonstrates a simple CD pipeline. As you can see, the pipeline starts from the developer’s IDE and uses Cloud Foundry’s Latticeto provide a sandbox to validate the delivery artifacts. Lattice, based on Cloud Foundry’s container scheduler, delivers a small Cloud Native Platform that can be scaled up in the cloud or scaled down to a laptop. It includes a cluster scheduler, HTTP load balancing, log aggregation and health management for containers. Best part, it offers developer choice. Lattice provides support for both Docker containers and Cloud Foundry buildpacks.
Lattice’s flexibility makes it extremely easy to test how the application, which runs in a Docker container, will function in a Cloud Native environment. It’s also extremely helpful for developers engaged in a spike (prototype phase) where they want to push, validate and demonstrate code and let the platform handle the container creation, runtime environment and deployment artifacts via Cloud Foundry buildpacks.
Extending the CD pipeline beyond the developer’s laptop to deliver value to the organization will require additional tools like the CloudBees Jenkins Platform, Artifactory and Pivotal Cloud Foundry. These enterprise build-and-deploy solutions help developers deliver to a Cloud Native platform and reduce the time to establish the feedback loop. If the enterprise maintains a Hybrid cloud strategy, these tools make it seamless to deploy across different cloud providers.
As developers build more Cloud Native applications for Cloud Native platforms, it’s important to establish good tool chains and best practices early in the development phase. Interested to see these tools in action? Join us at Jenkins User Conference West on September 2nd to learn how I use these tools to build Integrated Deployment Pipelines with Jenkins and Cloud Foundry.
This post is by Jamie O'Meara, Field Engineer at Pivotal. If you have your ticket to JUC U.S. West, you can attend his talk "An Integrated Deployment Pipeline with Jenkins and Cloud Foundry" on Day 1.
Still need your ticket to JUC? If you register with a friend you can get 2 tickets for the price of 1! Register here for a JUC U.S. West, the last JUC of the year!
Thank you to our sponsors for the 2015 Jenkins User Conference World Tour:
Last year's JUC West 2014 was packed with good gems of information – such as "how we did it" talks where the speakers shared their points of view on the tools they use for automating their pipeline. At JUC and other conferences I especially seek out talks about how others implement their Continuous Delivery processes. At the upcoming JUC West 2015, it is my turn to share “how we did it” at Perforce. I will present my talk "Continuous Delivery: Driving Lessons” and describe our journey, the rewards we reaped, and the challenges we faced along the way.
At Perforce, we see Continuous Delivery as taking the proven technique of automation and expanding it to a solid set of practices that make the pipeline even more efficient. This includes empowering the product teams to own production and quality all the way from requirements to delivery, and moving from a central build and release team to a self-serve infrastructure to remove the "friction" in the workflow. These changes have allowed us to quickly, efficiently and reliably adapt our software in line with user feedback, shifts in the market, and changes to the business strategy.
I look forward to seeing you there!
This post is by Laurette Cisneros, Engineering Tools Manager at Perforce Software. If you have your ticket to JUC U.S. West, you can attend her talk "The Road to Continuous Delivery: Driving Lessons" on Day 1.
Still need your ticket to JUC? If you register with a friend you can get 2 tickets for the price of 1! Register here for a JUC U.S. West, the last JUC of the year!
Thank you to our sponsors for the 2015 Jenkins User Conference World Tour:
Most Jenkins masters with a distributed build configuration will leverage nodes that run a slave.jar to start a slave agent. Regardless of whether the slave.jar is launched through a Java Web Start or SSH launcher, the jar will be copied from http://yourserver:port/jnlpJars/slave.jar to the build node. Keeping this jar up to date ensures that it picks up the newest features in a more recent release, such as the self-restart feature to keep slave JVMs “clean” and to automatically reconnect to their master. Additionally, newer versions of this component may fix bugs or implement newer protocol versions with various improvements.
Launch methods designed to pull the latest slave.jarare not always reliable and some launch methods don’t even try to update the slave.jar. Therefore it can be useful to see what slave.jar version is running on a given build node and take offline any nodes which fails to update to the latest version of the jar.
The Version Column Plugin allows Jenkins masters to do just this, adding a new column to the “Manage Nodes” view and a new option for version enforcement on the node configuration screen.
After installing the Version Column Plugin, navigate to the list of nodes in your Jenkins instance by clicking Build Executor Status in the executors widget below the side panel on the Jenkins home page.
If the plugin installed successfully, you will see a new column simply called “Version”. This column displays the version of the slave.jar that each build node is using.
This column is simply displaying the versions, so enforcement of slave.jar versions will need to be configured elsewhere. To activate this, click on the “Configure” link in the node manager’s left-hand menu.
You will then see a set of options for slaves. To activate version enforcement, check the “Version” box and apply your changes.
When you update Jenkins, there’s a chance it’ll come with a new version of slave.jar. Now if the slave.jar on a particular slave doesn’t get updated automatically, the master will take it offline and show a warning next to the out-of-date slave’s version number:
The Version Column Plugin is available for download in the Jenkins plugin manager or from its wiki page.
Just as in past years, we are running a survey this year, to get some objective insights into what our users would like to see in the Jenkins project. Obviously, the developers in the project deal with individual bug reports and feature requests all the time, but sometimes those day-to-day issues distract you from a bigger picture.
This year, we kept some of the questions the same, so that we can see the trend over time. But we also wanted to bring in some questions around how you are using Jenkins and what other technologies you leverage such as Linux containers and cloud services.
The survey will close at the end of September and, if you participate, you'll get to see the results first. CloudBees is sponsoring the survey and as an added incentive for us to fill it out, CloudBees has pitched in a $100 Amazon gift card (thanks CloudBees!). Information you submit is only going to be used by the community and not by CloudBees. So please take the survey and let your voice be heard.
A few years ago, the Jenkins community announced the Jenkins CIA program - the Continuous Integration Ambassador initiative to spread the word of Jenkins. As of recently, there hasn't been as much activity, so this program needs to be revived!
There are over 120,000 active Jenkins installations now and that number just keeps climbing and climbing. It's important to bring all of us together through big events like the Jenkins User Conference, but not everyone can get there. That is why Meetups and smaller Jenkins events are crucial.
To support this effort, CloudBees has announced that they will be sponsoring the kickoff of the CIA revival/JAM to help the Jenkins community host these Meetups!
To kick this off, the first Jenkins Area Meetup (JAM) is in the San Jose CloudBees office on Sept 23. We are shooting to have a JAM everything 3rd Wednesday of every month to consistently bring the community together.
Boy, what a day! This is the 5th annual JUC in San Francisco bay area, and the crowd is getting bigger.
I brought the LEGO Jenkins + CloudBees logo mosaic that we built at the CloudBees San Jose office:
The community booth was very busy. We have people like Dean Yu (board), Andrew Bayer (board), Mark Waite (git), Jesse Glick (workflow and core), Daniel Beck (core), Vincent Latombe (literate), Steven Christou (subversion) and Owen Mehegan (community outreach) talking to people all day long.
If you are here, make sure to stop by, and if you are not, follow news with #jenkinsconf.
Gus Reiber will host this week's office hour on Wednesday, 11 AM PDT. He'll talk about some of the UI/UX improvements in Jenkins that he's working on, and will answer your questions about it.
He's already given several talks about this, so you can check these out to learn more before the office hour:
There are also some mailing list threads where he's discussing his designs with the community:
The links to the Google Hangout (participate) and Youtube (watch live) will be posted to the wiki before the event. If you don't get into the Hangout (limited number of participants), don't worry: You'll be able to send questions and suggestions to his Twitter account @gusreiber.
Update: This week's office hour has been canceled.
This Wednesday, Sep 23, at 11 am PDT I will host another office hour on Stapler, the web framework used in Jenkins. This time, I'll show you how structured form submission in Jenkins works, and how Stapler can help you with it.
As usual, the office hour will use Hangout on Air, and a limited number of people will be able to join and participate. The others will be able to watch the office hour live on YouTube. Links to participate and watch will be posted before the event on the Office Hours wiki page.
Update: This week's office hour has been canceled.
