Scaling Jenkins at Jenkins World 2016

NPR’s Digital Media team uses Jenkins to build, test and deploy code to various staging and production environments. As the complexity of the software components, environments and tests have grown - both generally and due to our quest to achieve continuous deployment - management of Jenkins has become a challenge. In this talk, we share information about our “JenkinsOps” effort which has allowed us to automate many of the administrative tasks necessary to manage feature code branches, handle deployments, run tests and configure our environments properly.

At Riot Games, we build a lot of software. Come learn how we built an integrated Docker solution using Jenkins that accepts Docker images submitted as build environments by engineers around the company. Our containerized farm now creates over 10,000 containers per week and handles nearly 1,000 jobs at a rate of about 100 jobs per hour. All this is done with readily available, open source Jenkins plugins. We’ll explore lessons learned, best practices and how to scale and build your own system, as well as why we chose to solve the problem this way…and whether or not we succeeded!

In this talk, we’ll show how to use Jenkins Pipeline together with Docker and Kubernetes to implement a complete end-to-end continuous delivery and continuous improvement system for microservices and monolithic applications using open source software. We’ll demonstrate how to easily create new microservices projects or import existing projects, have them automatically built, system and integration tested, staged and then deployed. Once deployed, we will also see how to manage and update applications using continuous delivery practices along with integrated ChatOps - all completely automated!

The Jenkins platform can be dynamically scaled by using several Docker cluster and orchestration platforms, using containers to run agents and jobs and also isolating job execution. But which cluster technology should be used? Docker Swarm? Apache Mesos? Kubernetes? How do they compare? All of them can be used to dynamically run jobs inside containers. This talk will cover these main container clusters, outlining the pros and cons of each, the current state of the art of the technologies and Jenkins support. I believe people will be very interested in learning about the multiple options available.

How can we do it? We start with some real world results realized by Jenkins users who have built large clusters and review how they got there. Next, we will do experiments scaling some individual sub-components of Jenkins in isolation and see what challenges we will face when integrated. The famous large, distributed systems undoubtedly faced problems scaling - and we can learn from them, too. The result will be recipes for building Jenkins clusters with different scaling capabilities. After all of this, you can build the biggest Jenkins cluster in the world…or maybe just make your own Jenkins cluster more efficient.

This session will highlight how Splunk uses Jenkins to provide an end-to-end solution in the development CI system. Attendees will see how test results are delivered to a Splunk indexer, where they can be analyzed and presented in a variety of ways. This session will also include a live demonstration.

Last year, we presented our initial investigations and stress testing as we prepared to deploy a large-scale Jenkins installation at Google. Now, with a year of real-world use under our belts, we’ll discuss how our expectations held up, what new issues we encountered and how we have addressed them.