| This is a guest post by Liam Newman, Technical Evangelist at CloudBees. |
One of the great features of Jenkins is how far it can scale, not only from a software perspective, but also from an organizational one. From a single Jenkins master with one or two agents to a multiple master with thousands of agents, from a team of only a few people to a whole company with multiple disparate departments and organizations, you’ll find space where Jenkins is used.
Like any software or organization, there are common challenges for increasing scale with Jenkins and some common best practices, but there are also some unique solutions. A big conference likeJenkins World brings users from all scales together to see how people in other organizations at similar or greater scale are solving similar problems.
As I notedbefore,CloudBees recorded all the sessions and will eventually post them (editing takes time). I’ll post an update when they become available. In the meanwhile, here’s a recap of the some of the sessions related to scaling Jenkins, with links to slides:
Paul Miles andGrant Dickie ofNPR talked aboutJenkinsOps: An Initiative to Streamline and Automate Jenkins. They shared ways their team has used Jenkins to automate many of the administrative tasks related to managing feature code branches, handling deployments, running tests, and properly configuring their environments. They also showed code samples and talked about future challenges in their quest to achievecontinuous deployment.
Maxfield F Stewart ofRiot Games showed how they built an integrated Docker solution using Jenkins inThinking Inside the Container: A Continuous Delivery Story He showed how their system allows engineers around the company to submit Docker images as build environments. This has let their containerized farm now create over 10,000 containers per week and handles nearly 1,000 jobs at a rate of about 100 jobs per hour. And they have done this using readily available, open source Jenkins plugins. He also talked about how they settled on this design, lessons learned, best practices, and how to build and scale other similar system.
How to Do Continuous Delivery with Jenkins Pipeline, Docker and Kubernetes, presented byJames Strachan ofRed Hat, showed how to use Jenkins Pipeline with Docker and Kubernetes to implement a complete end-to-end continuous delivery and continuous improvement system using open source software for both microservices and monolithic applications. He demonstrated how to create or import projects, and have them automatically build, run system and integration tests, stage, and finally deploy. He also showed to manage and update those deployed applications using continuous delivery practices.
Carlos Sanchez ofCloudBees discussedScaling Jenkins with Docker: Swarm, Kubernetes or Mesos? He compared various Docker Swarm, Apache Mesos, and Kubernetes in terms of their ability to dynamically scale in Jenkins by running jobs inside containers. He also discussed the pros and cons, best practices, level of Jenkins support for each of these technologies.
Stephen Connolly ofCloudBees asked "So, You Want to Build the World’s Biggest Jenkins Cluster?" and explained how to do so. He started with real world results realized by Jenkins users who have built large clusters. Next, he showed experiments around scaling some individual sub-components of Jenkins in isolation to see what challenges have been faced when integrated. Finally, he arrived at recipes for building Jenkins clusters with different scaling capabilities and making existing Jenkins clusters more efficient.
Bill Houston andAli Raza ofSplunk gave a talk in two parts,Jenkins at Splunk and Splunking Jenkins In the first part, Bill showed how Splunk uses Jenkins to implement their end-to-end CI system. They discussed features and design goals, challenges they encountered, and how they addressed these challenges. In the second part, Ali showed how to use the Jenkins Splunk plugin. Using plugin, he gathered test results and Jenkins environment data, and delivered it to a Splunk indexer for analysis and presentation.
David Hoover ofGoogle talked aboutJenkins inside Google. Last year, theypresented their initial investigations and stress testing as they prepared to deploy a large-scale Jenkins installation at Google. Now, with a year of real-world use under their belts, they returned to present on how their expectations held up, what new issues they encountered, how they have addressed those issues, and the challenges and opportunities they see ahead.