Jenkins in Google Summer of Code 2018. Results

This year we started preparing for Google Summer of Code in early December. 14 project ideas and 12 potential mentors we published on our website, and we got dozens of students reaching out to us during the application period. After processing applications, we have selected 4 applications for GSoC. Unfortunately one project got cancelled due to student eligibility issues.

So, we had the following projects: Code Coverage API plugin, Remoting over Apache Kafka, and Simple Pull-Request Job Plugin (also known as Pipeline as YAML). All these projects have a significant value to the Jenkins community. They were focused on areas which have been discussed in the community for a long time, but which had no progress so far. Google Summer of Code allowed us to kick-start these projects, and to make significant progress there. All projects have been released and made available in the Jenkins community (common or experimental update centers).

In total there were 9 blogposts about GSoC projects on jenkins.io, and also 2 Jenkins Online Meetups. GSoC results have been also presented at DevOps World - Jenkins World conference and the contributor summit.

There are many code coverage plugins in Jenkins: Cobertura, JaCoCo, Emma, etc., etc. The problem with these plugins is that each of them implements all code coverage features on their own. So you get different feature sets, UIs, CLI commands and REST APIs. The idea of this project was to unify the existing functionality and offer a new API plugin which other plugins could extend. It would help to simplify existing plugin and to create new plugins for coverage tools.

The project has started really well, and we had the first demo after a week of coding. Then Shenyu continued extending the plugin’s functionality over coding periods. Here is the list of the key features offered by the plugin:

In addition to the Code Coverage API Plugin, Shenyu added integration to the Cobertura Plugin and also created a new llvm-cov plugin which is expected to be released soon.

After GSoC Shenyu continued contributing to the Jenkins project. He works on the Code Coverage API plugin and also participates in the Chinese Localization SIG.

This project focused on introducing a way to easily define pull-request build job definitions in YAML. This project has been shaped a lot during the application period and community bonding, so that the project fit the existing Jenkins Ecosystem better. Finally it was decided to build the new plugin on the top of Pipeline: Multi-Branch Plugin. There was also an idea to offer extra syntax sugar, templating and automatic resolution for common flows, so that users need less time to define Pipelines for common use-cases.

The plugin allows defining Pipeline jobs as YAML being stored in SCM. Original design presumed a new job type, but during community bonding and Phase 1 prototyping it was decided to build the plugin on the top of the existing Pipeline ecosystem and extension points. Currently the plugin generates Declarative Pipeline code from YAML so that it gets a lot of Pipeline features out-of-the box. In addition to that, Simple Pull Request Job Plugin uses a an engine provided by the Configuration as Code plugin to convert YAML snippets to Pipeline step definitions.

The plugin has been well described by Abhishek in his Pipeline as YAML blogpost in August. Currently it is available in the Experimental Update Center as an alpha version. Pham Vu Tuan, one of our GSoC students, have also joined the plugin team. At the DevOps World - Jenkins World hackfest we had discussions with the Jenkins Pipeline team, and we have a plan towards making this plugin available as an Incubated Pipeline project. The final implementation may change, but in any case the project gave us a working prototype and a lot of information about obstacles we need to resolve.

Last but not least, Remoting over Kafka is another challenging project we had. To implement communication between its masters and agents, Jenkins widely uses home-grown protocol implementations based on TCP (JNLP 1..4 protocols). There are some performance and stability implementations, and there have been discussions about using an industry-standard message bus or queue. Pham Vu Tuan proposed to use Apache Kafka for it, and after some experiments during community bonding and first coding phase we agreed to go forward with this implementation.

During his project Vu Tuan extended Jenkins Core and Remoting to allow implementing an agent communication channel in a plugin. Then he has created a new Remoting over Kafka plugin which is now available in the main Jenkins Update cente. Once the plugin is installed, it is possible to connect to agents over Apache Kafka and execute all types of Jenkins jobs there. There are also official jenkins/remoting-kafka-agent images available on DockerHub.

Vu Tuan continued contributing to the Jenkins project after GSoC, currently he maintains the Remoting over Kafka plugin. He visited the DevOps World - Jenkins World US conference in September, presented his GSoC project at theJenkins Contributor Summit. You can find his slides here. After the conference he also participated in the hackfest where he helped to migrate Jenkins' DNS services to Microsoft Azure.

After the end of GSoC we had a Retrospective with GSoC students and mentors. We discussed the issues we encountered during the projects, and ways to improve the student and mentor experience.

Main takeaways for us:

For me personally the main takeaway is also to reduce direct involvement into the project as a mentor and technical advisor. Doing org administration, logistics and mentorship is not good from a bus factor PoV, and I believe I was pushing my vision too hard in few cases. Will do my best to prevent it next year.

If you want to share your feedback and ideas, please reach out to us using the GSoC mailing list.

In order to improve GSoC organization in Jenkins, we have have created a GSoC Special Interest Group which will be running non-stop as other SIGs in Jenkins. The objective of the SIG is to organize GSoC, work with potential students/mentors, and to help students stay involved in the community after GSoC ends. In this SIG we will have monthly meetings to sync-up on GSoC. If you are interested to contribute, please join the SIG.

According to the Retrospective, next year we plan to invest more into communication with mentors. We will also try to tie new project proposals to JenkinsSpecial Interest Groups so that the students become a part of ongoing coordinated efforts. This weekend Martin d’Anjou, Jeff Pearce and me are participating in the GSoC Mentor summit to share experiences and to study from other GSoC organizations. On October 17 we will have a GSoC SIG meeting to discuss our experience and to discuss next steps.

In addition to that, Jenkins Google Summer of Code will be presented at DevOps World - Jenkins World Nice and at the contributor summit. If you plan to visit the conference and you are interested to participate in Google Summer of Code and other community activities, please join us at the contributor summit or stop by at the community booth.

And, elephant in the room…​ GSoC 2019. Of course we are going to apply, stay tuned for new announcements. We have already started collecting project ideas for the next year. If you are interested to participate as a student or mentor, please reach out to us using the GSoC SIG mailing list.