We are announcing theSCM API 2.0.x andBranch API 2.0.x release lines.
Downstream of this there are also some great improvements to a number of popular plugins including:
There are some gotcha’s that Jenkins administrators will need to be aware of.
Always take a backup of your JENKINS_HOME before upgrading any plugins. |
We want to give you the whole story, but the take home message is this:
When updating theSCM API and/orBranch API plugins to the 2.0.x release lines, if you have any of theGitHub Organization Folders,GitHub Branch Source and/orBitBucket branch source plugins installed then you must upgrade them all to 2.0.x at the same time or Bad Things™ will happen.
Do NOT upgrade some of these plugins but not others! Doing so may cause your jobs to fail to load. |
If you don’t care about the hows and whys, you can just skip down to this section but if you are curious… here we go!
The back-story
Way back in September 2013 we announced theLiterate plugin, as an experimental new way of modeling branch development in Jenkins.
When you are performing an experiment, the recommendation is to do just enough work to let you perform the test. However, the culture in Jenkins is to always try and produce reusable components that others can use in ways you have not anticipated.
So when releasing the initial version of theLiterate plugin we also separated the Literate specific bits from the SCM specific concepts and multi-branch concepts. These were lower level concepts were organized into the following plugins:
SCM API - which was intended to be a plugin to hold a next generation API for interacting with source control systems.
Branch API - which was intended to be a plugin to hold the multi-branch functionality that was abstracted from the usage by the Literate plugin.
In addition, we released updates to three of the more common SCM plugins which included implementations of the SCM API:
Git plugin
Subversion plugin
Mercurial plugin
While there was some interest in the Literate plugin, it did not gain much traction - there are only 39 Jenkins instances running the Literate plugin as of December 2016.
In terms of the reusable components, we had only made a minimal implementation with some limitations:
Very basic event support - events can only trigger a re-scan of the entire repository. This was acceptable at the time because the only three implementations use a local cache of the remote state so re-scanning is quick.
No implementation of the
SCMFileSystemAPI. As a result it is not possible for plugins likePipeline Multibranch to get theJenkinsfilefrom the remote repository without needing to checkout the repository into a workspace.No documentation on how plugin developers are supposed to implement the SCM API
No documentation on how plugin developers are supposed to consume the SCM API (if they wanted to do something like Branch API but not the same way as Branch API)
No documentation on how plugin developers are supposed to implement the Branch API to create their own multi-branch project types
No documentation on for users on how the Branch API based project types are expected to work.
Roll forward to November 2015 and Jenkins Pipeline got a release of thePipeline Multibranch. It seems that pairing Pipeline with Branch API style multi-branch is much more successful than Literate - there are close to 60,000 instances running the pipeline multi-branch plugin as of December 2016.
There also were two new SCM plugins implementing the SCM API:
GitHub Branch Source Plugin
BitBucket Branch Source Plugin
Unlike the previous implementations of the SCM API, however, these plugins do not maintain a local cache of the repository state. Rather they make queries via the GitHub / BitBucket REST APIs on demand.
The above design decision exposed one of the initial MVP compromises of the SCM API plugin: very basic event support.
Under the SCM API 1.x model, the only event that an SCMSource can signal is something changed, go look at everything again.
When you are accessing an API that only allows 5,000 API calls per hour, performing a full scan of the entire repository just to pick up a change in one branch does not make optimum usage of that 5,000 calls/hour rate limit.
So we decided that perhaps the SCM API and Branch API plugins have left their Minimum Viability Experiment state and the corresponding limitations should be addressed.
Enter SCM API 2.0.x and Branch API 2.0.x
So what has changed in theSCM API 2.0.x andBranch API 2.0.x release lines? These plugin releases include:
documentation on how plugin developers are supposed toimplement the SCM API
documentation on how plugin developers are supposed toconsume the SCM API (if they wanted to do something like Branch API but not the same way as Branch API)
documentation on how plugin developers are supposed toimplement the Branch API to create their own multi-branch project types
generic documentation for users onhow Branch API based project types are intended to work
a full featuredevent system that allows implementers to provide fine grained notifications to consumers
a mock implementation of the SCM API to help consumers of the SCM API test their usage.
In addition, we have upgraded the following plugins to include the new fine-grained event support:
Git Plugin
Mercurial Plugin
Ok, that was the good news. Here is the bad news.
We found out that the GitHub Branch Source and BitBucket Branch Source plugins had made invalid assumptions about how to implement the SCM API. To be clear, this was not the plugin developers fault: at the time there was no documentation on how to implement the SCM API.
But fixing the issues that we found means that you have to be careful about which specific combinations of plugin versions you have installed.
- SCM API Plugin
Technically, the 2.0.x line of this plugin is both API and on-disk compatible with plugins compiled against older version lines.
However, the 1.x lines of both the GitHub Branch Source and BitBucket Branch Source plugins have hard-coded assumptions about internal implementation of the SCM API that are no longer valid in the 2.0.x line.
If you upgrade to SCM API 2.0.x and you have either the GitHub Branch Source or the BitBucket Branch Source plugins and you do not upgrade those instances to the 2.0.x line then your Jenkins instance will fail to start-up correctly.
The solution is just to upgrade the GitHub Branch Source or the BitBucket Branch Source plugin (as appropriate) to the 2.0.x line.
If you upgrade the SCM API plugin to the 2.0.x line and do not upgrade the Branch API plugin to the 2.0.x line then you will not get any of the benefits of the new version of the SCM API plugin.
- Branch API Plugin
The 2.0.x line of this plugin makes on-disk file format changes that mean you will be unable to roll back to the 1.x line after an upgrade without restoring the old data files from a back-up. Technically, the API is compatible with plugins compiled against older version lines.
The 1.x lines of both the GitHub Branch Source and BitBucket Branch Source plugins have implemented hacks that make assumptions about internal implementation of the Branch API that are no longer valid in the 2.0.x line.
The Pipeline Multibranch plugin made a few minor invalid assumptions about how to implement a Multibranch project type. For example, if you do not upgrade the Pipeline Multibranch plugin in tandem then you will be unable to manually delete an orphaned item before the orphaned item retention strategy runs, which should be significantly less frequently with the new event support.
If you upgrade to Branch API 2.0.x and you have either the GitHub Branch Source or the BitBucket Branch Source plugins and you do not upgrade those instances to the 2.0.x line then your Jenkins instance will fail to start-up correctly.
The solution is just to upgrade the GitHub Branch Source or the BitBucket Branch Source plugin (as appropriate) to the 2.0.x line.
- Git Plugin
The new releases of this plugin are both API and on-disk compatible with plugins compiled against the previous releases.
The 2.0.x lines of both the GitHub Branch Source and BitBucket Branch Source plugins require that you upgrade your Git Plugin to one of the versions that supports SCM API 2.0.x. In general, the required upgrade will be performed automatically when you upgrade your GitHub Branch Source and BitBucket Branch Source plugins.
- Mercurial Plugin
The new release of this plugin is both API and on-disk compatible with plugins compiled against the previous releases.
The 2.0.x line of the BitBucket Branch Source plugins require that you upgrade your Mercurial Plugin to the 2.0.x line. In general, the required upgrade will be performed automatically when you upgrade your BitBucket Branch Source plugins.
- BitBucket Branch Source Plugin
The 2.0.x line of this plugin makes on-disk file format changes that mean you will be unable to roll back to the 1.x line after an upgrade without restoring the old data files from a back-up.
- GitHub Branch Source Plugin
The 2.0.x line of this plugin makes on-disk file format changes that mean you will be unable to roll back to the 1.x line after an upgrade without restoring the old data files from a back-up.
If you upgrade to GitHub Branch Source 2.0.x and you have the GitHub Organization Folders plugin installed, you must upgrade that plugin to the tombstone release.
- GitHub Organization Folders Plugin
The functionality of this plugin has been migrated to the GitHub Branch Source plugin. You will need to upgrade to the tombstone release in order to ensure all the data has been migrated to the classes in the GitHub Branch Source plugin.
Once you have upgraded to the tombstone version and all GitHub Organization Folders have had a full scan completed successfully, you can disable and uninstall the GitHub Organization Folders plugin. There will be no more releases of this plugin after the tombstone. The tombstone is only required for data migration.
Summary for busy Jenkins Administrators
Upgrading should make multi-branch projects much better. When you are ready to upgrade you must ensure that you upgrade all the required plugins. If you miss some, just upgrade them and restart to fix the issue.
- SCM API Plugin
2.0.1 or newer
- Branch API Plugin
2.0.0 or newer
- Git Plugin
Either 2.6.2 or newer in the 2.6.x line or 3.0.2 or newer
- Mercurial Plugin
2.0.0 or newer
- GitHub Branch Source Plugin
2.0.0 or newer
- BitBucket Branch Source Plugin
2.0.0 or newer
- GitHub Organization Folders Plugin
1.6
- Pipeline Multibranch Plugin
2.10 or newer
Other plugins that may require updating:
- GitHub API Plugin
1.84 or newer
- GitHub Plugin
1.25.0 or newer
- Folders Plugin
5.16 or newer
After an upgrade you will see the data migration warning
Summary for busy Jenkins users
SCM API 2.0.x adds fine-grained event support. This should significantly improve the responsiveness of multi-branch projects. This should significantly reduce your GitHub API rate limit usage.
If you are using theGitHub Branch Source orGitHub Organization Folders plugins then upgrading will significantly reduce the API calls made by Jenkins to GitHub.
If you are using any of the upgraded SCM plugins (e.g. Git, Mercurial, GitHub Branch Source, BitBucket Branch Source) then upgrading will significantly improve the responsiveness to push event notifications.
Summary for busy SCM plugin developers
You should read the newdocumentation on how plugin developers are supposed to implement the SCM API
Where to now dear Literate Plugin
The persistent reader may be wondering what happens now to the Literate plugin.
For me, the logical heir of the Literate Plugin is thePipeline Model Definition plugin. This new plugin has the advantage of an easy to read pipeline syntax with the extra functionality that I suspect was preventing people from adopting Literate.
The good news is that the Pipeline Model Definition already has 5000 installations as of December 2016 and I expect up-take to keep on growing.