Versions for large multi-module osgi projects

Recently I went through JBoss Tools main source code base and fixed a few long standing issues that had made it cumbersome to manage versions over the years. This blog tries to explain what we changed, why and how Tycho helped.

Tycho Version Plugin

Maven has it’s own versions plugin but it only takes care of version references in pom.xml, it does not consider osgi/p2 related references. Tycho therefore has it’s own versions plugin.

Unfortunately there aren’t alot of documentation for this plugin, but by poking to Sonatype, follow the tycho mailing list and looking at a few hints out there I managed to get the it working. Here is my attempt to make the world more aware of the Tycho Version plugin.

Usage

The core command for Tycho Versions plugin are executed in your Maven module that represents your plugin/features. The Tycho Version plugin command looks like this:

mvn org.eclipse.tycho:tycho-versions-plugin:set-version -DnewVersion=<version>

Here <version> is the version number you want for your plugins, i.e. 1.2.3.GA. If you use 1.2.3-SNAPSHOT the plugin will use 1.2.3.qualifier in the places necessary (i.e. in manifest.mf and feature.xml).

The beauty of this plugin is that with a single command you can update your plugin, features, product and updatesite references.

No manual tweaking required, except that to use it effectively your pom.xml’s should follow a few maven conventions.

That last part is what kept us in JBoss Tools to use this command until recently.

Why you ask ? Well, read on…

Maven Parent/Child relationships

The reason we couldn’t use the tycho version plugin previously was that we had the following layout for each of our modules: 

Each top level module has a set of submodules: plugins, tests, features and site (there are also a doc module but that is for another blog).

Each of these submodules can have children, for plugins that are all the individual plugins the main feature will include, tests are the plugin tests and so forth.

That parent represents the parent pom where the shared build configuration for JBoss Tools is stored - we use that to avoid repeating the build instructions in all 350+ buildable units.

But there is a big problem with this structure.

First of each artifact have a manual maintained version section, but do you notice all those thick red outlined lines going to the Parent from each leaf plugin, feature, etc. ?

Those red lines illustrates that each child leaf point directly to the main parent pom which makes it possible to have the build configuration in one central place but it also means that everytime we update the parent pom version we have to change it in each and every node (for us that is 605 locations).

It also have the consequence that even though we have this nice modular division of each module into tests and plugins there is not a place to store module or submodule specific settings for our builds - that all have to take place in each leaf plugin instead of being just stated in one place.

Thus to make this sharing easier and to make it more explicit what structure is actually being versioned with the Tycho module Denis Golovin and I did a few pom refactorings.

Notice the difference ?

Each “sub-node” now uses their actual structural relative parent as the pom parent (red lines) and only the top level module actually points back to the main JBoss Tools parent module (blue line).

This has the following positive features:

• you can stand in the root of a module and run mvn clean install and it will build the whole module
• you can override/customize a build locally for a module or submodule
• maven tycho version plugin will be able to easily version this module on its own

The last optimization I did lately was that now that we have this natural structure between parent and child setup we can remove the child modules redundant version info from the pom.xml’s. We unfortunately still have to list the full version info for the parent but with the Tycho versions plugin this becomes trivial to maintain.

The Dirty Details

The essentials parts of the structure is as follows (using hibernatetools as reference):

Module pom:

<project ...>
    ...
    <parent>
            <!-- the only reference to root build parent from a module -->
            <groupId>org.jboss.tools</groupId>
            <artifactId>org.jboss.tools.parent.pom</artifactId>
            <version>0.0.2-SNAPSHOT</version>
            <relativePath>../build/parent/pom.xml</relativePath>
    </parent>
    <groupId>org.jboss.tools</groupId>
    <artifactId>hibernatetools</artifactId>
    <name>hibernatetools.all</name>
    <!-- the only place the module version number needs to be in a pom.xml for a module -->
    <version>3.4.0-SNAPSHOT</version>
    <packaging>pom</packaging>
    <modules>
            <module>features</module>
            <module>plugins</module>
            <module>tests</module>
            <module>site</module>
   </modules>
</project>

Sub-Module pom:

<project ...>
    ...
    <parent>
            <!-- this section is maintained by the versions plugin --!>
            <groupId>org.jboss.tools</groupId>
            <artifactId>hibernatetools</artifactId>
            <version>3.4.0-SNAPSHOT</version>
    </parent>
    <groupId>org.jboss.tools.hibernatetools</groupId>
    <artifactId>plugins</artifactId>
    <name>hibernatetools.plugins</name>
    <packaging>pom</packaging>
    <modules>
            <module>org.hibernate.eclipse</module>
            <module>org.hibernate.eclipse.console</module>
            ...
</modules>

…and finally the actual plugin/feature pom’s:

<project ... >
<modelVersion>4.0.0</modelVersion> 
<parent>
    <groupId>org.jboss.tools.hibernatetools</groupId>
    <artifactId>plugins</artifactId>
    <version>3.4.0-SNAPSHOT</version>
</parent>
<groupId>org.jboss.tools.hibernatetools.plugins</groupId>
<artifactId>org.hibernate.eclipse</artifactId> 
<packaging>eclipse-plugin</packaging>

Notice how this setup makes the configuration at the plugin level very concise (or at least as concise as Maven allows them to be). Only if your plugin has custom build logic do they need to be different.

Conclusion

With this setup we can now easily customize our builds at a module level and easily bump our versions number across all the osgi/p2 artifacts within the module.

Meaning if I use the Tycho versions plugin to bump a module to 3.5.0, by default everything in that module gets bumped to 3.5.0, no matter if they have changed or not.

Purists of osgi might claim that is a bad practice and you should only bump the versions of the plugins that actually received changes and in principle I agree with them, but…

…JBoss Tools consists of about 202 plugins and 100 features each with their own feature.xml, MANIFEST.MF and when we included Tycho or rather Maven in the mix they also each have a pom.xml.

Add in the various updatesites, and other artifacts and just in the main source code tree of JBoss Tools it all adds up to us having at the current time of writing 628 versionable artifacts.

That is alot of versions to keep track of and ensure is uptodate.

Until now we have been maintaining the versions of these artifacts manually by hand and just as a precaution we (as many others in osgi/p2 land) do our builds with a .qualifier as the last part of the version number. This ensure the version number is always higher than the previous build. If we don’t, then p2 won’t install the newest build.

This actually means that if you don’t realize it then you can easily stay on version 1.0.0 for your plugins forever and ever - no users will complain since they can always get the the newest updated version.

The problem of being blind to versions of course show up when you start having multiple branches and even more so when you try to build up API plugins and want to ensure you are using the right compatible version - and here relying on the “random” qualifier string becomes unmanagable.

With a handful of plugins it’s “easy”, but with 628 artifacts it can become rather chaotic.

Thus I’ll argue its better to do it on the module level which reduces the versionable artifacts to 42 in our case - a much more managable number and much simpler to handle and understand for both developer and user IMO.

Notice, that if you are in a situation where you need to manually version specific plugins then you simply set the version explicitly for those plugins/features and Tycho versions plugin will not change them if they are not the same as the module pom.

That is another thing I like about this setup

- it allows you to do the most manageble thing easily, and if you need the full control you can do that for just the parts you wishes it for.

What do you think ?

Have you solved this problem differently/better/worse ?