Splitting up the Wildfly code base

Version 3

    Introduction

     

    This document is a work in progress!

     

    The Wildfly code base is currently over 1 million lines and takes around 2 hours to do a complete build. It was decided at the EAP team meeting in Brno that it is time to split the code base into smaller modules, which should make the code base much easier to manage, which in turn makes current developers more productive and makes it easier for people to contribute. This will hopefully also make the Wildfly runtime a much more attractive proposition for 3rd party developers, as they will be able to simply take the pieces they want.

     

    This document describes the proposed changes, and outlines the time frame and methodology that will be used to perform the split.

     

    General Overview


    The code base will be split into 7 modules:

     

    Core

    The Wildfly core distribution, basically the minimal

     

    Console

    The management console, as this is quite large it will live in a separate location to the core distribution (or maybe have the core repo produce two artefacts, one with and one without?)

     

    Arquillian

    All Arquillian code will be split into a separate module, which will solve a long running problem of the Wildfly and Arquillian adaptor code being intertwined. This may eventually be moved to live under the Arquillian organisation in github.

     

    Web

    Undertow subsystem, and all related dependencies, including a small subset of EE and JNDI. This is basically just a Servlet container, and will provide a platform for people that want to create web based appliances or applications, and don't need all the additional functionality that Wildfly provides. We should end up with something as lightweight as Tomcat or Jetty, but with all our advanced management functionality.

     

    EE

    The EE module will contain all other EE related functionality, such as EJB, JCA, Messaging etc. This module will likely depend on Web.

     

    Clustering

    Clustering contains Infinispan, and all functionality required for Web and EJB based clustering.

     

    Traditional Certified Distribution

    A module that brings together all of the above components and builds a complete traditional Wildfly server. Other distributions may be added in the future.

     

    Build Process

     

    In general the individual modules will have a similar file system layout to the existing Wildfly repository, to make back porting patches to EAP easier. At the moment our build process to build the server is basically comprised of a bunch of ant scripts, however this will probably have to change.

     

    One idea would be to create a Maven plugin, that knows how to build a server from a combination of existing server distributions, and some local module definitions and xml files. This would do a similar job to what we already do with our ant scripts, however would be easier to keep in sync across projects, as maintaining multiple copies of the ant scripts. This would also make it easier for 3rd party distributions to assemble a server, without copying/pasting our assembly code. As part of this build process the XML templates that are used to build the config should be preserved, so that other distribution builds can make use of them.

     

    Testing

     

    As part of the split the existing test suite will be split into multiple modules that correspond to the functionality of a given distribution, e.g. web related tests will be part of the web module, EE related tests will be part of EE and so on.

    This will require a significant refactor of the test suite. We will no longer use a multi execution per module strategy, as this has caused nothing but problems from the beginning. Instead we will have a single execution per module, and will use the failsafe plugin (rather than surefire) to execute the tests in the integration-test phase.

     

    The resulting test modules must also be usable against the full server module, so tests that are run against the web component individually must also be run as part of the full dist build. This will likely be accomplished by packaging up the test classes into test-jar artefacts, and should not pose a significant challenge.

     

    This test suite refactor should be performed before the distributions are split out, as EAP will likely want to adopt the new test suite structure at some point in the future to make back porting easier.

     

    Workflow

     

    After this change the individual components will have to be versioned separately, and treated the same as any other component that forms part of Wildfly. This means that we must follow a 'release early release often' mantra for all component types, basically performing a release will have to be as simple as updating a version number and uploading to Nexus. If components are not released often it will make it very difficult to make changes that affect multiple areas of the server, and will greatly slow the development process.

     

    Pull request processing will follow a similar pattern to today, however it should be greatly streamlined as mergers will no longer step on each others toes as they can work in different repositories simultaneously. Also for most repositories merging runs should be much faster, as the build will no longer take 2 hours.

     

    Having a runtime that is released often will also increase the attractiveness of the runtime to 3rd parties who might wish to build on it, as they will know that bug fixes will be released promptly.

     

    Timing

     

    This split will need to take place as soon as the Wildfly 9.x branch opens. We do not want to be making these changes late in the release cycle, so they should be done as soon as possible to allow any potential problems to be ironed out well before the release.

     

    Challenges

     

    There are a number of challenges that will need to be overcome in order to make this happen.

     

    Inter module dependencies

    There is a good chance that with the current module structure it will not be possible to create a nice clean split as defined above. Modules may have service and class loading dependencies on each other that will need to be untangled in order to make this work. This work should be fairly straightforward, however depending on just how many issues we find it may take a significant amount of time.

     

    Testing the core

    Because Arquillian will be split out the core will no longer be able to use Arquillian to run tests in order to avoid circular dependencies. In practice this should not be a major problem, as most of the core tests do not rely on Arquillian, and simply require the ability to start and stop the server, and execute management operation against it.