1 2 Previous Next 16 Replies Latest reply on Jul 29, 2010 1:25 AM by alexsmirnov

    Lessons Learned From M1 Build And How to Improve Build Structure.

    alexsmirnov

      M1 build structure has some pitfalls ( and advantages as well ), so I wish to bring all my things together there. Until 4.0.Final 9 or even up to some 4.0.x release ) the main goal for build structure should be maximum development speed, so most om my suggestions targeted for that aim.

      Good things:

      • stable project root that is not changed often ( hope, we would stop on v 8 for a while ).
      • Bill Of Materials projects.
      • release script.

      Pitfalls:

      • internal branch/tag/merge structures. Unless you are working with a single project ( that not is an option until RF framework API will be stabilized ) it's more convenient to check out whole root project and work with it as a working copy. Internal tags/brunches multiplies size of the working copy ( 3-4 times after M1 tag ), increases network traffic and slows down svn operations. Also, it's hardly possible to use these structures for development branches or backport fixes to the tags if it affects more then one module. Many tools ( svn clients, IDEs, Maven, and tools like svnmerge/hgsvn/git-svn ) cannot be used directly because they does nut expect these structures.
      • Exposed number of Maven projects. Some components have up to 7 projects ( bom,parent,root,api,impl,ui,dist) for just 3 java classes. That slows down builds ( I have about 10 minutes to build whole project, and it's going to be worse as more components included in UI library ), that wastes developer time for error fix iterations. As a result, it's more likely to commit broken code into repository. Also, the number of required projects in IDE increased dramatically, that makes hard and slow any refactoring operations that affect couple of modules.
      • Complicated build configurations. I got 70 ( sic! ) parameters in effective pom for framework implementations - most of them just describe paths to the current in the trunk/tags/branches folders.
      • Tricky release and deployment process.

      Suggestions to improve:

      • Move core project code into the 'trunk' folder, that contains only the files included into release source distribution, with the same layout. It should be available to build whole project by the single command. No internal t/b/t structures there. In the future, we can extract some projects into the modules with separate tags and brunches and make links in repository to preserve trunk structure, similar to MyFaces 'current'.
      • Change 'trunk' code requirements. It should always contain stable code only, pass build and all tests on the Hudson server. Therefore, developers that are working with a single module can relay on nightly builds, and these builds can be used by QA to start release tests early. Any development that is going for the time more then a few hours and could affect many modules should be done as development branch and merged into the trunk after local tests only. Any errors on continuous integration server should be fixed immediately.
      • put projects that is not intended to be included into distribution ( sandbox, internal examples, and QA projects ) in the 'modules' folder. It's questionable there to create tags and branches for these modules - using t/b/t folders in the module or create them in the global tags/branches folders.
      • Reduce the number of maven project as possible. Use the single framework (api/impl) project only, single 'docs','examples', and 'archetypes' folders. To distinguish classes related for different components, put them into different packages ( I'll add necessary changes to CDK ).
      • do not use build parameters to tune the current versions of modules but separate pom.xml in tags/branches. For the linked modules, version to include into build can be changed by svn link target.
      • Create 'development' build profile that skips goals that not required for build iterations ( enforcer checks, documentation/javadoc builds )

      Estimated build structure:

      /tags

          /archive - for pre 3.2.x releases ?

      /branches

         - everything goes to brunch there

         /development - temporary development branches.

      /trunk

          archetypes

          cdk ( lin to the CDK module in the future.

             maven-plugin

             generator

             .....

          bom - only versions of richfaces artifacts and its runtime dependencies should be defined here.

          docs

             - all docs goes here.

          examples

          parent - build configuration and non-runtime dependencies.

          core

             api

             impl

          ui

             ui-foo

             ui-external -> link to module.

             ........

             ui-zzz

          dist - library assembled here.

      /modules

           richfaces-parent

           sandbox

           QA-projects

           zzz-module

           [cdk]

        • 1. Re: Lessons Learned From M1 Build And How to Improve Build Structure.
          jbalunas
          For some perspective & options I've added some items below....
          Current change planned:
          • Collapse /ui/<component-type>/<t/b/t> down to just a single /ui/<t/b/t> module.   This means that all components will share a single module.
          Other options:
          • [Alex] Collapse everything down to a single <t/b/t>
            • cdk, core, commons, examples, ui, everything
          • Collapse down all or some of the following:
            • Examples ( instead of per example )
            • Archetypes ( instead of per archetype )
            • Common & core into one
              • Issue is that then CDK will rely on core
            • Core & UI  into one
            • Everything but :
              • Parent
              • build/resources
              • *-sandbox
              • tests
                  
          Requirements to keep in mind:
          • We must be able to exclude components from a build.
            • This means api, ui, impl, etc...
            • There will be some components we don't want to include
          • CDK
            • should have its own distribution
            • should not bleed dependencies to rest of project
          • Point releases of parts of project without releasing whole project
            • Not sure how that would work under combined t/b/t with release plugin & tags
            • Enforce propert separation of dependencies and cross dependencies
              • i.e. one side effect of multiple t/b/t is that you must not have circular dependencies

           

          Alex - I'll review the rest of your suggestion asap.

          • 2. Re: Lessons Learned From M1 Build And How to Improve Build Structure.
            alexsmirnov

            I suggest to move tags/branches out of the project structure and using svn links instead. It lets to keep working code small, eliminates requirements for wired parameters in the build, and also allows to extract module from any level, not only by the point created as t/b/t folders.

            • 3. Re: Lessons Learned From M1 Build And How to Improve Build Structure.
              alexsmirnov
              Collapse /ui/<component-type>/<t/b/t> down to just a single /ui/<t/b/t> module.   This means that all components will share a single module.

              I caught the better idea from some other projects, using svn links instead of folder structures. Advantages: - smaller size of svn checkout/working copy. - we can no more argue about the t/b/t level but create it at ANY level at any time. - eliminates necessity of build parameters. Also, I suggest to use 'trunk' folder for the current code, because it's svn convention and expected place for the head code.

               

              Issue is that then CDK will rely on core

              Atmost fixed. CDK got option for the renderer utils class name, and even has #built-it value to generate utility methods directly ( that would be convinient for developers who want to use CDK to create couple of components and does not intend to use richfaces library).

               

              Point releases of parts of project without releasing whole project

              I'm not sure that it's even possible with the current structure at all, because parts depends form the parent projects ( bom, parent, build projects like checkstyle ). Anyway, to release the single component, we have to create separate tag that contains proper configuration for that project. So, the release process seems for me like that: - copy project(s) intended to release into the tag. Using 'svn copy' command lets us to merge changes back and forth in the future. - create 'stub' root pom with only necessary dependencies. - deploy release. That does not require internal tags/branches at all.

              • 4. Re: Lessons Learned From M1 Build And How to Improve Build Structure.
                nbelaevski

                I like the idea of svn:externals usage - it allows to work with almost any structure of t/b/t easily, however IMO it makes sense to minimize number of t/b/t entries to decrease release process efforts to these top-level modules each containing t/b/t:

                 

                • cdk
                • core
                • ui
                • dist/build/archetypes
                • examples
                • docs
                • ui-sandbox
                • examples-sandbox
                • tests

                 

                Collapsing everything to a single t/b/t can save us some time, but then we may need to restructure build once again just near 4.0.0 release point.

                • 5. Re: Lessons Learned From M1 Build And How to Improve Build Structure.
                  jbalunas

                  Alexander Smirnov wrote:

                   

                  • stable project root that is not changed often ( hope, we would stop on v 8 for a while ).

                  One thing we need to update ( and there is already a jira for it ) is to remove jsf versions from it, but agree the richfaces-parent has worked well.

                  • internal branch/tag/merge structures. Unless you are working with a single project ( that not is an option until RF framework API will be stabilized ) it's more convenient to check out whole root project and work with it as a working copy. Internal tags/brunches multiplies size of the working copy ( 3-4 times after M1 tag ), increases network traffic and slows down svn operations. Also, it's hardly possible to use these structures for development branches or backport fixes to the tags if it affects more then one module. Many tools ( svn clients, IDEs, Maven, and tools like svnmerge/hgsvn/git-svn ) cannot be used directly because they does nut expect these structures.

                  After M1 release, and re-spin I agree we have too many internal t/b/t.  However checking out the entire /root is not the intended way to work with these modules, but I do agree it is easier than creating a script to checkout only what you need.  We can use the svn:externals as you and Nick describe to make checking out a working copy easier.

                  • Exposed number of Maven projects. Some components have up to 7 projects ( bom,parent,root,api,impl,ui,dist) for just 3 java classes. That slows down builds ( I have about 10 minutes to build whole project, and it's going to be worse as more components included in UI library ), that wastes developer time for error fix iterations. As a result, it's more likely to commit broken code into repository. Also, the number of required projects in IDE increased dramatically, that makes hard and slow any refactoring operations that affect couple of modules.

                  Same issue as with t/b/t - it is good practice to have the parent/bom/etc… structure ( even you agreed with that ).  However, as I said above - I agree there is too many t/b/t atm.

                  • Complicated build configurations. I got 70 ( sic! ) parameters in effective pom for framework implementations - most of them just describe paths to the current in the trunk/tags/branches folders.

                  All of these items above boil down to the same core issue - # of modules. But I'm not sure about the 70 parameters in effective pom.  Are you sure these are not version info inherited from the jboss-parent?  What command are you using to see these?

                  • Tricky release and deployment process.

                  After the M1 release/respin, I agree there is improvements to be made here.

                  • Move core project code into the 'trunk' folder, that contains only the files included into release source distribution, with the same layout. It should be available to build whole project by the single command. No internal t/b/t structures there. In the future, we can extract some projects into the modules with separate tags and brunches and make links in repository to preserve trunk structure, similar to MyFaces 'current'.

                  This seems like the extreme the other way.  From too many modules --> to none ( really ).  If we go with something in the middle, like what Nick and I were talking about and using svn:external to tie the fewer modules together it would be good.

                  • Change 'trunk' code requirements. It should always contain stable code only, pass build and all tests on the Hudson server. Therefore, developers that are working with a single module can relay on nightly builds, and these builds can be used by QA to start release tests early. Any development that is going for the time more then a few hours and could affect many modules should be done as development branch and merged into the trunk after local tests only. Any errors on continuous integration server should be fixed immediately.

                  Could not agree more with this.  Everyone should start using development branches.  I think it would be good for you to write up a little RF specific wiki page on how developers should use development branches.  I will tie that in with the development wiki pages that I will be working on.

                  • put projects that is not intended to be included into distribution ( sandbox, internal examples, and QA projects ) in the 'modules' folder. It's questionable there to create tags and branches for these modules - using t/b/t folders in the module or create them in the global tags/branches folders.

                  Not sure if I like the /modules name, but the idea is good.  We can certainly separate the distributed and non-distributed items.  As for tagging/branching - I think some of these might need their own t/b/t - like richfaces-parent.  I'll need to think on this more.

                  • Reduce the number of maven project as possible. Use the single framework (api/impl) project only, single 'docs','examples', and 'archetypes' folders. To distinguish classes related for different components, put them into different packages ( I'll add necessary changes to CDK ).
                  Agree as a good goal, but this can go too far as well.  When you refer to projects, do you mean sub-projects?  I think that docs, examples, and archetypes make sense as separate sub-projects, AND should have their packages named based on that as well.
                  • do not use build parameters to tune the current versions of modules but separate pom.xml in tags/branches. For the linked modules, version to include into build can be changed by svn link target.

                  Managing the svn link is not really trackable, so it would be hard to rebuild a release if needed ( would svn links be "tagged"?).  If we move to fewer t/b/t we will naturally move to fewer inter project version parameters.  ( Although I'm not sure where the 70 parameter number came from above ).

                  • Create 'development' build profile that skips goals that not required for build iterations ( enforcer checks, documentation/javadoc builds )

                  Agree, good idea, but full build required prior to commit from dev branch.

                  Estimated build structure:

                  /tags

                      /archive - for pre 3.2.x releases ?

                  /branches

                     - everything goes to brunch there

                     /development - temporary development branches.

                  /trunk

                      archetypes

                      cdk ( lin to the CDK module in the future.

                         maven-plugin

                         generator

                         .....

                      bom - only versions of richfaces artifacts and its runtime dependencies should be defined here.

                      docs

                         - all docs goes here.

                      examples

                      parent - build configuration and non-runtime dependencies.

                      core

                         api

                         impl

                      ui

                         ui-foo

                         ui-external -> link to module.

                         ........

                         ui-zzz

                      dist - library assembled here.

                  /modules

                       richfaces-parent

                       sandbox

                       QA-projects

                       zzz-module

                       [cdk]

                  I'm not sure the single t/b/t structure is the best, and as nick says may require additional refactoring prior to final release.

                  Not sure why there is both a "richfaces-parent" and a "parent" in your structure

                  • 6. Re: Lessons Learned From M1 Build And How to Improve Build Structure.
                    jbalunas

                    Alexander Smirnov wrote:

                     

                    Collapse /ui/<component-type>/<t/b/t> down to just a single /ui/<t/b/t> module.   This means that all components will share a single module.

                    I caught the better idea from some other projects, using svn links instead of folder structures. Advantages: - smaller size of svn checkout/working copy. - we can no more argue about the t/b/t level but create it at ANY level at any time. - eliminates necessity of build parameters. Also, I suggest to use 'trunk' folder for the current code, because it's svn convention and expected place for the head code

                    Whether we use links or not we are still talking about minimizing the number of t/b/t while maintaining the requirements from above.  I think that links may be useful, but also that we still have a need for some ( not as many ) but some t/b/t modules. 

                    Issue is that then CDK will rely on core

                    Atmost fixed. CDK got option for the renderer utils class name, and even has #built-it value to generate utility methods directly ( that would be convinient for developers who want to use CDK to create couple of components and does not intend to use richfaces library).

                    So you are not simply make a copy for CDK to maintain of the shared code correct?  I want to understand this better.  Can you think of any "shared" code that might be needing in the future?

                    Point releases of parts of project without releasing whole project

                    I'm not sure that it's even possible with the current structure at all, because parts depends form the parent projects ( bom, parent, build projects like checkstyle ). Anyway, to release the single component, we have to create separate tag that contains proper configuration for that project. So, the release process seems for me like that: - copy project(s) intended to release into the tag. Using 'svn copy' command lets us to merge changes back and forth in the future. - create 'stub' root pom with only necessary dependencies. - deploy release. That does not require internal tags/branches at all.

                    In the current structure it is certainly possible to have different modules at different point releases.  This is the strength of this approach. Your approach here actually seems more complex to me, but would also likely not be done often.

                    • 7. Re: Lessons Learned From M1 Build And How to Improve Build Structure.
                      jbalunas

                      I think an approach like what nick and I are suggesting might be a good middle ground, and limits the number of modules, while still allowing for many of the requirements from above.

                       

                       

                      ModuleM1 #M1 t/b/tProposed #Pro t/b/tNotes
                      /uin*3+n+14+m1n= # of component types, m= # of components.  Each component type had a bom, parent, aggr + its own source
                      /examplesn*3+nn+1n=# of examples.  Proposed is a single t/b/t containing sub-projects for all public examples.  Dev examples should be in /examples/dev, and sandbox examples should be in /sandbox-examples
                      /archetypesn*3+nn+1n=#of archetypes.  As with examples, except that CDK specific component archetypes should be in /cdk

                       

                      The other remain as they are, single ( non-growing ) t/b/t for core, cdk, parent, bom, etc...

                       

                      One question I have is structure of *-sandbox: leave as is?  no t/b/t? single t/b/t?  Also do we move all sandbox ( cdk, example, ui ) into a /sandbox directory?

                      • 8. Re: Lessons Learned From M1 Build And How to Improve Build Structure.
                        jbalunas

                        The changes above will give us a stable base, that should not need to be maintained every release.  The build scripts, and other things can be stabilized, and left alone.

                         

                        This does not satisfy all of Alex's points, and there would still be some trickiness with refactoring across projects, but it should be minimized.  Although using svn:external to create our own custom "/current" may make that easier for IDE's to handle it as a single module.  We'll need to look into that.

                         

                        The /current will also make it easy for people to just start working with the latest source.  If this information can also be tagged, and/or manipulated for development branches it would even make that work easier.

                         

                        There is still the question of tools like svnmerge.  Alex they can not handle this structure - correct?  If it is done through a svn:external links - will that help ( I doubt it )?  Either way this approach minimizes it.

                        • 9. Re: Lessons Learned From M1 Build And How to Improve Build Structure.
                          alexsmirnov

                          I suggest to use 'trunk' folder instead of the 'current', because it's svn naming convention.

                          he /current will also make it easy for people to just start working with the latest source.  If this information can also be tagged, and/or manipulated for development branches it would even make that work easier.

                          I already created the tag for M1 release that brings all modules together in the single tag. svn:externals also under version control, so they could be tagged.

                          I don't see the necessety to have tags/branches for modules, because they could be tagged in any place, including root tags/branches folder.

                          • 10. Re: Lessons Learned From M1 Build And How to Improve Build Structure.
                            alexsmirnov
                            In the current structure it is certainly possible to have different modules at different point releases.  This is the strength of this approach. Your approach here actually seems more complex to me, but would also likely not be done often.

                            Because UI modules depend from api/impl/cdk, how to release component without havind release of related modules ?

                            That's unpredictable that modules should be released together - that's why separate release for module seems hardly possible.

                            • 11. Re: Lessons Learned From M1 Build And How to Improve Build Structure.
                              alexsmirnov

                              Yes, it looks better.

                              I only suggest to not create t/b/t folders in advance, but keep plain folders structure for distributed code in the trunk, except modules that already stable and does not require separate tags on release - richfaces-root, and checkstyle resources that would be linked to the trunk by svn:externals.

                              We can be wrong with predictions that modules will be released separately, so making walls in advance would give us troubles in the future.

                              We always can create a tag with only necessary modules and custom root/bom projects. In the future, then we will be sure about modules structure, we can put necessary parts into the root ( with optional tags/branches ) and make a link from trunk.

                              The structure seems like this:

                              trunk

                                 build/parent - link to the current stable root.

                                 build/resources - link to the latest released.

                                 docs

                                 bom

                                 cdk

                                 core

                                 ui

                                 examples

                              root

                                 build - that may contain t/b/t folders or just using tags in the global /tags folder.

                                 sandbox - does not require tags.

                                 qa-tests

                                 whatever else not included in release.

                              Also, we have to put old 3.x release tags into archive.

                              • 12. Re: Lessons Learned From M1 Build And How to Improve Build Structure.
                                nbelaevski

                                OK, let's stick to the simplest variant now and create additional t/b/t-s later.

                                 

                                Jay,

                                 

                                WDYT?

                                • 13. Re: Lessons Learned From M1 Build And How to Improve Build Structure.
                                  jbalunas
                                  Based on our team meeting today, and the topics here, I'm putting together a more complete structure.
                                  Here are some updated requirements:
                                  • Only dev-stable features and components may be in /trunk
                                    • Development branches are required for all large updates
                                      • <link to Alex's wiki page on development branch usage>
                                    • Only small & clearly understood updates may be made directly in /trunk
                                  • Only items to be tagged during release cycle should be in the /trunk
                                  • Investigation and implementation of dependency/circular references plugin
                                    • Should not have to wait for release to find the issues
                                    • This will also make it easier to split out t/b/t if needed
                                  • Everything under /trunk gets tagged with release
                                    • Not everything is in distribution, controlled by assembler
                                  • Future t/b/t should be possible when needed
                                    • Such as CDK when it stabilizes, or has different release plan
                                  • svn:externals
                                    • Should be used only if needed
                                  • CDK must have its own distribution
                                  • Point releases must be possible for sub-trees of the project.
                                    • not required until after initial final release
                                    • when this is required we will revisit t/b/t for specific areas.
                                    
                                  /trunk
                                    /archetypes
                                      /<archetype-1>
                                      /<archetype-2>
                                      /etc...
                                    /bom
                                    /build (?? link to /build items ??)
                                    /cdk
                                      /dist
                                      /etc...
                                    /core
                                      /api
                                      /impl
                                    /dist
                                    /docs <not inherit or effect rf-parent, or BOM>
                                    /examples
                                      /<example-1>
                                      /<example-2>
                                      /etc...
                                    /tests
                                    /ui
                                      /output
                                      /input
                                      /iteration
                                      /etc...
                                  /modules ( or /root, or ??? )
                                    /build
                                      /parent
                                         /<t/b/t>
                                      /resources
                                         /<t/b/t>
                                            /checkstyle
                                            /shade
                                            /scripts
                                      /sandbox ( do not need tags )
                                         /examples
                                         /ui
                                         /tests
                                     /<anything else not tagged with release>
                                  /tags
                                     /archive
                                        /<pre-3.3.X>
                                        /3.3.X
                                        /4.0.0-legacy ( ??name??  for pre-structure update tags )
                                     /<4.X tags>
                                  /branches
                                     /archive
                                        /<pre-3.3.X>
                                     /community (move to archive?)
                                        /3.3.X
                                     /development
                                        /<all development branches here>
                                     /enterprise
                                  
                                  Note:
                                  • /commons not needed - but CDK should not depend on anything in /trunk
                                    • If CDK ever does require other parts of project, we should create /commons again
                                  • Custom distributions for including/excluding components
                                    • can be done with pom profiles
                                    • and/or branches and tags for custom structure
                                  • TBD
                                  TODOs:
                                  • Put 3.X tags into /tags/archive
                                  • Discuss the need for links to /build in /trunk
                                  • TBD

                                   

                                  If I missed something, or you have corrections, ideas please comment.  I'll break down some more tasks, etc.... from this.  Anyone else that would like to comment, or edit tasks/requirements please do.  Specifically Alex K, Alex S, Nick, Prabhat, Ilya, etc...

                                  • 14. Re: Lessons Learned From M1 Build And How to Improve Build Structure.
                                    alexsmirnov

                                    This is ok. The only one thing is 'tests' folder. For my opinion, they should be in 'modules' but tagged with release ( tags/4.0.x and tags/tests-4.0.x ). That allows to have identical sources in the distribution and SVN repository, so everyone could make a patch to fix particular issue without checking out QA tests, that would be pretty big. Patches for bugs is most often kind of community contributions, so we have to make it simple.

                                    Another concern is branches/development folder. Many tools ( including Maven, Eclipse SVN plugins, SVN viewers ) relies on the default trunk/tags/branches structure,  so it would be more convenient to have development branches in the standard place. It should not overload repository because most of them should be deleted after merging.

                                    The steps to move build structure:

                                    - make archive folder and move outdated tags and branches there.

                                    - make tags for Alpha/M1 releases and copy there all tags from the current structure, similar to the http://anonsvn.jboss.org/repos/richfaces/tags/4.0.0.20100715-M1/ but with real copies instead of svn:externals links. I can make a shell script and perform that task.

                                    - move trunk modules from the 'root' into /root folder. Once agin, the /trunk already contains desired structure created by svn:externals, so that task should only replace links by the real content.

                                    Important: only svn move commands on the http://anonsvn.jboss.org/repos/richfaces/ repository should be used, to keep history and space ( svn does not really move files on the server, but only modifies links for them ).

                                    Link to the 'build' folder would make more easily to build project from svn. If these projects aren't included into the source,  anyone who wants to build richfaces locally has to modify Maven settings to enable Jboss repositories.

                                    1 2 Previous Next