1 2 Previous Next 24 Replies Latest reply on Jan 28, 2005 6:22 PM by ryan.campbell

    New Build in JBoss-Head

      Continuing this thread...
      http://www.jboss.org/index.html?module=bb&op=viewtopic&t=57963

      I have committed the new jbossbuild project
      cvs ... co jbossbuild
      I think we should create a separate JIRA project for this where we can
      add tasks that need completing.

      I have also made a start at adding it to jboss-head. So far there is:
      1) A new top level project "jbossas" which serves as a replacement for build
      2) common has the alternate jbossbuild.xml
      3) tools/lib/jbossbuild.jar which is the current output of the jbossbuild project above
      4) tools/etc/jbossbuild/tasks.xml the common definitions

        • 1. Re: New Build in JBoss-Head

          Here is some of what is new from the previous discussion:

          1) I added an explicit

          <generate/>
          tag. This works around a problem
          with ant typedefs where there is no callback to tell you have finished parsing
          that xml element.
          This enabled me to remove of the complications with target generation where it had
          to be done lazily. There is almost certainly some code still present that is legacy
          from the previous way of doing it.

          2) There is now an "ant show" which also takes an optional property
          to see specific targets:
          [ejort@htimes2 common]$ ant -f jbossbuild.xml show -Dshow=build.namespace.jar
          Buildfile: jbossbuild.xml
          
          show:
          <!-- Build for the artifact namespace.jar -->
          <target name="build.namespace.jar">
          <mkdir dir="/home/ejort/newjboss/jboss-head/common/output/lib"/>
          <jar manifest="/home/ejort/newjboss/jboss-head/common/output/etc/default.mf" destfile="/home/ejort/newjboss/jboss-head/common/output/lib/namespace.jar">
           <fileset dir="/home/ejort/newjboss/jboss-head/common/output/classes/main">
           <include name="javax/xml/namespace/*"/>
           </fileset>
          </jar>
          </target>
          


          3) You can now specify any attribute on a definition which negates the need
          to explicitly add code for the use case where you want to pass a parameter from the
          definition to the targetdef.

          This says run junit against all source definitions that have a test attribute
          build.xml
           <source id="test" test="**/*TestCase.java">
           <include input="classes"/>
           <include input="test.path"/>
           </source>
          

          tasks.xml
           <source when="@{test}">
           <mkdir dir="@{testDir}"/>
           <junit fork="true"
           printSummary="true">
           <formatter type="plain"/>
           <classpath>
           <pathElements/>
           </classpath>
           <batchtest todir="@{testDir}">
           <fileset dir="@{sourceDir}" includes="@{test}"/>
           </batchtest>
           </junit>
           </source>
          


          4) Allow an output path to be directly overriden:
          This was done because slide is in apache-slide/client rather than lib

          5) Allowed includes/excludes when using filesets. This needs to be generalized
          to other groupings.

          6) Added componentmain as a target definition. This allows the top level build
          to run tasks against a component rather than running a target inside the component
          build file.
          e.g. in the synchronize task we want to do either a cvs update or co depending
          upon whether the component directory actually exists:

           <targetdef target="synchronize" description="Synchronize">
          
           <!-- Update the main build folder and tools from cvs
           then do the same for the components before running
           the after synchronization processing
           NOTE: Does not automatically invoke component builds
           as the list of components maybe out-of-date at this point
           and we need to conditionally do cvs co/update
           -->
           <main components="none">
           <cvs command="update -dP" failonerror="true"/>
           <invoke target="synchronize" dir="../tools"/>
           <execant target="synchronize.components"/>
           <execant target="synchronize.after.main"/>
           </main>
          
           <!-- If the component exists we just do a cvs update -->
           <componentmain if="@{exists}">
           <invoke target="synchronize" dir="@{dir}"/>
           <execant target="synchronize.after" dir="@{dir}"/>
           </componentmain>
          
           <!-- If the component doesn't exist and we want to
           get the source build check it out from cvs
           -->
           <componentmain unless="@{exists}" if="@{local}">
           <cvs dest="@{dir.parent}">
           <commandline>
           <argument value="-d"/>
           <argument value="@{build.cvsroot}"/>
           <argument value="co"/>
           <argument value="@{module}"/>
           </commandline>
           </cvs>
           <execant target="synchronize.after" dir="@{dir}"/>
           </componentmain>
          


          7) Some other minor changes I can't remember at the moment?

          • 2. Re: New Build in JBoss-Head

            TODO: This should be replaced with tasks in JIRA

            1) Do the other projects within jboss-head - this is an alternate build for now,
            i.e. you have to do ant -f jbossbuild.xml
            This basically involves copying common/jbossbuild.xml to the other projects
            and updating it.
            It will also involve specifiy those components in jbossas/jbossbuild.xml
            along with adding references to thirdparty jars

            2) Some things are missing from the targetdefs at time of writing:
            build.bin - processing for src/bin
            build.resources - processing for src/resources
            rmic, javacc, aopc compilation
            sar/war/ejb target builds
            docbook build (needs including in the overall doc target)
            I also haven't tested the notion of a file being an artifact, e.g. jbossjca-service.xml

            3) Need to externalize the filesets for normal ant targets. The pseudo

            <filesets/>
            is not available there.
            e.g. make the following
             <source id="main">
             <include input="commons"/>
             <include input="concurrent"/>
             <include input="dom4j"/>
             <include input="jaxme"/>
             <include input="log4j"/>
             <include input="regexp"/>
             <include input="slide"/>
             <include input="wutka"/>
             <include input="xerces"/>
             </source>
            

            generate a main.filesets, main.sourcepaths, etc., that can be referenced in the normal
            fashion

            4) Need to build the binary versioned repository (currently it is just using thirdparty as
            checked from cvs). This will enable developers to only checkout and build those components.

            5) Need to complete the cvs checkout processing so it takes into account
            user ids in the path and passwords for those without sshd.
            Also need to allow developers to specify what source they want to work on
            and want can just be binary download from last nights build.
            This will probably be best implemented using a local.properties file in jbossas

            6) Beyond (5) we then need to do the work for developing standalone
            projects alongside each other.
            e.g. 1) Allow the cache project to override the binary downloaded into thirdparty
            with their local source project
            e.g. 2) Allow the Seams/Portal projects to include a binary build of JBossAS which
            they can then deploy to rather than it being a multistage checkout/build process

            • 3. Re: New Build in JBoss-Head

              Ryan will be taking over the responsibility for this work,
              with lots of input from me.

              I imagine there are lots of other requirements and there are many things that become
              possible with a more declaritive and consistent build.

              One of the key issues before this becomes the real build will be to document
              how jbossbuild works.

              The basic philosophy of jbossbuild is that you declare:
              Inputs (source and thirdparty jars)
              Outputs (artifacts)
              and the links between them.

              These are then processed through common targetdefs
              which can be read as (in the example below about junit tests)
              for each "source" with the attribute "test" do this...

              The parameterization comes from

              @{property}

              which reads properties from the javabean typedefs declared in the build.xml

              e.g.
              <source when="@{test}">
               <mkdir dir="@{testDir}"/>
               <junit fork="true"
               printSummary="true">
               <formatter type="plain"/>
               <classpath>
               <pathElements/>
               </classpath>
               <batchtest todir="@{testDir}">
               <fileset dir="@{sourceDir}" includes="@{test}"/>
               </batchtest>
               </junit>
              </source>
              


              reads as:
              for each "source" in the build.xml that has a "test" attribute
              make the directory returned by SourceDefinition.getTestDir()
              (which is output/test/@{id} by default
              and run junit over those sources in @{test} putting the results
              in the @{testDir}

              The pseudo
              <pathElements/>
              creates a classpath
              that includes both the classes from the source compilation and
              its dependent jars as defined by the source's includes.

              • 4. Re: New Build in JBoss-Head

                How do I add a pre-compilation task like rmic? I think I would want to define the input source which required processing:

                <componentdef ..>
                 <source id="rmic">
                 <include input="classes">
                 </source>
                </componentdef>
                

                And then in tasks.xml
                <targetdef target="rmic">
                 <component/>
                 <source when="@{rmic}">
                 <rmic base="@{classesPath}">
                 <classpath>
                 <pathElements/>
                 </classpath>
                 </rmic>
                 </source>
                </targetdef>
                

                I guess I would also need to add a depends attribute to the source element in the build targetdef?
                <targetdef target="build" description="Build">
                 :
                 <source depends="rmic">
                 <mkdir dir="@{output}/etc"/>
                 :
                


                • 5. Re: New Build in JBoss-Head

                  I would do it like this:

                  <source id="main"
                   rmic="org/jboss/invocation/jrmp/server/JRMPInvoker*"
                  >
                  ...
                  


                  // snipped targetdef as now
                   <targetdef target="build" description="Build">
                   <source>
                   <mkdir dir="@{output}"/>
                   <depend srcdir="@{sourcePath}"
                  ...
                   </depend>
                   <javac srcdir="@{sourcePath}"
                  ...
                   </javac>
                   </source>
                  // end snipped
                  // new rmic code
                   <source when="rmic">
                   <rmic ...>
                   <include name="@{rmic}"/>
                   </rmic>
                   </source>
                  


                  jbossbuild allows multiple "source" elements in the targetdef and will take
                  only those definitions that apply to the given source.
                  It will append all the relevent tasks together in the order they were specified.

                  • 6. Re: New Build in JBoss-Head

                    I have checked in j2ee/jbossbuild.xml. It produces all three artifacts of j2ee and integrates into the top level build. However, I have a couple of issues.

                    1. j2ee includes some classes from jmx. For now, I access using a relative path which is what the existing build path essentially does. Not sure if we want to formalize this into an artifact.

                    2. I had to explicitly reference the libs that I needed from jboss-common. When I tried to do:

                     <source id="main">
                     <include input="common"/>
                     <include input="servlet"/>
                     <include input="jaf"/>
                     </source>
                    

                    ... the path was resolved to common/output. So instead I had to include the jars explicitly.
                     <source id="main">
                     <include input="jboss-common.jar"/>
                     <include input="namespace.jar"/>
                     <include input="servlet"/>
                     <include input="jaf"/>
                     </source>
                    


                    I shouldn't have to reference the jars for a local component any differently than for a non-local one, no?

                    • 7. Re: New Build in JBoss-Head

                       

                      "ryan.campbell@jboss.com" wrote:
                      I have checked in j2ee/jbossbuild.xml. It produces all three artifacts of j2ee and integrates into the top level build. However, I have a couple of issues.

                      1. j2ee includes some classes from jmx. For now, I access using a relative path which is what the existing build path essentially does. Not sure if we want to formalize this into an artifact.


                      It is one of the issues on my list to sort out the javax.management classes.
                      My current plan is to include them in a j2se module rather than j2ee.


                      2. I had to explicitly reference the libs that I needed from jboss-common. When I tried to do:
                       <source id="main">
                       <include input="common"/>
                       <include input="servlet"/>
                       <include input="jaf"/>
                       </source>
                      

                      ... the path was resolved to common/output. So instead I had to include the jars explicitly.
                       <source id="main">
                       <include input="jboss-common.jar"/>
                       <include input="namespace.jar"/>
                       <include input="servlet"/>
                       <include input="jaf"/>
                       </source>
                      


                      I shouldn't have to reference the jars for a local component any differently than for a non-local one, no?


                      The difference between servlet and jboss-common.jar is that an
                      includes

                      element has been created in the top level build for convenience:

                       <includes id="servlet">
                       <include input="servlet-api.jar"/>
                       <include input="jsp-api.jar"/>
                       </includes>
                      


                      The same code be done for common

                       <includes id="jboss-common">
                       <include input="jboss-common.jar"/>
                       <include input="namespace.jar"/>
                       </includes>
                      


                      This is similar to the way we have libraries.ent and modules.ent with the current build.

                      • 8. Re: New Build in JBoss-Head

                        Maybe

                        <includes id="common-project">
                        


                        Is a better name?

                        • 9. Re: New Build in JBoss-Head

                          Ok, thanks -- this makes sense. I've added includes for common-project and j2ee-project.

                          • 10. Re: New Build in JBoss-Head

                            I've hit a similar problem. Except with the testing.

                            Here's an example:

                            The kernel project contains some tests that use JBoss common's JBossXB
                            implementation.
                            It works fine for compilation in that all I need to include is jboss-common.jar.

                            But when it comes to do testing, I need to include the other dependencies
                            of jboss-common.jar for it to be able to instantiate the classes, namely
                            (in fact I only need a subset of these):

                             <include input="commons"/>
                             <include input="concurrent"/>
                             <include input="dom4j"/>
                             <include input="jaxme"/>
                             <include input="log4j"/>
                             <include input="regexp"/>
                             <include input="slide"/>
                             <include input="wutka"/>
                             <include input="xerces"/>
                            


                            With the current build, this isn't a problem (I suppose it depends how you define problem)
                            because the testsuite module just includes nearly all the modules and most of thirdparty.

                            But I was thinking we could do better than that this by adding the dependencies to
                            the manifest of the jar.
                            e.g. jboss-microcontainer.jar would have
                            Class-Path: jboss-common.jar, concurrent.jar

                            We could then automatically add these artifacts to the classpath when
                            running tests.

                            This information could also be useful at runtime, potentially allowing the
                            microcontainer to prevalidate the class dependencies.

                            Whether we use the spec defined Class-Path manifest entry (which has some semantics
                            we may not desire) or define our own entry is up for debate.

                            • 11. Re: New Build in JBoss-Head

                              The only problem I see is that we can't (shouldn't?) modify the manifest of thirdparty libraries. So you would only get the indirect dependencies of jboss libraries. Perhaps this is not a big deal, though.

                              • 12. Re: New Build in JBoss-Head
                                starksm64

                                I already modify all jar manifests when a release is done to tag them with the jboss release info. If a jar actually uses the version manifest (and few do) its preserved, but the implementation headers are updated. Picking the 4.0.1/client/jacorb.jar at random:

                                [starksm@banshee9100 client]$ extcheck -verbose jacorb.jar
                                Target file:jacorb.jar
                                Specification title:JBoss
                                Specification version:4.0.1
                                Specification vendor:JBoss (http://www.jboss.org/)
                                Implementation version:4.0.1 (build: CVSTag=JBoss_4_0_1 date=200412230944)
                                Implementation vendor:JBoss Inc.
                                
                                Comparing with file:/C:/usr/java/j2sdk1.4.2_06/jre/lib/ext/certj.jar
                                Comparing with file:/C:/usr/java/j2sdk1.4.2_06/jre/lib/ext/dnsns.jar
                                Comparing with file:/C:/usr/java/j2sdk1.4.2_06/jre/lib/ext/jsafeFIPS.jar
                                Comparing with file:/C:/usr/java/j2sdk1.4.2_06/jre/lib/ext/ldapsec.jar
                                Comparing with file:/C:/usr/java/j2sdk1.4.2_06/jre/lib/ext/localedata.jar
                                Comparing with file:/C:/usr/java/j2sdk1.4.2_06/jre/lib/ext/rsajsse.jar
                                Comparing with file:/C:/usr/java/j2sdk1.4.2_06/jre/lib/ext/sslj.jar
                                Comparing with file:/C:/usr/java/j2sdk1.4.2_06/jre/lib/ext/sunjce_provider.jar
                                No conflicting installed jar found.
                                



                                • 13. Re: New Build in JBoss-Head

                                  I've already added that to the new build:

                                   <component>
                                   <mkdir dir="@{output}/etc"/>
                                   <copy todir="@{output}/etc" filtering="yes">
                                   <fileset dir="@{component.dir}/src/etc/" includes="**"/>
                                   <filterset>
                                   <filter token="java.vm.version" value="@{component.build.VMVersion}"/>
                                   <filter token="java.vm.vendor" value="@{component.build.VMVendor}"/>
                                   <filter token="specification.title" value="@{component.specTitle}"/>
                                   <filter token="specification.version" value="@{component.specVersion}"/>
                                   <filter token="specification.vendor" value="@{component.specVendor}"/>
                                   <filter token="implementation.title" value="@{component.implTitle}"/>
                                   <filter token="implementation.url" value="@{component.implURL}"/>
                                   <filter token="implementation.version" value="@{component.implVersion}"/>
                                   <filter token="implementation.vendor" value="@{component.implVendor}"/>
                                   <filter token="implementation.vendor.id" value="@{component.implURL}"/>
                                   </filterset>
                                   </copy>
                                   </component>
                                  


                                  The idea would be to extend this to add other information that will be useful
                                  at runtime (including for support purposes).

                                  In this case it would be a list of what a particular jar uses at runtime,
                                  Rather than just a list of jars, this could also include thirdparty version
                                  information and licenses.

                                  We could also indicate that some are optional, e.g. jboss-common.jar
                                  doesn't need log4j.jar (it is an optional plugin for logging with log4j)

                                  We don't need to do this for thirdparty jars, only the our jars.


                                  • 14. Re: New Build in JBoss-Head
                                    starksm64

                                    Ultimately we do want this for thirdparty jars for support as very few projects actually correctly tag their jars so its difficult to know what version of x is in jboss. The new thirdparty management mechanism should make this information available and should encode it as part of the dist build if not the jars themselves.

                                    1 2 Previous Next