[starksm64]

Yes, something like that.

[ruel.loehr]

I have been working on implementing the functionality of the third party test which Scott defined.

Currently, I am able to set up a graph based upon the components defined in the top-level build and then traverse it and pull down the necessary component-info.xml files from the repository (in this case, the component-info.xml for hibernate). The component-info from the repository currently looks like this:

<project name="hibernate-component-info">

 <component id="hibernate"
 licenseType="lgpl"
 version="3.0.5"
 projectHome="http://hibernate.org/"
 description="ultra-high performance object/relational persistence">

 <artifact id="hibernate3.jar"/>
 <artifact id="hibernate-annotations.jar"/>
 <includes id="thirdparty">
 <include component="asm" />
 <include component="antlr" />
 </includes>
 <export>
 <include input="hibernate3.jar"/>
 <include input="hibernate-annotations.jar"/>
 </export>
 </component>


</project>

My next step is to resolve the secondary dependencies, however version information is currently not defined at the component-info level for dependencies. I would like to confirm what this should look like and if it is acceptable. Is the following suitable to what you have in mind?

<project name="hibernate-component-info">

 <component id="hibernate"
 licenseType="lgpl"
 version="3.0.5"
 projectHome="http://hibernate.org/"
 description="ultra-high performance object/relational persistence">

 <artifact id="hibernate3.jar"/>
 <artifact id="hibernate-annotations.jar"/>
 <import component="asm">
 <compatible version="1.3.4"/>
 </import>
 <import> component="antlr">
 <compatible version="3.3.2"/>
 </import>

 <export>
 <include input="hibernate3.jar"/>
 <include input="hibernate-annotations.jar"/>
 </export>
 </component>


</project>

Thanks!

Ruel Loehr
JBoss QA

[starksm64]

See the following thread for the far from complete discussion on this.

http://www.jboss.org/index.html?module=bb&op=viewtopic&t=64049

I don't have a problem with the proposed syntax as a starting point, but we should have a notion of an overriding version coming in from the top level build as an example.

[starksm64]

The other more likely source of the overriding dependency component version info would be the xxx/version-info.xml descriptor mentioned in the thread I referenced.

It does make sense to state some base dependent version info in the component-info.xml of the dependee, but the actual version pulled into a given project has to be a compatible union of all dependees and the current thinking is that this is coming either from the master build directly or by a label that is mapped onto a version using the xxx/xxx/version-info.xml descriptor.

[ruel.loehr]

I have implemented the functionality required to demonstrate the third party test which was described earlier in this thread.

mkdir test
cd test
cvs checkout jbossas
cvs checkout tools
cd jbossas
ant -f secondary-dependency-test.xml testdependencies

The secondary-dependency-test.xml contains a task, synchronizeinfo, which will construct a graph containing the componentref's which are defined in the top level build. The task resolves each componentref into a component, resolves its dependencies as defined in the corresponding component-info.xml file, and adds the component to the build. This process is repeated until all dependencies are resolved.

The target testdependencies then performs a check to ensure the proper jar files have been downloaded. In this case, after synchronization has been performed the following will have been retrieved from the repository:

/thirdparty/hibernate/3.0.5/component-info.xml
/thirdparty/hibernate/3.0.5/lib/hibernate3.jar
/thirdparty/hibernate/3.0.5/lib/hibernate-annotations.jar

/thirdparty/antlr/2.7.5H3/component-info.xml
/thirdparty/antlr/2.7.5H3/lib/antlr.jar

/thirdparty/asm/1.5.3/component-info.xml
/thirdparty/asm/1.5.3/lib/asm.jar
/thirdparty/asm/1.5.3/lib/asm-attrs.jar

Ruel Loehr
JBoss QA

[starksm64]

Looks good. I'll move forward with trying to setup the jboss-4.0/thirdparty structure from a new build script.

[starksm64]

In working on generating thirdparty from the repository I have run into this problem. If I update the secondary-dependency-test.xml to include multiple component-refs that depend on apache-logging (here jgroups and tomcat):

 <componentref name="hibernate"
 version="3.0.5" />
 <componentref name="jgroups"
 version="2.2.7" />
 <componentref name="tomcat"
 version="5.5.9" />

the build fails because there are multiple synchronize.apache-logging targets:

[starksm@banshee9100 jbossas]$ ant -buildfile secondary-dependency-test.xml
Buildfile: secondary-dependency-test.xml
 [null] Created dir: C:\cvs\JBossHead\jboss-build\thirdparty\jgroups\2.2.7
 [null] Getting: http://cruisecontrol.jboss.com/repository/jgroups/2.2.7/component-info.xml
 [null] To: C:\cvs\JBossHead\jboss-build\thirdparty\jgroups\2.2.7\component-info.xml
 [null] Created dir: C:\cvs\JBossHead\jboss-build\thirdparty\tomcat\5.5.9
 [null] Getting: http://cruisecontrol.jboss.com/repository/tomcat/5.5.9/component-info.xml
 [null] To: C:\cvs\JBossHead\jboss-build\thirdparty\tomcat\5.5.9\component-info.xml
 [null] Created dir: C:\cvs\JBossHead\jboss-build\thirdparty\apache-logging\1.0.4jboss
 [null] Getting: http://cruisecontrol.jboss.com/repository/apache-logging/1.0.4jboss/component-info.xml
 [null] To: C:\cvs\JBossHead\jboss-build\thirdparty\apache-logging\1.0.4jboss\component-info.xml.
 [null] Getting: http://cruisecontrol.jboss.com/repository/apache-logging/1.0.4jboss/component-info.xml
 [null] To: C:\cvs\JBossHead\jboss-build\thirdparty\apache-logging\1.0.4jboss\component-info.xml.
 [null] Created dir: C:\cvs\JBossHead\jboss-build\thirdparty\apache-modeler\1.1
 [null] Getting: http://cruisecontrol.jboss.com/repository/apache-modeler/1.1/component-info.xml
 [null] To: C:\cvs\JBossHead\jboss-build\thirdparty\apache-modeler\1.1\component-info.xml.
 [null] Created dir: C:\cvs\JBossHead\jboss-build\thirdparty\commons-el\1.0
 [null] Getting: http://cruisecontrol.jboss.com/repository/commons-el/1.0/component-info.xml
 [null] To: C:\cvs\JBossHead\jboss-build\thirdparty\commons-el\1.0\component-info.xml
.
BUILD FAILED
C:\cvs\JBossHead\jboss-build\jbossas\secondary-dependency-test.xml:38: Duplicate
 target: `synchronize.apache-logging'

There needs to be some tracking of whether a synchronization target has already been added.

[ruel.loehr]

The problem was occurring when the same dependency occurred more than once in the graph. Each time the dependency appeared, we would import its component-info.xml file, which of course, led to the duplicate components being present in the build and ultimately this caused duplicate targets to be created.

I have modified the functionality so that a check is performed before importing a component info.xml file. If the component is already present in the ant project (e.g. we have already imported it's component-info.xml), then a new one is not created. The dependency tree is still accurate, we are simply not adding the component to the project if it already exists.

I have already committed these changes so that work on the thirdparty dir can proceed. The visitor attribute functionality is next on the task list.

[ruel.loehr]

I have implemented the functionality which will allow a visitor to be defined as an attribute and then sent to the graph.

A new target has been created in the secondary-dependency-test.xml file to support this.

<target name="generate-license-xml" depends="synchronize">
 <visit-componentref-graph
componentVisitor="org.jboss.ant.util.graph.ComponentRefGraphLicenseVisitor"/>
</target>

When this target is executed, the class defined by componentVisitor will be instantiated and sent to the graph for a visit. Currently, this visitor does nothing more than print out the component id which is contained in the vertex it is visiting.

To implement the full use case of creating the thirdparty-licenses.xml, the visit method of class org.jboss.ant.util.graph.ComponentRefGraphLicenseVisitor and possibly the execute method of class: org.jboss.ant.tasks.build.VisitComponentRefGraphTask will need to be modified, depending on the final implementation.

To see this in action execute the following commands:

mkdir test
cd test
cvs checkout jbossas
cvs checkout tools
cd jbossas
ant -f secondary-dependency-test.xml generate-license-xml

Ruel Loehr
JBoss QA

[starksm64]

So this visit-componentref-graph task is something I can use for any navigation task with this update? There is an additional need to deal with a mismatch between the new build thirdparty repository that includes version information in the output structure, and the thirdparty legacy structure that has no such notion. The problem being that all the existing classpaths have no version notion and I don't want to have to propgate the version info other than automatically.

I'll look into the new visitor capability to see if this can be handled as part of the license-info.xml generation as well.

[ruel.loehr]

In it's current form, yes. The task takes whatever vistitor was defined, instantiates it, and sends it to the graph where it will visit each node. The visitor will perform the exact same set of operations on each vertex.

That being said...considering the case of license generation for example. If the visitor were to handle doing all the work then for each vertex it encounters it would need to check if a license file already exists/or create one, get the component version information and then append some information to the license file.

Another approach, if we didn't use a generic task, would be for the visitor to only be responsible for compiling a list of version information. Once it returned to the task, the task itself would use the information the visitor gathered to create the license xml. This is the approach I used for the synchronize info task.

Ultimately, I think there should be a generic task to handle any and all visitors who will be doing simple things which do not requrire any pre or post processing. For complex functionality that we do not wish to repeat at each vertex, we will need more specific tasks.

[ruel.loehr]

I've created a new task which will generate a libraries.ent file based on the components contained in the graph which includes the correct version numbers in the classpath.

A new target has been created in the secondary-dependency-test.xml file to support this.

<target name="generate-lib-file" depends="synchronize">
 <gen-lib-file filename="versionedLibraries.ent" path="../tools/etc/buildmagic/" />
 </target>

The target accepts 2 attributes, file name, and the path to where the file should be written. If these are not included, a default filename and path are used.

The general premise is that for each component in the graph which the visitor encounters the following will be written to the file:

1) An identifier comment
2) A root property name
3) A library property name (now with version number included in the path)
4) A list of path elements (the jar files)

The values used are currently (this may need to be adjusted) based off of the component id as shown below for component "antlr".

<!-- antlr -->
<property name="antlr.antlr.root" value="${project.thirdparty}/antlr"/>
<property name="antlr.antlr.lib" value="${antlr.antlr.root}/2.7.5H3/lib/"
<path id="antlr.antlr.classpath">
 <pathelement path="${antlr.antlr.lib}/antlr.jar"/>
</path>

For some entries in the existing libraries.ent file, various naming schemes were used for the property name. Finding a common method to faithfully reproduce them programatically may be troublesome, but I think we can get close.

To see this in action execute the following commands:

mkdir test
cd test
cvs checkout jbossas
cvs checkout tools
cd jbossas
ant -f secondary-dependency-test.xml generate-lib-file
cd ../tools/etc/buildmagic
emacs versionedLibraries.ent

Ruel Loehr
JBoss QA

[ryan.campbell]

The only problem I see is that the properties and classpaths produced by this script will be the new ones based on the repository, and will therefore be slightly different (ie, apache-logging instead of apache-commons). So the existing libraries.ent and each module's build.xml classpaths need to be updated to work with these slightly different classpaths.

Scott, if you could commit the work you have for the toplevel build, we can work on updating the the libraries.ent and module build.xml's to be compatible with the generatedLibraries.ent.

Just to be clear, we won't be changing anything in /thirdparty.

[starksm64]

I have checked in what I last had as the jboss-4.0/build/build-thirdparty.xml script. There should be no need to update the build.xml scripts to test this out. Using the jboss-4.0.x alias which is equivalent to the jboss-4.0 without the thirdparty directory strucutre, one should be able to do:

cvs co -r Branch_4_0 jboss-4.0.x
cd jboss-4.0/build
ant -f buildfile build-thirdparty.xml
ant

The libraries.ent in this workspace should simply be replaced:

<target name="generate-lib-file" depends="synchronize">
 <gen-lib-file filename="libraries.ent" path="../tools/etc/buildmagic/" />
 </target>

Once this is working we can update the build files and move the libraries.* files to the thirdparty structure where it belongs as a thirdparty artifact.