Build Repository Structure

[ryan.campbell]

The build repository has the following purposes:

+ Replace the thirdparty module. Instead of storing libraries in CVS, they are pulled from the HTTP build repository by the build system.

+ Enable partial builds. Developers can check out the source for some components, while the snapshot (or versioned) builds of other components are pulled from the build repository.

The repository has the structure: [organization-module]/[version]/[artifact]

Let's take the example of a jboss module:

jboss-cache/
 1.1.3/
 jboss-cache-1.1.3.jar
 LICENSE
 component-info.xml
 1.2-SNAPSHOT/
 jboss-cache-1.2-SNAPSHOT.jar
 LICENSE
 component-info.xml

From this we see 1) JAR artifacts contain the version string for convenience 2) License is versioned with component artifacts 3) component-info.xml contains metadata about the component

Snapshots are produced by CruiseControl upon commit. If a snapshot fails to build in any of its integration projects, it will not be published. Versioned artifacts are published by developers to a CVS repository. A scheduled process updates the build repository from CVS at regular intervals.

component-info.xml tells you about the source of the component :

<component-info name="jboss-cache" version="1.2-SNAPSHOT">
 <repository
 cvsroot="cvs.sourceforge.net/cvsroot/jboss"
 module="jboss-cache"
 branch="Branch_1_2"
 projectHome="http://www.jboss.org/products/jbosscache"/>
 <license type="lgpl" file="LICENSE"/>
 <artifact name="jboss-cache-1.2-SNAPSHOT.jar"/>
</component-info>

So, the build system could expose a checkout.jboss-cache target which downloadeds the component-info.xml and uses the supplied information to checkout jboss-cache from cvs.

For log4j, the repository would look like:

apache-log4j/
 1.2.8/
 log4j-1.2.8.jar
 LICENSE
 component-info.xml

In this case, the metadata is mostly for documentation purposes.

All artifacts are cached in a local build repository which mirrors the online repository.

[adrian.brock]

I think it would be useful to mirror our output folder structure in the
repository. This would show what is libraries (lib), resources (resources),
javadocs (api), docbook (doc), etc.

e.g. as a developer I probably wouldn't be interested in pulling down javadocs,
docbook, examples/tutorial source code, etc., if I just want to build against that
projects binary.
But these will be useful when creating any final release that
includes that information.

It might be more logical to have the notion of main project/version first,
but such a structure would probably be only useful to somebody browsing
the repository on-line?
e.g.

JBossAS/
 4.0.1/
 jboss-common/
 lib/
 jboss-common.jar
 resources/
 {runtime config xml here}
 api/
 {javadocs here}
 doc/
 {docbook here}
 LICENSE
 component-info.xml
 jboss-system/
 lib/
 jboss-system.jar
 run.jar
 shutdown.jar
 resources/
 {runtime config xml here}
 api/
 {javadocs here}
 doc/
 {docbook here}
 LICENSE
 component-info.xml

[ryan.campbell]

I think mirroring the output directory makes sense.

However, having a top level project directory (JbossAS) prevents us -- at least conceptually-- from having jboss-common be a part of multiple integration builds independent of JBossAS. For example, the integration build for jboss-aop standalone will need to reference jboss-common.

Also I think the flatter structure will make it easier when performing reverse lookups - ie, I've got a jboss-jmx-5.1.1.jar, where can I get source/javadocs/etc for it.

[starksm64]

So we need to finalize the repository structure and component-info so I can recreate the jboss-4.0.x thirdparty contents from a jbossbuild.xml.

One current issue is the license info. The existing jboss-4.0/thirdparty/licenses/thirdparty-licenses.xml now has an index of the licenses used along with the acceptable source code headers used in the codebase to reference the license. The latter is used by the org.jboss.tools.license.ValidateLicenseHeaders utility to run across every source file in the codebase to validate that a license header exists, and that it maps to an acceptable license.

So the first change that needs to be made to the repository structure is to pull out all license files from the component dirs into a repository/licenses structure so that there is a canonical representation of the licenses in use. When I look at a licenseType="apache-2.0" reference from a component-info.xml, its a reference to either repository/licenes/apache-2.0 contents, or a reference to an element in a single repository/licenes/license-info.xml element not unlike the current thirdparty-licenses.xml licenses section:

<licenses>
 <license id="apache-1.1" licenseFile="apache-1.1.txt">
 <terms-header id="apache#0">
 * The Apache Software License, Version 1.1
...

I would lean towards a single repository/licenes/license-info.xml with the license text in repository/licenes as currently is done in jboss-4.0/thirdparty/licenses.

I have some general questions about the overall build structure relationships and xml descriptors format that I'll create a seperate post on.

[starksm64]

Another thing that is missing in the component-info.xml is an expression of the depenencies on other repository components. For example, the javagroups component has no expressed dependency on the commons-logging component. I don't think the consumer of javagroups should have to understand implied dependencies.

[adrian.brock]

Another thing that is missing in the component-info.xml is an expression of the depenencies on other repository components. For example, the javagroups component has no expressed dependency on the commons-logging component. I don't think the consumer of javagroups should have to understand implied dependencies.

Yes, this has broken the "runtest" target where a test would automatically
import the dependencies of its dependencies without each module having
to worry about what each project uses.

e.g. kernel uses jboss-common.jar which was compiled over xerces, etc.

[ejort@htimes2 kernel]$ ant -f jbossbuild.xml runtest
Buildfile: jbossbuild.xml

runtest.tests:
Unable to locate 'commons-logging.jar' in Class-Path of jboss-common.jar
Unable to locate 'commons-httpclient.jar' in Class-Path of jboss-common.jar
Unable to locate 'commons-discovery.jar' in Class-Path of jboss-common.jar
Unable to locate 'dom4j.jar' in Class-Path of jboss-common.jar
Unable to locate 'jaxmexs.jar' in Class-Path of jboss-common.jar
Unable to locate 'log4j.jar' in Class-Path of jboss-common.jar
Unable to locate 'gnu-regexp.jar' in Class-Path of jboss-common.jar
Unable to locate 'webdavlib.jar' in Class-Path of jboss-common.jar
Unable to locate 'dtdparser121.jar' in Class-Path of jboss-common.jar
Unable to locate 'xercesImpl.jar' in Class-Path of jboss-common.jar

[adrian.brock]

Bug report:
http://jira.jboss.com/jira/browse/JBBUILD-72

[adrian.brock]

One problem with doing this, is that it would potentially allow a

<source>

to indirectly reference a component that was imported through a component-info.xml

We don't want this to happen.

I would suggest that the two different mechanisms for importing components
(directly and indirectly) set a threadlocal which is then stored inside the
component/artifact definition.

The

<source>

would then throw a BuildException if it attempted to
reference something that was not directly imported.

[ryan.campbell]

I have created a JIRA task for the license consolidation.

http://jira.jboss.com/jira/browse/JBBUILD-73

[starksm64]

I'm not following how that lone source element triggers imports.

Going through this currently for the 4.0 thirdparty, the unversioned import still leaves open the problem of what version is being referenced. We can build a graph of components and attempt to resolve the components, but I see a number of components that have references to components only used by them (dom4j to jaxen, cglib to asm). We can attempt to taken the latest version of each unversioned reference, but this requires a latest link pointing to latest, or an index of the components so that it can be computed.

The import should also support a version in the case that there is an explicit version requirement. Are you working on adding the import addition Ryan? If not, I'll look at this since I need to complete it to finish the thirdparty build.

[ryan.campbell]

Currently, the version metadata is only used when synchronizing, and this only at the toplevel build. This was purposeful, because I assumed that the toplevel build should have a complete manifest of all versions used in the build, and that there should only be one version of each component used.

At the component level, you don't need to resolve imported component versions because they have already been specified and downloaded by the toplevel build. The component uses whatever is in thirdparty/component-id.

If we were using the component definitions to synchronize, we would need to do as you suggest. What am I missing???

I agree the import mechanism should support a version for the purposes of flagging unmet version requirements.

I'm not working on the import functionality currently.

[starksm64]

As was mentioned in this post which spawned the JBBUILD-72 issue, its not practical for the component user to have to know the dependencies of the component. Case 1, hibernate depends on cglib and asm, neither of which are used anywhere else in jbossas. In creating the top level build I only know I need hibernate 3.0.3. The component-info.xml for hibernate 3.0.3 needs to tell the synchronize process that cglib and asm are also needed.

This should not be in the jbossas build component list, because hibernate 3.0.4 may no longer have these dependencies. A given components dependeices should be a listing of those apis that are actually used.

I'll take a look at import addition and assign myself the task as I am looking at how the build actually works currently.

[adrian.brock]

What do you do if two components specify different versions
of the same dependency?

My guess is to take the latest and warn the developer if there is nothing
saying a component has been tested with that version.

This would then lead to a QA task where they test it and either:
1) update the manifest to say that version is ok
2) choose a later version of the component in question that has been tested
3) get the component developer(s) to fix any problems found from the testing

[starksm64]

Right, assuming by manifest you mean the repository component-info.xml import statements?

We do need to figure out to handle the multiple version mismatches that will show up. As part of generating the license-info.xml from the thirdparty repository references, detection of version conflicts will be flagged. That should in fact be a custom cruisecontrol action that results in an email to qa and the development list. Maybe its best to require the version on component-info.xml import statements rather than trying to default to the latest to start with in the absence of such info to start with.

As you say I don't see any magic resolution. There needs to be testing and lead on the component using the older version consulted to see if the changes look to be problematic. This gets back to exactly what should be in the thirdparty repository. Release notes, source and patches would seem to be a good minimum in addition to the binaries.

[ryan.campbell]

As was mentioned in this post which spawned the JBBUILD-72 issue, its not practical for the component user to have to know the dependencies of the component. Case 1, hibernate depends on cglib and asm, neither of which are used anywhere else in jbossas. In creating the top level build I only know I need hibernate 3.0.3. The component-info.xml for hibernate 3.0.3 needs to tell the synchronize process that cglib and asm are also needed.

Ok, so if we want this behavior to apply to toplevel builds, the only problem I see is that the toplevel build still has the responsibility to define *where* to put cglib and asm in the release.

I guess we could just decide that all transitive dependencies are placed in the same directory as the direct dependency? E.g., if hibernate goes in server/all/lib, so do asm and cglib.