cvs2svn Repository Import Test

[dsicore]

I tested an import of our cvs repository using cvs2svn to determine what would happen to the CVSROOT/modules file used to create aliases for all the modules appearing in a jboss-head (and others) checkout.

This is what I found:

1. The aliases are ignored.

2. Branches and tags still exist in the new repository.

3. As expected, the following repository structure is created:

/trunk
/branches
/tags

4. One cannot just checkout jboss/trunk and expect it to build. The raw module names are used in the working copy on checkout of trunk. For example:

$ svn ls https://svn.jboss.org/repos/trunk
...
CacheBenchFwk/
JBossCache/
JBossRemoting/
JVoip/
TrailBlazer/
admin/
applications/
blocks-jboss/
build/
build-jboss/
buildmagic/
cheese/
cluster-jboss/
...

Now, how to build jboss-head?

In order to build a working copy reconstructing the directories in a way similar to the CVSROOT/modules file, I created an /externals repo location to contain structures mapping to the correct layout for any given alias structure (i.e., jboss-head, jboss-4.0, jboss-4.0.x, jboss-3.2, etc). I only re-created jboss-head for this proof of concept.

$ svn mkdir https://svn.jboss.org/repos/test/externals -m "Creating dir to hold project definitions."
$ svn mkdir https://svn.jboss.org/repos/test/externals/jboss-head -m "Creating the jboss-head project."

To create the proper jboss-head structure, I mapped, one by one, the aliases in the CVSROOT/modules files to the proper locations in trunk. I created a temp file to define the externals:

snippet from /tmp/externals.txt:

...
admin-console https://svn.jboss.org/repos/test/trunk/jboss-admin-console
aop https://svn.jboss.org/repos/test/trunk/jboss-aop
jbossas https://svn.jboss.org/repos/test/trunk/jbossas
aspects https://svn.jboss.org/repos/test/trunk/jboss-aspects
build https://svn.jboss.org/repos/test/trunk/build/jboss
cluster https://svn.jboss.org/repos/test/trunk/jbossmx
common https://svn.jboss.org/repos/test/trunk/jboss-common
connector https://svn.jboss.org/repos/test/trunk/jbosscx
console https://svn.jboss.org/repos/test/trunk/jboss-console
container https://svn.jboss.org/repos/test/trunk/container
...

Then, I set the svn:externals property on jboss-head:

$ svn propset svn:externals -F /tmp/externals.txt ./workingcopy/externals/jboss-head
property 'svn:externals' set on 'jboss-head'
$ svn ci -m "Setting externals for jboss-head." ./workingcopy/externals/jboss-head
Sending jboss-head
Committed revision 57085.

Now, when I updated my working copy of ./externals/jboss-head:

$ cd ./workingcopy/externals/jboss-head && svn up

Fetching external item into 'admin-console'
A admin-console/jbossbuild.xml
A admin-console/.classpath
...
 U admin-console
Updated external to revision 57085.

Fetching external item into 'aop'
A aop/SuperClassesFirst_Changes_For_AOP.txt
A aop/jbossbuild.xml
...
U aop
Updated external to revision 57085.

...

This builds out a working copy called jboss-head containing everything one would need, in the same structure as a CVS checkout of jboss-head, to do a build. I checked out the working copy and jboss-head builds successfully.

There were a few passes at this; however, the CVS/modules file contained non-existent aliases for jboss-head in /trunk. In other words, things like jboss-cache have no commits in CVS HEAD of jboss-head and doesn't necessarily exist. So, when cvs2svn mapped over the repository branches, including /trunk, it correctly created a /trunk which does not contain jboss-cache. I had to remove external definitions which no longer exist in trunk. Bottom line: we'll have to keep our external definitions in synch with what reality is in whatever branches they refer to. This is a good thing, I believe.

Important details regarding this externals mechanism:

1) Branches must contain their own external references. For example, external definitions for Branch_4_0 must not only specify which aliases are required, but they also must point to the proper branch of the projects it references (i.e., it uses a special branch of cluster for example). Since externals are absolute URLs, moving or copying a directoy to which they are attached doesn't affect what gets checked out as an external. This means, if I branch /externals/jboss-head to /branches/externals/my-jboss-head, the branch still points to project dirs in /trunk. You must update your externals in your branch.

2) Working copies created via the externals definition support are still disconnected from the primary working copy (on whose versioned directories the svn:externals property was actually set. If you want to commit changes you've made in one or more of those external working copies, you must run svn commit explicitly in those working copies--committing on the primary working copy will not recurse into any external ones. (words taken from the svn manual). This can be a good thing as we are moving to binary dependencies anyway and may not necessarily want to commit to a bunch of different projects unless we explicitly say so.

3) External definitions are versioned. This means, if I go back in time in the repository, I will find the proper external definitions for jboss-head at that time. Our externals definitions are not immortal like they are in the CVSROOT/modules file. They are maintained to be accurate for the time they are being used.

4) Since externals can reference svn structures existing in any repository, either within the main JBoss repository or say Hibernate's external repository, we can pick and choose which projects make up structures, like jboss-head or jboss-4.0.x, etc., and use this to reflect a true binary dependency mechanism where one only needs to checkout the source he needs and rely on the jboss maven build system to fetch the binaries required.

5) Again, since externals are absolute URLs, they reference a single repository. For anonymous access, users would be hitting http://anonsvn.jboss.org/repos/jboss, or an equivalent. When they do a checkout, they will get externals which reference https://svn.jboss.org/repos/jboss and be asked to login. We need to decide to either a) offer read only access to all users on our primary repo and do away with the anonrepo, or b) provide a set of externals definitions referencing the anon repository URLs (and be kept in sync with the committer projects).

6) Externals are moving targets. In other words, when I point an external to /trunk of a project, my working copy is updated after someone checks something into /trunk of their project and I update my externally referenced copy. There will be times when we want to pin down to certain releases using either references to certain revisions or tags.

This test repository can be viewed at the following locations:

Committer: https://svn.jboss.org/repos/test
Anonymous: http://anonsvn.jboss.org/repos/test

[starksm64]

1) Branches must contain their own external references. For example, external definitions for Branch_4_0 must not only specify which aliases are required, but they also must point to the proper branch of the projects it references (i.e., it uses a special branch of cluster for example). Since externals are absolute URLs, moving or copying a directoy to which they are attached doesn't affect what gets checked out as an external. This means, if I branch /externals/jboss-head to /branches/externals/my-jboss-head, the branch still points to project dirs in /trunk. You must update your externals in your branch.

What about everytime I tag a branch for a release? As I understand svn there is really no difference between a tag and branch. Its just a naming convention with different url roots?

2) Working copies created via the externals definition support are still disconnected from the primary working copy (on whose versioned directories the svn:externals property was actually set. If you want to commit changes you've made in one or more of those external working copies, you must run svn commit explicitly in those working copies--committing on the primary working copy will not recurse into any external ones. (words taken from the svn manual). This can be a good thing as we are moving to binary dependencies anyway and may not necessarily want to commit to a bunch of different projects unless we explicitly say so.

I don't understand how this applies to the jboss-head example. Your saying that changes made in the jboss-head/server directory and jboss-head/system directory cannot be committed at once? I never do a global commit across all modules so this seems like a minor issue.

3) External definitions are versioned. This means, if I go back in time in the repository, I will find the proper external definitions for jboss-head at that time. Our externals definitions are not immortal like they are in the CVSROOT/modules file. They are maintained to be accurate for the time they are being used.

A good thing.

4) Since externals can reference svn structures existing in any repository, either within the main JBoss repository or say Hibernate's external repository, we can pick and choose which projects make up structures, like jboss-head or jboss-4.0.x, etc., and use this to reflect a true binary dependency mechanism where one only needs to checkout the source he needs and rely on the jboss maven build system to fetch the binaries required.

I don't follow how the ability to reference external repositories helps with the binary usecase. Take the simplest refactoring currently underway which is the splitup of the jboss-head/common into jboss-common-core.jar, jboss-xml-binding.jar, jboss-logging.jar that will be integrated via a maven2 repository. The jboss-head/common simply disappears at that point. The only useful source variation we have talked about is one that requires support from maven. If I wanted to debug a version of the jboss common jars the idea is that I would be able to set a property that indicated the dependency on jboss-common-core 1.0.0.GA was a source level dependency, and this should cause the source to be pulled from the repository(or a tag cvs/svn repository) to obtain a maven project structure that would be built to create the binary.

5) Again, since externals are absolute URLs, they reference a single repository. For anonymous access, users would be hitting http://anonsvn.jboss.org/repos/jboss, or an equivalent. When they do a checkout, they will get externals which reference https://svn.jboss.org/repos/jboss and be asked to login. We need to decide to either a) offer read only access to all users on our primary repo and do away with the anonrepo, or b) provide a set of externals definitions referencing the anon repository URLs (and be kept in sync with the committer projects).

That is something of a problem as I don't see providing read-only access to the dev repository lest we open up the performance problems sourceforge went through. It would be nice if there was a notion of property references that the externals files could use to make them portable between repositories. Can we setup a trigger that updates the anon repo externals file with a url subsitution when the main repository file is changed?

6) Externals are moving targets. In other words, when I point an external to /trunk of a project, my working copy is updated after someone checks something into /trunk of their project and I update my externally referenced copy. There will be times when we want to pin down to certain releases using either references to certain revisions or tags.

In general we would only want to fix versions when creating a tagged release. This gets back to 1).

This test repository can be viewed at the following locations:

Committer: https://svn.jboss.org/repos/test
Anonymous: http://anonsvn.jboss.org/repos/test

I don't see any files under the https://svn.jboss.org/repos/test/externals/jboss-head for example.