I don't think this is a huge issue.
The only problem I see is that the versions we are testing are not explicitly documented in the source code. They are implicit based on what directories your ant task finds.
Generally, it is better to have explicit expectations of the environment, rather than letting your environment to drive your tests. IE, these are the versions we want to test and we will fail if we don't find them.
I suggest that the desired compatability matrix should be documented in testsuite/build.xml.
By the other side, when developers are using the matrix testsuite they may want to test against a specific version.
Adding this to the build.xml would require everybody to run only targets we have mapped in the build.xml.
IMO, the way we have now, anyone who wants to execute the matrix test can manage his own environment while we can put any version we want at cruisecontrol.
I'm not seeing how 4.0.2/4.0.1 are treated specially, and they should not be so can you explain how these are run by the matrix tests?
I don't see much benefit to including a list of expected versions in the build.xml as its non-functional unless the directory and these versions exist. This is really a test to be run by qa and published to the testsuite results site. Occasionally developers will run one legacy client against the pending current release to fix issues.
We are trying to fix the testsuite cruisecontrol to post these results.
I will still have to filter out some tests, but I want to do it after I got some output to show developers how this is working.
I will post a message in the dev forum since we get the testsuite fixed/matrix working.