MatrixVersion design question

[clebert.suconic]

The way I'm doing cross version tests is based in a directory.

By defining a directory, for example /jboss-versions, I'm going to look for sub-directories and dynamically execute the version check against each client defined on that directory.

For example, while testing 4_0_2, if you have 3_0_0 and 4_0_0 at /jboss-versions we will be doing cross refs tests against 3_0_0 and 4_0_0. This is very dynamic, and when we move to 4_0_3 the only thing we will have to do will be to add another client library on this directory.

The only issue is that we won't any reference to 4.0.2 and 4.0.1 in the testsuite/build.xml as this test is done dynamically. So, the question is: Is this an issue?


Clebert

[ryan.campbell]

I don't think this is a huge issue.

The only problem I see is that the versions we are testing are not explicitly documented in the source code. They are implicit based on what directories your ant task finds.

Generally, it is better to have explicit expectations of the environment, rather than letting your environment to drive your tests. IE, these are the versions we want to test and we will fail if we don't find them.

I suggest that the desired compatability matrix should be documented in testsuite/build.xml.

[clebert.suconic]

By the other side, when developers are using the matrix testsuite they may want to test against a specific version.

Adding this to the build.xml would require everybody to run only targets we have mapped in the build.xml.

IMO, the way we have now, anyone who wants to execute the matrix test can manage his own environment while we can put any version we want at cruisecontrol.

[starksm64]

I'm not seeing how 4.0.2/4.0.1 are treated specially, and they should not be so can you explain how these are run by the matrix tests?

I don't see much benefit to including a list of expected versions in the build.xml as its non-functional unless the directory and these versions exist. This is really a test to be run by qa and published to the testsuite results site. Occasionally developers will run one legacy client against the pending current release to fix issues.

[clebert.suconic]

We are trying to fix the testsuite cruisecontrol to post these results.

I will still have to filter out some tests, but I want to do it after I got some output to show developers how this is working.

I will post a message in the dev forum since we get the testsuite fixed/matrix working.