Matrix Tests List - Input requested from Project leads

[clebert.suconic]

This is not done yet, but this gives an idea about how this is working:
http://cruisecontrol.jboss.com/cc/artifacts/jboss-4.0-testsuite/20050504194421/results/org/jboss/test/compatibility/test/matrix/package-summary.html


I've created a Test called MatrixTestContainer that includes configured tests passed as parameters by the test suite.

Every configured MatrixTestContainer will run over all the defined clients.

For example, we have MatrixTestContainer(BMP_v_3_2_7), MatrixTestContainer(BMP_v_4_0_2) and MatrixTestContainer(BMP_v_4_0_1_sp1). You can realize by looking at the link that the compatibility with 3_2_7 doesn't work. (this is expected I guess).


So, I will need every project lead to take a look on these results, and tell me if I should add other tests to the suite. There is few obvious tests that I will remove anyway (as Scott has appointed some tests), but I would prefer to have more people looking at it and validating it.

Like, all the JMS tests are passing. I didn't see any other tests in our tree, but it doesn't cost anything having the JMS lead validating it :-)

thanks,



Clebert Suconic

[tom.elrod]

I'm sorry, but I am a little lost. So this is to test all the different client version with all the other server versions?

How do I tell if a 4.0.1 client passed all tests calling against a 4.0.2 server? Any way to make it so there are two axis to the matrix? For example:

client | 4.0.1 server | 4.0.2 server
4.0.1 | 0 failures | 12 failures
4.0.2 | 0 failures | 0 failures

[starksm64]

This is a test of client oriented tests using different client jars against one version of the server. In this case 4.0.2.

[tom.elrod]

Guess the issue I am having is how would I be able to determine this from looking at http://cruisecontrol.jboss.com/cc/artifacts/jboss-4.0-testsuite/20050504194421/results/org/jboss/test/compatibility/test/matrix/package-summary.html
?

[starksm64]

You can't. This should be a frame based report with the index specifying the server version under test. If this stays as a single report there needs to be a header added for each server version under test.

[clebert.suconic]

In the URL I sent you have for example:

For the BMP tests:


MatrixTestContainer(BMP_v_4_0_2) - This is BMP tests executed using the client library version 4.0.2
MatrixTestContainer(BMP_v_4_0_sp1) - The same tests executes againt 4.0.1_sp1. (I think I need to rename the file.. I missed a 1)

And so on...


Each container will have several other tests:

Like:
org.jboss.test.bmp.test.BmpUnitTestCase::testBMP was contained by the container.


Let me know if you still have an issue.


Clebert Suconic

[clebert.suconic]

How do I tell if a 4.0.1 client passed all tests calling against a 4.0.2 server? Any way to make it so there are two axis to the matrix? For example:

The test is always done against the current version.

If it's needed to test against older versions we will have to test against other branches.


We are using the plain JUnitReport, so we can't change the output format for this, that's I'm using the concept of container and tests.

Clebert