Version compatibility tests

[clebert.suconic]

As for http://jira.jboss.com/jira/browse/JBAS-1598, I'm creating a TestSuite that will execute a bunch of defined tests against 4.0.1 and 4.0.0.

In other words, we will execute tests against current version using client libraries from 4.0.0 and them 4.0.1.

The problem I'm having now is to define what's the basic set of tests I could use in that suite. Right now I'm using everything defined into "tests-standard-unit" and this seems to be huge.

So... if you have any clue for shrinking the number of tests used int that list will be more than welcome. I guess we want basically test client libraries, so I guess we will have to execute mostly remote tests.

Thanks,


Clebert Suconic

[starksm64]

Yes, that is what I wanted to test. Any client facing api needs to be tested, so remote ejbs, jms, webservices, jndi, jmx. These all have well defined packages in the testsuite.

[clebert.suconic]

My point is to find a minimal set of tests.

For example in webservices,
we have about 28 or 27 packages in the testsuite. (I've copied in this message for convenience).
Do we need to exercize all of them to make sure the client API works?

List of tests in web-services:

org.jboss.test.webservice.admindevel
org.jboss.test.webservice.attachment
org.jboss.test.webservice.contextroot
org.jboss.test.webservice.encstyle
org.jboss.test.webservice.exception
org.jboss.test.webservice.handlerflow
org.jboss.test.webservice.header
org.jboss.test.webservice.jbas897
org.jboss.test.webservice.jbws124
org.jboss.test.webservice.jbws128
org.jboss.test.webservice.jbws153
org.jboss.test.webservice.jbws64
org.jboss.test.webservice.jbws68
org.jboss.test.webservice.jbws70
org.jboss.test.webservice.jbws71
org.jboss.test.webservice.jbws79
org.jboss.test.webservice.jbws82
org.jboss.test.webservice.jbws83
org.jboss.test.webservice.jbws84
org.jboss.test.webservice.marshalltest
org.jboss.test.webservice.message
org.jboss.test.webservice.samples
org.jboss.test.webservice.samples2
org.jboss.test.webservice.samples2docclient
org.jboss.test.webservice.version
org.jboss.test.webservice.ws4eesimple
org.jboss.test.webservice.wsdlimport

[starksm64]

I would say all should be included. You can check with Thomas to see if some of these tests are excersizing features that really should not be considered as supportable across versions.

[clebert.suconic]

This is almost done, I'm just validating it before I commit the code.

I have only one consideration. Some of the tests are requiring jboss.jar.

As jboss.jar has all the classes contained into jboss-client.jar I'm not sure if I could add jboss.jar in the classpath for this tests as well.

I don't know, it looks like some of these classes contained into jboss.jar should be present in jboss-client.jar as well, what would represent some point to be fixed.

For example, org.jboss.test.client.test.MetaDataUnitTestCase requires org/jboss/metadata/ClientMetaData which is not represent into jboss-client.jar

Any thoughts Scott?


Clebert

[starksm64]

A test that requires jboss.jar either is not an external client test (and MetaDataUnitTestCase is not an external client test that should be run in this compatibility testsuite), or it indicates a problem with the definition of the client jars.

Every unit test in this compatibility test run should be able to run using only the jars in the client directory of the dist.

[clebert.suconic]

Ok!

Just as a FYI (Let me know if you foresee any problem):

I'm composing the classpath with:
- build.classes = compiled classes from the testsuite
- all the client libraries for a specific version
- library.classpath - as some of the classes are make usage of these libraries:
apache.avalon.classpath
apache.commons.classpath
apache.log4j.classpath
apache.xerces.classpath
apache.jaxme.classpath
wutka.dtdparser.classpath
dom4j.dom4j.classpath
oswego.concurrent.classpath
ibm.wsdl4j.classpath
jacorb.jacorb.classpath
jgroups.jgroups.classpath
junit.junit.classpath
junitejb.junitejb.classpath
javassist.classpath
sun.jaf.classpath
sun.javamail.classpath
sun.servlet.classpath
trove.classpath
gnu.regexp.classpath
apache.bcel.classpath
hsqldb.hsqldb.classpath
hibernate2.classpath
odmg.classpath
cglib.classpath

[starksm64]

Yes, that's what we want to do. This also helps test another compatibility issue with classes missing from client jars that has shown up from time to time due to the promiscuous use of the jboss module classes as the testuite classpath.

[clebert.suconic]

The code is already commited into Branch_4_0.

The matrix testsuite is part of the regular testsuite.

The only requirement for this is to have defined a variable -Dmatrix-versions.

I'm trying to fix cruise control testsuite for jboss-4.0 and configure the matrix execution against:


- jboss-4.0. (It's really broken anyway. I guess the only thing that passed was jbossMX. I will remove as soon as people are familiar with this)
- jboss-4.1-sp1
- we will always check the current version just to validate classpath issues

[clebert.suconic]

oops... I forgot to mention the wiki page for this:

http://www.jboss.org/wiki/Wiki.jsp?page=HowToExecuteMatrixTests