Speeding up the test suite

[timfox]

I have spent the last week or so trying to speed up the test suite, since it was taking far too long (up to 5 hours on Hudson).

I have made some good progress so far, now I have got all tests apart from the clustering tests, i.e. crash tests + in vm tests + remote tests + bridge tests, running in 30 minutes on my laptop with mysql.

I have created new targets "short-tests" and "short-report" which runs all the tests apart from clustering.

Before this combination would take around 2 hours, maybe 3 hours on Hudson. :)

Next stop is the clustering tests. I aim to have the entire test suite running in less than an hour.

This pretty critical for releases since we need to have 100% pass on Mysql, db2, postgresql, mssql and oracle for all GA releases.

[ovidiu.feodorov]

I noticed that running tests on Windows is not part of the release checklist anymore. Is this on purpose? People are still running messaging on windows ...

[timfox]

Hudson has some windows machines configured.

Once I've got everything running fast we'll be able to do full runs on Windows and Linux boxes.

[timfox]

But there is another issue with Hudson.

Since Hudson uses VMWare, and does not fully utilise the machines, it is:

a) Significantly slower than running the test suite on a dedicated machine

b) More seriously, because of the virtualization, any timing issues are brought to the front. Although it still runs a Java program "correctly" you will observer wild differences in thread scheduling and timing which is good for showing up race conditions but extremely bad for any tests that rely on timing to give correct results.
Unfortunately, for messaging, some tests do rely on timing. E.g. testing that the expiration of a message happens pretty much on time (also scheduled delivery etc).
Now, of course Java even without virtualization is not real time, and gc can happen anytime screwing the timing, but, in practice without virtualization if we add a small margin of error in our timing tests they give good results, this is not the case with virtualization in to play - the required margin of error because so large as to make the test pointless.
This may mean I will separate any timing sensitive tests into their own (small) testsuite and this will need to be run on a dedicated team.