Hibernate Mail Lists (and testing)

[mikezzz]

I have added Hibernate Mail List, which is a replacement for the last of the EJB implementated modules in the Mail Server.

I have also added JUnits for the Mailboxes and Maillists. In order to simplify the writing of the tests I have created a JMXTestRunner MBean which serves as a proxy to run JUnits within a JBoss app server. I have moved the store tests over to using this mechanism also. If there are no major objections to this I would like to modify all of the tests to work this way.

To run the tests there is a JMXTestRunnerTask that will call the MBean with all of the class names that are present within a specified fileset. The JMXTestRunner has 2 methods. runTests() and runTestXML(). The first will run the test suite specified and will return a simple string summary. Useful for ad-hoc testing via the JMX-Console. The second will also execute a named test suite, but will return an XML structure similar to the one produces but the JUnit ant task, such that the JUnit report task can be reused to generate testing reports.

The run-test task will delete the hibernate data directory, starts the JBoss instance, runs the tests, stops the server. In order run correctly the CLASSPATH needs to set. The build.bat and build.sh scripts have been modified to cater for this. Also I have added log4j.jar to the CLASSPATH setting to fix the annoying (but harmless) exception printed out when running hibernate doclet.

Mike.

[acoliver]

I'm not yet satisfied with our testing effort or comprehensiveness. Its too painful ATM so something needs to change...

So I see different types of testing and we need to fill out to handle them as automatically as possible:

1. Unit testing -- does a particular tiny component or class work. Should be as independent of the appserver/etc as possible.

2. Component testing -- does a service work -- that probably should run in the server.

3. protocol testing -- run in the server but pass in scripts. We have stuff for this but I don't know that we've really completed the toolkit.

4. performance testing/functional testing - actually sending emails and making sure they are properly routed and how many can be routed per second/minute/hour. This is the most difficult to automate.

Do I want to start the appserver to test some minor class or feature? Probably not. I'm probably willing to start a subset (which is what the present stuff does) to satisfy dependencies. So do what you think is best but make sure I can just type "ant test" and that they don't take 5 years to run but I may need multiple targets depending on what i'm trying to accomplish. We probably need to change our dir structure a bit for this.

[mikezzz]

I think what would be sensible is make it possible (but not necessary) to run all of the tests* inside of the app server. Therefore it would be a subset of the tests that could be run outside of the app server.

I have been looking at the tests that start their own instance of the MBean Server. I'm a little wary of these as a number of reasons.

1) The are far to complex to write >50% of the code is used to handle the setUp and tearDown methods (Most of this code disappears when running in the app server).

2) They rely to heavily of stubbed classes. In my experience stubbed classes simply hid problems until later on in the development process. Also it creates a lot of extra code to maintain. Its much easier to test against real instances when running in the app server.

3) They tend to be brittle, with all extra mbean management code, stubbed classes and our own depends-service.xml, breakages occur often when our code or JBoss changes.

Am I willing to sacrifice an extra minute or 2 for the test suite to run, for simplicity, completeness and maintainablilty? Definitely. The JMXTestRunner does provide a mechanism to run test suite ad-hoc via the JMX Console (I generally run an app server consantly while developing).

However I will make sure that the subset of tests that fall into category 1 can be run outside the app server. This can be achieved by using a fileset in the build.xml which contains the subset of Category 1 tests. I think also some of our code could be refactored to reduced dependencies. E.g. should the Mail class depend on a installed an configured SMTP protocol?

*Performance test should generally be run seperately. The load testing client should also be on seperate box to more realistically similate load.

[acoliver]

Okay I agree, except for the performance testing. I think that performance requirements on individual units of code can also be useful. Meaning even if I'm running on my laptop, parsing mail headers ought not take more than a few MS right? The challenge is always the GC, but that can be handled via ant passing the right parms anyhow.

I'm about to start work on the izpack stuff. I'd really appreciate if you keep watch and provide feedback and assistance to fascillitate its use. I'd like to avoid being the only maintainer of the descriptors (there will be a later post regarding these in general).

[mikezzz]

Re: izPack, cool, will we be seeing it in M3?

Re: Performance testing, I agree. Micro benchmark for critical code section should be unit tests. I think the term I should of used was load/scalability testing.

[acoliver]

Yes, izPack is what I'm dinking with now. It isn't so cheesy!

You're right. They are different. humm...I need to fix my lab. I parts some thinks at various times.... I have two boxen for this at the moment "bobford" and "badmojo". We'll need better of course but there we can get some idea and do basic performance regression testing (I want to know when things we do make things slower). Shortly I need to get specmail up and running for some of this. Its actually going to be tough with all the componentes like DNS and the such (probably use /etc/hosts to defactor that).