I'm not yet satisfied with our testing effort or comprehensiveness. Its too painful ATM so something needs to change...
So I see different types of testing and we need to fill out to handle them as automatically as possible:
1. Unit testing -- does a particular tiny component or class work. Should be as independent of the appserver/etc as possible.
2. Component testing -- does a service work -- that probably should run in the server.
3. protocol testing -- run in the server but pass in scripts. We have stuff for this but I don't know that we've really completed the toolkit.
4. performance testing/functional testing - actually sending emails and making sure they are properly routed and how many can be routed per second/minute/hour. This is the most difficult to automate.
Do I want to start the appserver to test some minor class or feature? Probably not. I'm probably willing to start a subset (which is what the present stuff does) to satisfy dependencies. So do what you think is best but make sure I can just type "ant test" and that they don't take 5 years to run but I may need multiple targets depending on what i'm trying to accomplish. We probably need to change our dir structure a bit for this.
I think what would be sensible is make it possible (but not necessary) to run all of the tests* inside of the app server. Therefore it would be a subset of the tests that could be run outside of the app server.
I have been looking at the tests that start their own instance of the MBean Server. I'm a little wary of these as a number of reasons.
1) The are far to complex to write >50% of the code is used to handle the setUp and tearDown methods (Most of this code disappears when running in the app server).
2) They rely to heavily of stubbed classes. In my experience stubbed classes simply hid problems until later on in the development process. Also it creates a lot of extra code to maintain. Its much easier to test against real instances when running in the app server.
3) They tend to be brittle, with all extra mbean management code, stubbed classes and our own depends-service.xml, breakages occur often when our code or JBoss changes.
Am I willing to sacrifice an extra minute or 2 for the test suite to run, for simplicity, completeness and maintainablilty? Definitely. The JMXTestRunner does provide a mechanism to run test suite ad-hoc via the JMX Console (I generally run an app server consantly while developing).
However I will make sure that the subset of tests that fall into category 1 can be run outside the app server. This can be achieved by using a fileset in the build.xml which contains the subset of Category 1 tests. I think also some of our code could be refactored to reduced dependencies. E.g. should the Mail class depend on a installed an configured SMTP protocol?
*Performance test should generally be run seperately. The load testing client should also be on seperate box to more realistically similate load.
Okay I agree, except for the performance testing. I think that performance requirements on individual units of code can also be useful. Meaning even if I'm running on my laptop, parsing mail headers ought not take more than a few MS right? The challenge is always the GC, but that can be handled via ant passing the right parms anyhow.
I'm about to start work on the izpack stuff. I'd really appreciate if you keep watch and provide feedback and assistance to fascillitate its use. I'd like to avoid being the only maintainer of the descriptors (there will be a later post regarding these in general).
Re: izPack, cool, will we be seeing it in M3?
Re: Performance testing, I agree. Micro benchmark for critical code section should be unit tests. I think the term I should of used was load/scalability testing.
Yes, izPack is what I'm dinking with now. It isn't so cheesy!
You're right. They are different. humm...I need to fix my lab. I parts some thinks at various times.... I have two boxen for this at the moment "bobford" and "badmojo". We'll need better of course but there we can get some idea and do basic performance regression testing (I want to know when things we do make things slower). Shortly I need to get specmail up and running for some of this. Its actually going to be tough with all the componentes like DNS and the such (probably use /etc/hosts to defactor that).