For multithreaded tests it may also be worth investigating byteman as this can make the concurrent execution of the code deterministic.
Of course it will still require an analysis of the codebase to determine the appropriate tests but the determinism could be worth the time spent.
yep Byteman is one of the tools on my plate
I've investigated more deep byteman, and I think it could be useful for some tests where (as Kev said) deterministic concurrent execution will be needed.
I'm thinking about (for example) to connection pooling state after some particular situation happened to some thread using connections or statements from that pool. Errors situation or not clean use of the api by the user.
This kind of advanced tests, verifying robustness of our implementation have to be well designed to keep them readable and to stress our codebase in a correct way. Moreover we have to decide also which kind of "not clean" use of our implementation and user's situations we would monitor and verify by the test suite and which we would just well document as not correct. Of course even testes ones have to be documented.
That said, I think we need also some simpler way to write multithreaded tests to implement a stress test suite. In my ideal world we could have a lot (maybe not all) unit tests that could be launched in a stress tests suite with relatively high concurrence and tracking execution time (with a report as fine as possible to spot at least at macro level if and when a component is not scaling well). As already said a declarative (aka annotation) solution to define test concurrence would be more than nice.
The multithreaded tests comes into play for the connection manager especially.
Using ByteMan can prove valuable in many cases, however I think if we can do simple stuff like inject a CyclicBarrier or something like that the test will become more clear.
Maybe there are various extensions - like http://groboutils.sourceforge.net/testing-junit/using_mtt.html - to JUnit we can use.
The key requirement is to have the tests be deterministic, such that they will work on any machine. I don't want to see any Thread.sleep() conditions...