4 Replies Latest reply on Nov 29, 2010 1:26 PM by jesper.pedersen

    TestSuite - multithreaded tests and stress test


      I'm convincing that we need to develop some tests in our test suite running concurrently to stress some multithread feature in our code. Of course needs of this kind of tests isn't immediately exposed by a coverage tool, but a deep analysis of our code base make it surely needed.

      In fact also old AS5/6 test suite has some of these tests, both to prove features/behaviors and to make some kind of stress tests.

      What I don't like of these tests is that is easily become ard to read and understand, putting a lot of threading code into tests it self.

      I'm instead thinking about to some external tool (maybe something to write, inside or outside ironjacamar code base, but not as part of testssuite for sure) to hide this threading infos, and trying to make threading pure declarative in tests.

      TestNG does something like that, but it doesn't seems perfect at a first look. I'll come back with some proposal very soon.




        • 1. Re: TestSuite - multithreaded tests and stress test

          For multithreaded tests it may also be worth investigating byteman as this can make the concurrent execution of the code deterministic.


          Of course it will still require an analysis of the codebase to determine the appropriate tests but the determinism could be worth the time spent.

          • 2. Re: TestSuite - multithreaded tests and stress test

            Hi Kevin,


            yep Byteman is one of the tools on my plate

            • 3. Re: TestSuite - multithreaded tests and stress test

              I've investigated more deep byteman, and I think it could be useful for some tests where (as Kev said) deterministic concurrent execution will be needed.

              I'm thinking about (for example) to connection pooling state after some particular situation happened to some thread using connections or statements from that pool. Errors situation or not clean use of the api by the user.

              This kind of advanced tests, verifying robustness of our implementation have to be well designed to keep them readable and to stress our codebase in a correct way. Moreover we have to decide also which kind of "not clean" use of our implementation and user's situations we would monitor and verify by the test suite and which we would just well document as not correct. Of course even testes ones have to be documented.


              That said, I think we need also some simpler way to write multithreaded tests to implement a stress test suite. In my ideal world we could have a lot (maybe not all) unit tests that could be launched in a stress tests suite with relatively high concurrence and tracking execution time (with a report as fine as possible to spot at least at macro level if and when a component is not scaling well). As already said a declarative (aka annotation) solution to define test concurrence would be more than nice.




              • 4. Re: TestSuite - multithreaded tests and stress test

                The multithreaded tests comes into play for the connection manager especially.


                Using ByteMan can prove valuable in many cases, however I think if we can do simple stuff like inject a CyclicBarrier or something like that the test will become more clear.


                Maybe there are various extensions - like http://groboutils.sourceforge.net/testing-junit/using_mtt.html - to JUnit we can use.


                The key requirement is to have the tests be deterministic, such that they will work on any machine. I don't want to see any Thread.sleep() conditions...