11 Replies Latest reply on Dec 12, 2007 5:21 AM by jmesnil

    New test structure

    timfox

      Following on from the new JBM 2.0 test strategy, we need to come to concensus on a package structure for the tests, since currently we're all putting them in different places.

      I suggest:

      org.jboss.test. parent directory - all tests are under this
      org.jboss.test.unit unit tests - only use mocks and fakes of 3rd party components - fully deterministic
      org.jboss.test.unit.fakes any shared fake implementations that can be used by different tests
      org.jboss.test.integration integration tests - use real third party components - fully deterministic
      org.jboss.test.timing timing sensitive tests - e.g. scheduled delivery tests - allowed to contain Thread.sleeps(), theorertically may be non deterministic due to timing differences in thread scheduling.
      org.jboss.test.concurrent - concurrent tests - like unit tests but may use multiple threads to test a component with concurrently.
      org.jboss.test.stress - stress tests


      under each of the preceeding directories the actual tests would be structured according to the classname.

      e.g. org.jboss.test.unit.messaging.core.impl would contain unit tests for the core implementation classes

      Comments/suggestions?

        • 1. Re: New test structure
          jmesnil

           

          "timfox" wrote:
          Following on from the new JBM 2.0 test strategy, we need to come to concensus on a package structure for the tests, since currently we're all putting them in different places.

          I suggest:

          org.jboss.test. parent directory - all tests are under this


          Shouldn't we put our code under org.jboss.messaging.test to avoid (very unlikely) name collision with other JBoss projects?

          "timefox" wrote:

          e.g. org.jboss.test.unit.messaging.core.impl would contain unit tests for the core implementation classes

          Comments/suggestions?


          There is also the alternative to put the different types of test related to a single "component" under the same tree.

          e.g. for remoting; all tests related to remoting would be under org.jboss.messaging.test.remoting:
          - org.jboss.messaging.test.remoting.unit
          - org.jboss.messaging.test.remoting.integration
          - org.jboss.messaging.test.remoting.timing
          - ...

          This mimics the way the code is hierarchized (all remoting code is under org.jboss.messaging.remoting and its subpackages) and the tests related to a same component/"theme are closer to each other than spread all over the test suite.
          imho, it makes it easier to navigate the code and run tests from an IDE

          And this won't change the way the tests are run automatically (org.jboss.messaging.test.**.unit instead of org.jboss.messaging.test.unit.**)

          What do you think?



          • 2. Re: New test structure
            timfox

             

            "jmesnil" wrote:

            There is also the alternative to put the different types of test related to a single "component" under the same tree.

            e.g. for remoting; all tests related to remoting would be under org.jboss.messaging.test.remoting:
            - org.jboss.messaging.test.remoting.unit
            - org.jboss.messaging.test.remoting.integration
            - org.jboss.messaging.test.remoting.timing
            - ...



            So, "remoting" in your above example should be "core.remoting" surely? So we mirror the exact package names?

            • 3. Re: New test structure
              timfox

              If we were to put them all in the same hierarchy surely they would be

              <package name>/test/unit
              <package name>/test/timing
              <package name>/test/concurrent

              e.g.:
              org.jboss.messaging.core.impl.remoting.test.unit
              org.jboss.messaging.core.impl.remoting.test.timing
              etc

              • 4. Re: New test structure
                jmesnil

                 

                "timfox" wrote:
                "jmesnil" wrote:

                There is also the alternative to put the different types of test related to a single "component" under the same tree.

                e.g. for remoting; all tests related to remoting would be under org.jboss.messaging.test.remoting:
                - org.jboss.messaging.test.remoting.unit
                - org.jboss.messaging.test.remoting.integration
                - org.jboss.messaging.test.remoting.timing
                - ...



                So, "remoting" in your above example should be "core.remoting" surely? So we mirror the exact package names?


                yes. I started doing that when I developed Eclipse applications using their naming conventions: [url=http://wiki.eclipse.org/index.php/Naming_Conventions]

                The idea is that a developer most of the time is only interested by a specific piece of code and want to focus on it.
                E.g. if I had to add a feature to the new persistence manager and I want to run the tests to make sure I do not break it, I find it simpler to run all tests under org.jboss.messaging.core.pm.test (or something else) than to have to navigate to all the subfolders of unit, fake, integration, etc. to check which ones have tests related to the persistence manager.

                Of course, Hudson will still be configured to run all unit tests together, as well as integration tests et al.

                • 5. Re: New test structure
                  timfox

                  Yes, I think your suggestion is sound for most unit tests / concurrent tests / timing tests.

                  However some tests don't fit into this since they're not specific to a particular package, e.g. stress tests or some integration tests which may use many classes, so there is no natural home w.r.t package name.

                  In those cases we could just use a package like:

                  org.jboss.messaging.test.stress.blah

                  org.jboss.messaging.test.integration.blah

                  wdyt?

                  • 6. Re: New test structure
                    jmesnil

                     

                    "timfox" wrote:
                    Yes, I think your suggestion is sound for most unit tests / concurrent tests / timing tests.

                    However some tests don't fit into this since they're not specific to a particular package, e.g. stress tests or some integration tests which may use many classes, so there is no natural home w.r.t package name.

                    In those cases we could just use a package like:

                    org.jboss.messaging.test.stress.blah

                    org.jboss.messaging.test.integration.blah

                    wdyt?


                    +1

                    • 7. Re: New test structure
                      timfox

                      Ok lets do that then :)

                      • 8. Re: New test structure
                        timfox

                        I have added four new targets to the tests build script corresponding to the new types of tests identified in the testing strategy page:

                        unit-tests
                        concurrent-tests
                        timing-tests
                        integration-tests

                        Also the short-tests target has been extended to call these targets too.

                        As you do new work please ensure your code is appropriately tested and the tests placed in the correct directories so these targets pick them up.

                        (Currently the short-tests target is all passing apart from one test which was failing after Andy's last commit:

                        ConnectionFactoryTest::testQueueConnectionFactory - which reproducably fails for me.

                        If we could get this passing we would be on 100% :) )

                        Any commits from now, we should all ensure we run the short-tests target and get 100% before committing.

                        • 9. Re: New test structure
                          clebert.suconic

                          I think of the "integration-tests" we have more as qa-tests.

                          I believe integration-tests should be something sitting on the JBAS code (not supposed to take 12 hours running, just to validate the integration itself).

                          If we start using spi-projects as everybody else is doing, this could also confuse witth testing the SPI.

                          • 10. Re: New test structure
                            timfox

                            integration-tests in this context doesn't mean AS style integration tests - just ones that use Real implementation as opposed to fake.

                            Take a look in the trunk for examples.

                            • 11. Re: New test structure
                              jmesnil

                               

                              "jmesnil" wrote:

                              This mimics the way the code is hierarchized (all remoting code is under org.jboss.messaging.remoting and its subpackages) and the tests related to a same component/"theme are closer to each other than spread all over the test suite.
                              imho, it makes it easier to navigate the code and run tests from an IDE

                              And this won't change the way the tests are run automatically (org.jboss.messaging.test.**.unit instead of org.jboss.messaging.test.unit.**)



                              To be more precise, I suggest to keep the tests under test/src/java source folder.
                              Related tests are closer to each other (sharing a common super package) but remain separate from the code they test which is under src/main/java source folder.