14 Replies Latest reply on Sep 27, 2010 4:43 AM by wolfc

    AS7 Integration TestSuite

    alrubinger

      I see Brian's started:

       

      https://jira.jboss.org/browse/JBAS-8223

       

      Was thinking the other day that perhaps for AS7 it makes more sense to move the integration testsuite outside of the AS release cycle.  In this way we can include technologies which depend upon AS releases for the testing (breaking the cyclic dependency); we'd for instance be able to use Arquillian.

       

      Also it'd speed up checkout/builds of the main tree.

       

      Thoughts on this approach?

       

      S,

      ALR

        • 1. Re: AS7 Integration TestSuite
          pgier

          How about a minimal integration testsuite (smoke-tests) as a module of the AS 7 build, and the complete testsuite as a separate project?  That way you we could either run smoke tests by default, or just set a property or profile to run them after the main build.

          • 2. Re: AS7 Integration TestSuite
            jason.greene

            We definitely need the main testsuite to be part of AS. I just don't want someone to have to download a separate project just to run tests. Things like stress tests and benchmarks will definitely need to be segregated. Also another goal is to have the testsuite that's part of AS actually complete in a reasonable timeframe.

             

            As to Arquillian, I think we just have to figure out were it fits in. Our testsuite needs to cover the standard code execution paths (e.g. deploying through non-embedded mechanisms) in addition to the embedded ones.

            • 3. Re: AS7 Integration TestSuite
              alrubinger

              Arquillian != Embedded.  We can use both the Embedded JBossAS Container, or the Remote one.  In fact the remote JBossAS container is likely to be a more accurate representation of what we'd like to test.

               

              As far as downloading a separate project, sure, that's a bit less convenient at first.  But consider then that launching tests would be able to be done from the IDE.  Using ShrinkWrap as the deployment mechanism we'd get IDE incremental compilation, meaning we wouldn't have to perform a build inbetween runs.  No more editing build.xml to declare the JAR/WAR/EAR structure.  And hooking in a debugger would be a one-click operation.  It's much more friendly, despite the extra checkout location.

               

              Finally, the AS testsuite is a complex arrangement of suites targeting different profiles (ie. server configurations) and by aligning our own testsuite to be Arquillian-based, we'd be pushing requirements to the ARQ project to ensure it covers all cases for our users as well.

               

              S,

              ALR

              • 4. Re: AS7 Integration TestSuite
                alrubinger

                Paul Gier wrote:

                 

                How about a minimal integration testsuite (smoke-tests) as a module of the AS 7 build, and the complete testsuite as a separate project?  That way you we could either run smoke tests by default, or just set a property or profile to run them after the main build.

                 

                Yes, I agree here.  I have something of this nature already in place with the "embedded" module of AS/trunk.  We could port the smoke tests to be part of the build, certifying it before it's installed into any repo.  The full integration testsuite, IMO, should live elsewhere and take advantage of ARQ.

                 

                S,

                ALR

                • 5. Re: AS7 Integration TestSuite
                  alrubinger

                  Another point:

                   

                  We're pushing Arquillian as *the* approved mechanism to test EE.  What does it say if we don't implement this on our own, and drive project features to match our own requirements?


                  S,

                  ALR

                  • 6. Re: AS7 Integration TestSuite
                    brian.stansberry

                    I'm not understanding the need for an external testsuite project with a separate release cycle. The AS that's going to be tested is essentially going to be a snapshot. It will have to comply with whatever contract exists between it and Arquillian, but as long as it does Arquillian should be able to work with it.

                     

                    If we decide to change the contract in an incompatible way, the testsuite won't work until a new release of Arquillian is available that works with the new contract. Putting the testsuite in a separate project doesn't change that.

                     

                    Seems more like whatever AS element expresses the contract between the AS and Arquillian needs a separate release cycle.

                    • 7. Re: AS7 Integration TestSuite
                      brian.stansberry

                      OT: turns out there already was a skeleton testsuite module in the AS  repo. It's just not hooked into the top level pom so it didn't appear in  eclipse. It has child modules for integration, benchmark and stress. None do anything at the moment.

                       

                      Should we formalize the smoke-test notion? That is, actually create a smoke-tests child module at the same level as integration/benchmark/stress (ignoring for now the question of whether integration/benchmark/stress belong in an external project). The concept being that a normal build runs the tests in each non-testsuite module plus the testsuite's smoke-tests module. A more complete testsuite run would run integration (and perhaps benchmark and stress.)

                       

                      Looping back to the separate testsuite project question, would any of the tests we consider as requirements for a successful build, i.e. smoke-tests, use AQ? If yes, there's no benefit to an external project for the non-smoke tests; you'd have to solve the circular-dependency issue anyway to run the smoke tests.

                      • 8. Re: AS7 Integration TestSuite
                        brian.stansberry

                        Continuing my monologue.

                         

                        The dependency issues need to be sorted out, but I like the idea of using Arquillian as our testing framework instead of some AS7-ish equivalent to jboss-test + Ant.

                        • 9. Re: AS7 Integration TestSuite
                          alrubinger

                          Brian Stansberry wrote:

                           

                          Seems more like whatever AS element expresses the contract between the AS and Arquillian needs a separate release cycle.

                          This is an option as well, which will amount to the Container extensions of Arquillian getting extracted out from Arquillian proper into their own release cycle.

                           

                          We've talked about this, and last consensus was that this can happen once the Deployment SPIs become stable.  I'll raise the issue with those guys to see how we feel about expediting that process.

                           

                          S,

                          ALR

                          • 10. Re: AS7 Integration TestSuite
                            alrubinger

                            Brian Stansberry wrote:

                             

                            Looping back to the separate testsuite project question, would any of the tests we consider as requirements for a successful build, i.e. smoke-tests, use AQ? If yes, there's no benefit to an external project for the non-smoke tests; you'd have to solve the circular-dependency issue anyway to run the smoke tests.

                            I don't think it's a requirement for smoke-tests to use ARQ, though if we solve the circular dependency issue at the Arquillian level instead of AS, they certainly can and IMO should .  I can't emphasize enough how much simpler ARQ made writing/debugging my EJB Book Examples. 


                            S,

                            ALR

                            • 11. Re: AS7 Integration TestSuite
                              brian.stansberry

                              No one objected for formalizing smoke-tests and 1 person explicitly agreed, so I did it.

                               

                              AS 7 repo now includes a testsuite module, which is just a pom with 4 child modules:

                               

                              smoke

                              integration

                              benchmark

                              stress

                               

                              The pom's are configured such that only the tests in "smoke" will execute as part of a normal build. All the test modules will compile in a normal build.

                               

                              The integration, benchmark and stress testsuites can be made to execute as part of the build via maven profiles, e.g.

                               

                              mvn -Pintegration-tests install
                              

                               

                              The profile names are integration-tests, benchmark-tests, smoke-tests and all-tests.

                              • 12. Re: AS7 Integration TestSuite
                                brian.stansberry

                                I think I should rename the tests-integration module and integration-tests profile to tests-standard so relatively long running tests can go there, even if they are not true integration tests. Then individual module testsuites and tests-smoke should be reserved for stuff that runs fast. Now some of the module testsuites are running some fairly slow tests for stuff that probably doesn't need to be run every build.

                                • 13. Re: AS7 Integration TestSuite
                                  wolfc

                                  We need to have a combination of "Big Bang" and "Bottum Up" integration tests. What I don't want to see is everybody committing on a Friday and we see literally a big bang.

                                   

                                  Separated testsuites can be used for both use cases. Some of these testsuites should also target a spec, not a runtime. We will have different runtimes conforming to multiple specs.

                                  • 14. Re: AS7 Integration TestSuite
                                    wolfc

                                    Also we need to have guards that fixes are applied to multiple runtimes, for example both upstream and downstream AS.