14 Replies Latest reply on Mar 13, 2009 6:30 AM by ropalka

    JBossWS Test Approach

    ropalka

      This thread is related to JBWS-2530.

        • 1. Re: JBossWS Test Approach
          ropalka

          I'd like to start discussion about above JIRA issue.
          I don't like our current testing approach because:

          it's error prone approach
          we don't test against AS distributions but against local maven repository
          it wastes our development time


          I'll explain each point in next posts.

          • 2. Re: JBossWS Test Approach
            ropalka

             

            "richard.opalka@jboss.com" wrote:
            it's error prone approach

            We're using maven surefire plugin to invoke our junit integration tests.
            The problem is our tests are not just java component tests but they test whole
            JBossWS integration to JBossAS. The concrete problem is we're using maven
            dependencies mechanism to configure our tests classpath.
            Unfortunately this introduces many defects to our tests:


            We're not testing AS using AS provided libraries in distribution
            We're using libraries from local maven repository where

            many of these libraries has wrong version (because of outdated dependency exclusions)
            we don't have control over which version of particular component jar will
            be used for testing when there are two libraries of the
            same component with different version.
            The only way to achieve it is to use maven
            dependency exclusions (a lot of work without any evident added value).
            These exclusions should be updated on daily basis when testing
            AS moving branches/trunks.
            we're using out of date libraries for moving AS versions - many updates
            of the required dependencies are not released to the maven repository
            To solve this AS maven artifacts (SNAPSHOTS) should be uploaded to maven repo on daily basis.




            • 3. Re: JBossWS Test Approach
              ropalka

               

              "richard.opalka@jboss.com" wrote:
              we don't test against AS distributions but against local maven repository

              This is fundamental problem for me.

              There are two types of AS we're testing against to consider here:

              released AS GAs - current testing approach works properly for it
              moving AS branches/trunks - current testing approach is highly error prone for it


              And because our testing approach is identical for released AS GAs and moving AS branches/trunks we should use AS distributions jars only to prevent this problem.
              In that case we'd be always up2date with jars in the distribution and there'd be no overhead to exclude dependencies on regular basis, upload maven artifacts, ... etc.


              • 4. Re: JBossWS Test Approach
                ropalka

                 

                "richard.opalka@jboss.com" wrote:
                it wastes our development time

                Because:


                we have to review and fix dependencies (usually exclusions) each time some tests start failing because of classpath inconsistency (applies to moving AS version only)
                we have to upload maven artifacts to the repo if there's no other option available (AS upload to maven repo takes ~1 hour and it fails sometimes lately).


                • 5. Re: JBossWS Test Approach
                  ropalka

                  Please, let me know your opinion on this problem.
                  I'd like to start how to solve it discussion soon ;)

                  • 6. Re: JBossWS Test Approach
                    asoldano

                    My point of view here is that we're evaluating this still a bit early. Let me explain, the problem here is that the AS is not completely mavenized yet. In particular the AS testsuite is not run against maven artifacts yet and that's IMHO one of the reasons why almost nobody besides us cares about updoading snapshots of the org.jboss.jbossas artifacts to the repo. Once the process will be completed (don't know when, but the work is being done even while we discuss this here now), I think we will be pretty sure we're testing against the right artifacts.

                    This said, the specific failures that made us to stop and think about our testing approach were caused by some mistakes we actually did in the webservice module pom.xml in the AS. IOW it didn't have the right dependencies which were instead specified in the container integration pom.xml files. Of course this caused the trunk and 5_x branch only to fail, but I fixed that. The whole "exclusion game" in the stacks' testsuite pom.xml is not required anymore, there're few dependencies still specified there and they're quite straightforward.

                    Finally, considering we're using maven for the project, I think it's important to run tests using surefire. IOW my point of view is that either we use this tool (maven) completely or we don't use it (which currently is not an option given the direction taken 1 year ago by the whole AS development).
                    Basing the tests on maven dependencies was the most natural solution when the mavenization took place, what would you suggest instead of that? I don't like that much the idea of specifying the surefire additionalClasspath with a list of jar from the curent AS... any other idea? After all, we also get advantages from using maven dependencies to build test classpath, otherwise we need keep track of jar changes (names, position), addition of jars (while we simply get the new dependency for free when somebody updates the dependency in an artifact we include), etc.

                    • 7. Re: JBossWS Test Approach
                      asoldano

                      A side note to make this clear to anybody not familiar with the jbossws testsuite. The classpath being dicussed here is of course the one used on client side in an integration test. Most of those tests deploy an endpoint on the current jboss instance; the endpoint thus actually uses the right classpath. It's just a matter of client classpath.

                      • 8. Re: JBossWS Test Approach
                        ropalka

                         

                        "alessio.soldano@jboss.com" wrote:

                        ... we're evaluating this still a bit early ...
                        I don't think so. We were talking about this problem a year ago for the first time. I'm afraid we'll be talking about it next years :(
                        "alessio.soldano@jboss.com" wrote:

                        ... AS is not completely mavenized yet ...
                        and won't be next few years ;) Note that AS mavenization process runs already more than 1 year and it's still at its beginnings.
                        "alessio.soldano@jboss.com" wrote:

                        ...Once the process will be completed I think we will be pretty sure we're testing against the right artifacts.
                        Almost true, but not absolute ;)

                        "alessio.soldano@jboss.com" wrote:

                        This said, the specific failures that made us to stop and think about our testing approach were caused by some mistakes we actually did in the webservice module pom.xml in the AS. IOW it didn't have the right dependencies which were instead specified in the container integration pom.xml files. Of course this caused the trunk and 5_x branch only to fail, but I fixed that. The whole "exclusion game" in the stacks' testsuite pom.xml is not required anymore, there're few dependencies still specified there and they're quite straightforward.
                        Thanks for the hotfix Alessio, but it didn't solve the problem I'm discussing here. It just hotfixed the result of the problem. I'd like to fix the reason of all these problems instead. I'll explain this argument in next post following this one.

                        "alessio.soldano@jboss.com" wrote:

                        Finally, considering we're using maven for the project, I think it's important to run tests using surefire. IOW my point of view is that either we use this tool (maven) completely or we don't use it (which currently is not an option given the direction taken 1 year ago by the whole AS development).
                        Basing the tests on maven dependencies was the most natural solution when the mavenization took place, what would you suggest instead of that? I don't like that much the idea of specifying the surefire additionalClasspath with a list of jar from the curent AS... any other idea?


                        • 9. Re: JBossWS Test Approach
                          ropalka

                          JBossWS supports AS 5.0.0, 5.0.1, Branch_5_x and trunk (version numbers will change in the future).
                          Each AS has it's dependencies specifed using maven.
                          Each of these dependencies specified in AS components matrix
                          has its own dependencies on another subprojects.

                          To be absolutely clear the main problem I'm talking here is:

                          Because we're supporting multiple AS versions we can't
                          construct proper client classpath using maven dependencies..

                          Example:

                          JBossWS 3.0.5
                          AS 5.0.0

                          Which jaxb-impl version will be used in testing if we don't exclude it?

                          (dependency:tree output from testsuite module in jboss500 configuration)
                          [INFO] org.jboss.ws.native:jbossws-native-testsuite:pom:3.0.5.GA
                          [INFO] +- org.jboss.ws.native:jbossws-native-client:jar:3.0.5.GA:compile
                          [INFO] | +- org.jboss.ws.native:jbossws-native-core:jar:3.0.5.GA:compile
                          [INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.1.6:compile
                          ...
                          [INFO] +- org.jboss.ws:jbossws-jboss500:jar:3.0.5.GA:compile
                          [INFO] | +- org.jboss:jbossxb:jar:2.0.0.GA:compile
                          [INFO] | | +- sun-jaxb:jaxb-api:jar:2.1.4:compile
                          ...

                          We can't exclude jaxb-impl neither from jbossws-native-client (note doesn't matter the different group.id)
                          neither jaxb-impl from jbossws-jboss500 because it's compile dependency (it won't compile without it).
                          We could exclude jaxb-impl either from jbossws-native-client or from jbossws-jboss500 in testsuite.pom xml.
                          But doing so for each conflicting component (there are many such) would be a real maintainability nightmare (especially for moving AS versions even they would be fully mavenized as mentioned in above post)

                          • 10. Re: JBossWS Test Approach
                            ropalka

                             

                            "alessio.soldano@jboss.com" wrote:
                            Basing the tests on maven dependencies was the most natural solution when the mavenization took place,

                            Well I have different point of view on it:( It don't think it was the most natural solution to mavenize our test suite.

                            "alessio.soldano@jboss.com" wrote:
                            what would you suggest instead of that? I don't like that much the idea of specifying the surefire additionalClasspath with a list of jar from the curent AS... any other idea?

                            I'm thinking exactly about that ;) But I have to googlize this problem much more. I don't believe we're the first one facing this problem.

                            "alessio.soldano@jboss.com" wrote:

                            After all, we also get advantages from using maven dependencies to build test classpath, otherwise we need keep track of jar changes (names, position), addition of jars (while we simply get the new dependency for free when somebody updates the dependency in an artifact we include), etc.

                            Wrong point of view. I'd maintain all this staff like we did it before JBossWS mavenization and like we do it now (see our binary distribution) and be sure we're using proper jars directly from AS than using current approach which is broken.

                            Current approach doesn't save anything. It just makes me sleep really very bad :(
                            I have a nightmare regularly at nights where I'm debugging for hours/days something and finally solving the problem with conclusion we're using wrong client classpath and hotfixing it with one maven dependencies exclusion in our testsuite pom.xml


                            • 11. Re: JBossWS Test Approach
                              asoldano

                               

                              "richard.opalka@jboss.com" wrote:
                              <b>Because we're supporting multiple AS versions we can't
                              construct proper client classpath using maven dependencies.</b>.


                              Given that I see the issue that drove to this forum thread, I don't get why you think we can't construct proper client classpath using m2 dependencies because we're using multiple AS versions.


                              Example:

                              JBossWS 3.0.5
                              AS 5.0.0

                              Which jaxb-impl version will be used in testing if we don't exclude it?

                              (dependency:tree output from testsuite module in jboss500 configuration)
                              [INFO] org.jboss.ws.native:jbossws-native-testsuite:pom:3.0.5.GA
                              [INFO] +- org.jboss.ws.native:jbossws-native-client:jar:3.0.5.GA:compile
                              [INFO] | +- org.jboss.ws.native:jbossws-native-core:jar:3.0.5.GA:compile
                              [INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.1.6:compile
                              ...
                              [INFO] +- org.jboss.ws:jbossws-jboss500:jar:3.0.5.GA:compile
                              [INFO] | +- org.jboss:jbossxb:jar:2.0.0.GA:compile
                              [INFO] | | +- sun-jaxb:jaxb-api:jar:2.1.4:compile
                              ...

                              The artifact that's closest to the root will be used (with the distance being defined by the number of nested dependencies you need to go through to reach the artifact).


                              We can't exclude jaxb-impl neither from <b>jbossws-native-client</b> (note doesn't matter the different group.id)
                              neither jaxb-impl from <b>jbossws-jboss500</b> because it's compile dependency (it won't compile without it).
                              We could exclude jaxb-impl either from jbossws-native-client or from jbossws-jboss500 in testsuite.pom xml.


                              Exclusions are required only when the closest version of a given artifact is not the one you want to use, but this should not happen if the dependency hierarchy is well done (which did not apply in our webservice module since some days ago).

                              This is besides the original problem, just to be clear on maven.

                              • 12. Re: JBossWS Test Approach
                                asoldano

                                 

                                "richard.opalka@jboss.com" wrote:
                                "alessio.soldano@jboss.com" wrote:
                                what would you suggest instead of that? I don't like that much the idea of specifying the surefire additionalClasspath with a list of jar from the curent AS... any other idea?

                                I'm thinking exactly about that ;) But I have to googlize this problem much more. I don't believe we're the first one facing this problem.

                                OK, let's see if there's a better solution :-)

                                "alessio.soldano@jboss.com" wrote:

                                After all, we also get advantages from using maven dependencies to build test classpath, otherwise we need keep track of jar changes (names, position), addition of jars (while we simply get the new dependency for free when somebody updates the dependency in an artifact we include), etc.

                                Wrong point of view. I'd maintain all this staff like we did it before JBossWS mavenization and like we do it now (see our binary distribution) and be sure we're using proper jars directly from AS than using current approach which is broken.

                                Current approach doesn't save anything. It just makes me sleep really very bad :(
                                I have a nightmare regularly at nights where I'm debugging for hours/days something and finally solving the problem with conclusion we're using wrong client classpath and hotfixing it with one maven dependencies exclusion in our testsuite pom.xml


                                Don't get me wrong, I'm not saying there's not a problem and we have an easy life with this way of working ;-) Just trying to evaluate the situation, let's think about alternatives (at least given the current state of the AS).

                                • 13. Re: JBossWS Test Approach
                                  ropalka

                                   

                                  "alessio.soldano@jboss.com" wrote:

                                  Don't get me wrong, I'm not saying there's not a problem and we have an easy life with this way of working ;-) Just trying to evaluate the situation, let's think about alternatives (at least given the current state of the AS).

                                  Don't worry, the nightmare part was a joke ;)

                                  • 14. Re: JBossWS Test Approach
                                    ropalka

                                    I evaluated both
                                    1) current maven dependencies approach
                                    2) and additionalClassPath approach

                                    My conclusion:

                                    Current 1) approach is better one according to the fact our tests framework is fully mavenized. It's because dependencies classpath specified in modules/testsuite/pom xml is also reused in modules/testsuite/*-tests/pom.xml submodules for invoking JBossWS tools. If we would want to remove current testing approach it would mean to antize it again (step back).