0 Replies Latest reply on Mar 6, 2012 2:29 PM by rcd

    Arquillian/TestNG bug?

    rcd

      I have an EJB project that's part of an EAR. The EAR and all parts of it are built with Maven. I've been trying to set up Jenkins to do continuous integration testing using Arquillian and TestNG, and I've hit a weird problem. I'm not sure if this is a bug or if I'm just not configuring something correctly, but I'm leaning towards the former after tinkering with it for a while.

       

      Basically, I can't declare all of my tests in the same TestNG <test> element, like this:

       

      {code:xml}

      <suite name="test">

      <test name="all">

        <packages>

         <package name="packageA"/>

         <package name="packageB"/>

        </packages>

      </test>

      </suite>


      {code}

       

       

      If I do, then one of two things happens, depending on how group-by-instances is set.

      • If group-by-instances is left undefined, it defaults to false, and all the tests try to run at the same time. This is a problem because different test classes load the database with different datasets. I'm using the method I described here to load them. Specifically, what happens is that for each package, each class in the package gets its @Deployment and setup() methods run, then each class has the rest of its test methods run, then Arquillian undeploys the test EAR. That is, ALL the setup() methods in a package get invoked, then all the rest of the test methods get invoked. I haven't told TestNG to run anything in parallel, and the TestNG docs say parallel defaults to off if you don't specify it. I tried specifying it and explicitly saying to not run tests in parallel anyway; that changed nothing. Even Arquillian seems confused about what's going on, because before each setup() method, it executes the @Deployment method, which produces the same deployment in all cases, then tries to undeploy it after it finishes executing tests for a class. This means the same deployment gets deployed on top of itself repeatedly... and all tests and undeploys after the first class fail because the deployment is already gone.
      • If group-by-instances is set to true, this has the curious effect of causing only a single test class to be executed, regardless of how many are in the packages.

       

      If I define each class inside its own <test> element, however, everything works as expected: for each class, the @Deployment is called, then the setup() method is run, followed by all the actual test methods, followed by undeploying.

       

      {code:xml}

      <suite name="test">

      <test name="test1">

        <classes>

         <class name="classA"/>

        </classes>

      </test>

       

      <test name="test2">

        <classes>

         <class name="classB"/>

        </classes>

      </test>

      </suite>


      {code}

       

      This is annoying, and I can deal with it, but I'd prefer to not have to do this. So is there a bug here or am I just not configuring something properly? Here's the relevant section of the POM, for reference:

       

      {code:xml}

         <plugin>

          <groupId>org.apache.maven.plugins</groupId>

          <artifactId>maven-surefire-plugin</artifactId>

          <version>2.12</version>

          <configuration>

           <testFailureIgnore>true</testFailureIgnore>

          

           <systemPropertyVariables>

            <test.jbossHome>C:\jboss-as-7.1.0.Final</test.jbossHome>

           </systemPropertyVariables>

          

           <properties>

            <listeners>org.testng.reporters.XMLReporter</listeners>

           </properties>

          

           <suiteXmlFiles>

            <suiteXmlFile>src/test/resources/testng-all.xml</suiteXmlFile>

           </suiteXmlFiles>

          </configuration>

         </plugin>


      {code}

       

      Edited to fix formatting