10 Replies Latest reply on Jul 11, 2014 9:41 AM by Bartosz Majsak

    Arquillian tests always passing

    Todd Morrison Newbie

      Diving into Arquillian testing finally.

       

       

      Unfortunately, I am getting false positives across the board.

       

       

       

       

        <dependency>
         <groupId>org.testng</groupId>
         <artifactId>testng</artifactId>
         <version>5.14</version>
         <scope>test</scope>
        </dependency>
      
      
      
      
        <dependency>
         <groupId>org.jboss.arquillian</groupId>
         <artifactId>arquillian-testng</artifactId>
         <version>${arquillian.version}</version>
         <scope>test</scope>
        </dependency>
        <dependency>
      
      
        <dependencies>
          <dependency>
           <groupId>org.jboss.arquillian.container</groupId>
           <artifactId>arquillian-jbossas-remote-6</artifactId>
           <version>${arquillian.version}</version>
           <scope>test</scope>
          </dependency>
      
      
      

       

       

      With Arg version

       

       

        <arquillian.version>1.0.0.Alpha5</arquillian.version>
      
      
      

       

       

      A Simple Test-case that inherits from Arquillian test base that does an assertFalse(true) always passes.

       

       

       

       

      Is this a known issue?

        • 1. Arquillian tests always passing
          Aslak Knutsen Master

          Can you see the test actually execute?

           

          e.g. System.out.println() before the Assert. and check the server log.

           

          And your using the Assert object right, not the assert keyword? If the keyword, you have to activate assertion on the container jvm

          • 2. Re: Arquillian tests always passing
            Todd Morrison Newbie

            Aslak, thanks for the advice.

             

            Calling

             

            assertTrue(false);
            

             

            on TestNG 5.14 results in 'success', no logging output and no break-points. (this is with a solid mvn clean on the target too, very creepy).

             

             

             

             

            Updating to 5.14.6 and modifying the code to

             

            Assert.assertTrue(false);
            

             

            resulted in correct failure and logging working just fine.

             

             

            This is not sane, so I would like to conclude that 'My Environment was jacked'.

             

             

            I am working just fine now. Thanks for the help!

            • 3. Re: Arquillian tests always passing
              Todd Morrison Newbie

              Continuing on this subject...

               

               

              Test class

               

              public class SimpleRegistrationTest extends Arquillian {
                @Deployment
                        public static Archive<?> createTestArchive() {
                                  return ShrinkWrap
                                                      .create(WebArchive.class, "test.war")
                                                      .addAsResource("test-persistence.xml",
                "META-INF/persistence.xml")
                                                      .addAsWebInfResource(EmptyAsset.INSTANCE, "beans.xml");
                        }
              
                @Inject
                        Logger log;
              
                @Test
                        public void dummyTest() {
                                  Assert.assertTrue(false);
                        }
              
              
                @Produces
                        public Logger produceLog(InjectionPoint injectionPoint) {
                                  return Logger.getLogger(injectionPoint.getMember().getDeclaringClass());
                        }
              }
              

               

               

              Fails on profile

               

              mvn clean test -Parq-jbossas-remote 
              

               

              with JBoss 6 Final container. As expected

               

              Failed tests:

                dummyTest(com.deepelement.membership.test.registration.SimpleRegistrationTest): java.lang.AssertionError : expected:<true> but was:<false> [Proxied because : Could not find suitable constructor]

               

               

               

               

              But, when I inject a bean without defining prerequisites I get a success even though everything catastrophically should fail.

               

              So, executing

               

              public class SimpleRegistrationTest extends Arquillian {
                @Deployment
                        public static Archive<?> createTestArchive() {
                                  return ShrinkWrap
                                                      .create(WebArchive.class, "test.war")
                                                      .addAsResource("test-persistence.xml",
                "META-INF/persistence.xml")
                                                      .addAsWebInfResource(EmptyAsset.INSTANCE, "beans.xml");
                        }
              
                @Inject
                        Logger log;
              
                @Inject
                MyUberBean uberBean;
              
                @Test
                        public void dummyTest() {
                                  Assert.assertTrue(false);
                        }
              
              
                @Produces
                        public Logger produceLog(InjectionPoint injectionPoint) {
                                  return Logger.getLogger(injectionPoint.getMember().getDeclaringClass());
                        }
              }
              

               

              With the same container and profile results in

               

              Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
              

               

              With no sign of the traditionally Deployment Dependency error on the JBoss Log (or test log)

               

              But, when I define the correct Injection Depedencies in the ShrinkWrap, the results go back to 'failure'.

               

               

               

              This is concerning because no JBoss/TestNG stack-trace and no test failure sure are a combination for false-quality.

               

              Any advice?

              • 4. Re: Arquillian tests always passing
                Todd Morrison Newbie

                This sounds allot like http://community.jboss.org/thread/164212

                 

                But, I am not using any fancy life-cycle annotations that would collide here.

                • 5. Re: Arquillian tests always passing
                  Todd Morrison Newbie

                  Analysing the TestNG stacks for both the 'correct failure' case and the 'false success' case, it is clear that the only difference is about 25 instances of

                   

                  DEBUG-SocketClientInvoker[2dc8b884, socket://127.0.0.1:4446] disconnecting ...
                  

                   

                  On the 'correct failure' case.

                   

                  Jboss logs are all identical.

                  • 6. Re: Arquillian tests always passing
                    Todd Morrison Newbie

                    This bug occurs on all 5.14.* builds.

                     

                    Not reproducible with JUnit

                     

                      <dependency>
                       <groupId>junit</groupId>
                       <artifactId>junit</artifactId>
                       <version>4.8.2</version>
                       <scope>test</scope>
                      </dependency>
                    

                     

                    Issue Resolved (sort of)

                    • 7. Re: Arquillian tests always passing
                      Ian Hands Newbie

                      Hi, I almost opened a new discussion but then I stumbled upon this one.

                       

                      I am having and similar situation where apparently the @Test annotated method bodies are not being run, but all tests seem to pass.

                      Unlike the OP my generated war is free of all testng jars (and I am not trying to use testng).

                       

                      Has anyone else experienced similar issues?

                       

                      Here is the simplest example I can come up with:

                      package com.example;
                      
                      
                      import java.io.File;
                      import java.util.ArrayList;
                      import java.util.Arrays;
                      import java.util.List;
                      import junit.framework.Assert;
                      import org.jboss.arquillian.container.test.api.Deployment;
                      import org.jboss.arquillian.junit.Arquillian;
                      import org.jboss.seam.mock.JUnitSeamTest;
                      import org.jboss.shrinkwrap.api.Archive;
                      import org.jboss.shrinkwrap.api.ShrinkWrap;
                      import org.jboss.shrinkwrap.api.exporter.ZipExporter;
                      import org.jboss.shrinkwrap.api.spec.WebArchive;
                      import org.jboss.shrinkwrap.resolver.api.maven.Maven;
                      import org.jboss.shrinkwrap.resolver.api.maven.strategy.RejectDependenciesStrategy;
                      import org.junit.Test;
                      import org.junit.runner.RunWith;
                      
                      
                      @RunWith(Arquillian.class)
                      public class ExampleTest extends JUnitSeamTest {
                      
                      
                        @Deployment(name = "ExampleTest")
                        public static Archive<?> createDeployment() {
                        final WebArchive war = ShrinkWrap.create(WebArchive.class, "portal-test.war").as(WebArchive.class);
                      
                      
                        war.delete("/WEB-INF/web.xml");
                        war.addAsWebInfResource("./web.xml");
                      
                      
                        // List for libs
                        final List<File> libs = new ArrayList<File>();
                      
                      
                        // Add seam libs
                        libs.addAll(Arrays.asList(Maven.resolver().loadPomFromFile("pom.xml").resolve("org.jboss.seam:jboss-seam")
                        .using(new RejectDependenciesStrategy(false, "org.testng:testng")).asFile()));
                      
                      
                        // add test libs
                        libs.addAll(Arrays.asList(Maven.resolver().loadPomFromFile("pom.xml").importTestDependencies().resolve().withTransitivity().asFile()));
                      
                      
                        // add all libs
                        war.addAsLibraries(libs.toArray(new File[libs.size()]));
                      
                      
                        // export the war for fun!
                        war.as(ZipExporter.class).exportTo(new File("/tmp/test.war"), true);
                      
                      
                        return war;
                        }
                      
                      
                        @Test
                        public void testFoo() throws Exception {
                        System.out.println("Test\nTest\nTest\nTest\nTest\nTest\nTest\nTest\nTest\nTest\nTest\n");
                        Assert.assertTrue(false);
                        Assert.assertEquals(1, 2);
                        }
                      }
                      

                       

                      Thanks!

                      • 8. Re: Arquillian tests always passing
                        Nick Hutt Newbie

                        I had a similar issue with Arquillian.

                         

                        I tried to instantiate an EJB that wasn't actually an EJB. Instead of reporting an error JUnit reported success for each test without running the tests.

                         

                        It seems in this, and possibly other cases of serious error, all tests are incorrectly reporting success but are not run.

                        • 9. Re: Arquillian tests always passing
                          Roman Besolov Newbie

                          Here is the answer: Injecting EJB into tests

                           

                          casta_oh said:

                          When something goes bad with the injection, the test doesn`t work but the test results as "OK".

                          • 10. Re: Arquillian tests always passing
                            Bartosz Majsak Master

                            This is strange. Did you try with latest core 1.1.5.Final?