Arquillian tests always passing

[toddpi314]

Diving into Arquillian testing finally.

Unfortunately, I am getting false positives across the board.

  <dependency>
   <groupId>org.testng</groupId>
   <artifactId>testng</artifactId>
   <version>5.14</version>
   <scope>test</scope>
  </dependency>




  <dependency>
   <groupId>org.jboss.arquillian</groupId>
   <artifactId>arquillian-testng</artifactId>
   <version>${arquillian.version}</version>
   <scope>test</scope>
  </dependency>
  <dependency>


  <dependencies>
    <dependency>
     <groupId>org.jboss.arquillian.container</groupId>
     <artifactId>arquillian-jbossas-remote-6</artifactId>
     <version>${arquillian.version}</version>
     <scope>test</scope>
    </dependency>

With Arg version

  <arquillian.version>1.0.0.Alpha5</arquillian.version>

A Simple Test-case that inherits from Arquillian test base that does an assertFalse(true) always passes.

Is this a known issue?

[aslak]

Can you see the test actually execute?

e.g. System.out.println() before the Assert. and check the server log.

And your using the Assert object right, not the assert keyword? If the keyword, you have to activate assertion on the container jvm

[toddpi314]

Aslak, thanks for the advice.

Calling

assertTrue(false);

on TestNG 5.14 results in 'success', no logging output and no break-points. (this is with a solid mvn clean on the target too, very creepy).

Updating to 5.14.6 and modifying the code to

Assert.assertTrue(false);

resulted in correct failure and logging working just fine.

This is not sane, so I would like to conclude that 'My Environment was jacked'.

I am working just fine now. Thanks for the help!

[toddpi314]

Continuing on this subject...

Test class

public class SimpleRegistrationTest extends Arquillian {
  @Deployment
          public static Archive<?> createTestArchive() {
                    return ShrinkWrap
                                        .create(WebArchive.class, "test.war")
                                        .addAsResource("test-persistence.xml",
  "META-INF/persistence.xml")
                                        .addAsWebInfResource(EmptyAsset.INSTANCE, "beans.xml");
          }

  @Inject
          Logger log;

  @Test
          public void dummyTest() {
                    Assert.assertTrue(false);
          }


  @Produces
          public Logger produceLog(InjectionPoint injectionPoint) {
                    return Logger.getLogger(injectionPoint.getMember().getDeclaringClass());
          }
}

Fails on profile

mvn clean test -Parq-jbossas-remote 

with JBoss 6 Final container. As expected

Failed tests:

  dummyTest(com.deepelement.membership.test.registration.SimpleRegistrationTest): java.lang.AssertionError : expected:<true> but was:<false> [Proxied because : Could not find suitable constructor]

But, when I inject a bean without defining prerequisites I get a success even though everything catastrophically should fail.

So, executing

public class SimpleRegistrationTest extends Arquillian {
  @Deployment
          public static Archive<?> createTestArchive() {
                    return ShrinkWrap
                                        .create(WebArchive.class, "test.war")
                                        .addAsResource("test-persistence.xml",
  "META-INF/persistence.xml")
                                        .addAsWebInfResource(EmptyAsset.INSTANCE, "beans.xml");
          }

  @Inject
          Logger log;

  @Inject
  MyUberBean uberBean;

  @Test
          public void dummyTest() {
                    Assert.assertTrue(false);
          }


  @Produces
          public Logger produceLog(InjectionPoint injectionPoint) {
                    return Logger.getLogger(injectionPoint.getMember().getDeclaringClass());
          }
}

With the same container and profile results in

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

With no sign of the traditionally Deployment Dependency error on the JBoss Log (or test log)

But, when I define the correct Injection Depedencies in the ShrinkWrap, the results go back to 'failure'.

This is concerning because no JBoss/TestNG stack-trace and no test failure sure are a combination for false-quality.

Any advice?

[toddpi314]

This sounds allot like http://community.jboss.org/thread/164212

But, I am not using any fancy life-cycle annotations that would collide here.

[toddpi314]

Analysing the TestNG stacks for both the 'correct failure' case and the 'false success' case, it is clear that the only difference is about 25 instances of

DEBUG-SocketClientInvoker[2dc8b884, socket://127.0.0.1:4446] disconnecting ...

On the 'correct failure' case.

Jboss logs are all identical.

[toddpi314]

This bug occurs on all 5.14.* builds.

Not reproducible with JUnit

  <dependency>
   <groupId>junit</groupId>
   <artifactId>junit</artifactId>
   <version>4.8.2</version>
   <scope>test</scope>
  </dependency>

Issue Resolved (sort of)

[iphands]

Hi, I almost opened a new discussion but then I stumbled upon this one.

I am having and similar situation where apparently the @Test annotated method bodies are not being run, but all tests seem to pass.

Unlike the OP my generated war is free of all testng jars (and I am not trying to use testng).

Has anyone else experienced similar issues?

Here is the simplest example I can come up with:

package com.example;


import java.io.File;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import junit.framework.Assert;
import org.jboss.arquillian.container.test.api.Deployment;
import org.jboss.arquillian.junit.Arquillian;
import org.jboss.seam.mock.JUnitSeamTest;
import org.jboss.shrinkwrap.api.Archive;
import org.jboss.shrinkwrap.api.ShrinkWrap;
import org.jboss.shrinkwrap.api.exporter.ZipExporter;
import org.jboss.shrinkwrap.api.spec.WebArchive;
import org.jboss.shrinkwrap.resolver.api.maven.Maven;
import org.jboss.shrinkwrap.resolver.api.maven.strategy.RejectDependenciesStrategy;
import org.junit.Test;
import org.junit.runner.RunWith;


@RunWith(Arquillian.class)
public class ExampleTest extends JUnitSeamTest {


  @Deployment(name = "ExampleTest")
  public static Archive<?> createDeployment() {
  final WebArchive war = ShrinkWrap.create(WebArchive.class, "portal-test.war").as(WebArchive.class);


  war.delete("/WEB-INF/web.xml");
  war.addAsWebInfResource("./web.xml");


  // List for libs
  final List<File> libs = new ArrayList<File>();


  // Add seam libs
  libs.addAll(Arrays.asList(Maven.resolver().loadPomFromFile("pom.xml").resolve("org.jboss.seam:jboss-seam")
  .using(new RejectDependenciesStrategy(false, "org.testng:testng")).asFile()));


  // add test libs
  libs.addAll(Arrays.asList(Maven.resolver().loadPomFromFile("pom.xml").importTestDependencies().resolve().withTransitivity().asFile()));


  // add all libs
  war.addAsLibraries(libs.toArray(new File[libs.size()]));


  // export the war for fun!
  war.as(ZipExporter.class).exportTo(new File("/tmp/test.war"), true);


  return war;
  }


  @Test
  public void testFoo() throws Exception {
  System.out.println("Test\nTest\nTest\nTest\nTest\nTest\nTest\nTest\nTest\nTest\nTest\n");
  Assert.assertTrue(false);
  Assert.assertEquals(1, 2);
  }
}

Thanks!

[nickhutt]

I had a similar issue with Arquillian.

I tried to instantiate an EJB that wasn't actually an EJB. Instead of reporting an error JUnit reported success for each test without running the tests.

It seems in this, and possibly other cases of serious error, all tests are incorrectly reporting success but are not run.

[besolov]

Here is the answer: Injecting EJB into tests

casta_oh said:

When something goes bad with the injection, the test doesn`t work but the test results as "OK".

[bmajsak]

This is strange. Did you try with latest core 1.1.5.Final?