arquillian, jacoco and client mode testing

[abendt]

Hi all,

i am trying to collect coverage data for my arquillian based end-to-end tests for a jboss as7 based app.

my first approach was the arquillian jacoco extension http://arquillian.org/modules/jacoco-extension/. This did not work as my tests use @Deployment(testable = false) (see https://issues.jboss.org/browse/ARQ-1014).

next i tried to use the agent approach using the maven jacoco plugin. i started the agent using the maven plugin and passed the argLine parameter via arquillian.xml to the as. this partly works. however the coverage data is too low due to the class manipulations of the jboss as.

my last approach was to instrument the deployment programatically (similiar to the extension). this generally seems to work. however i need to write the coverage data after the tests have run. i tried to achieve this by registering a shutdown hook inside the jboss as. unfortunately the shutdown hook does not seem to be invoked on test shutdown.

my questions:

* ideally i would like to use the jacoco extension. are there any plans to support client mode testing?

* why is my shutdown hook never invoked? how does arquillian shutdown the container? i hope it shuts down jboss as properly

* does anyone have a working solution for collecting coverage data using arquillian and client mode testing?

thanks for any advice,

   Alphonse Bendt

[trepel]

Hi Alphonse,

maybe you can try the agent approach using maven-surefire-plugin instead of maven jacoco plugin. Sample configuration:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
         <configuration>
             <argLine>-javaagent:${basedir}/../../build/lib/jacocoagent.jar=append=true,destfile=${basedir}/../../build/coverage-classes/target/jacoco.exec</argLine>
         </configuration>
</plugin>

But I don't know whether this solve your "coverage data is too low due to the class manipulations of the jboss as" problem.

Regards,

Tomas

[abendt]

Hi Tomas,

thanks for your reply. Unfortunately your approach won't work. There are two JVMs: One started by the maven-surefire-plugin running the tests. Another one started by Arquillian, running the AS and the deployed application. The maven plugin configuration will affect only the first (test) JVM. As the actual code is executed in the second (AS) JVM this will not help.

If you start the second JVM with the javaagent commandline (e.g. via arquillian.xml) you will run into the other problem i have mentioned. The AS seems to execute dynamically generated classes instead of your original ones. In the coverage report you will have too low coverage for your classes.

In the meantime i found a way to collect the coverage. Here are my steps:

1) instrument my deployment programmatically inside your @Deployment method

2) add a servlet to my deployment that allows me to dump the coverage from the outside. I tried first to dump the coverage data within a JVM shutdown handler. However it seems arquillian does not shutdown the JVM properly and kills it instead ...

3) run my tests

4) invoke the servlet after all tests have run (@AfterTest)

regards,

   Alphonse

[bmajsak]

Hi Alphonse,

many thanks for sharing your solution. Would you be so kind and elaborate it a bit, so others can also try it out?

Cheers,

Bartosz

[abendt]

Hi Bartosz,

sure. However i don't have a stripped down sample so i cut out all the relevant code snippets (hopefully!).

This solution originally used some code from the arquillian-jacoco-extension. Then i upgraded to a newer version of jacoco to be able to interact with current sonar versions. The jacoco API changed quite a bit so

i could not use much of the arquillian-extension anymore.

regards,

    Alphonse

Maven dependencies:

<dependencies>

        <dependency>
            <groupId>org.jacoco</groupId>
            <artifactId>org.jacoco.core</artifactId>
            <version>0.6.2.201302030002</version>
        </dependency>


        <dependency>
            <groupId>org.jacoco</groupId>
            <artifactId>org.jacoco.agent</artifactId>
            <version>0.6.2.201302030002</version>
        </dependency>

</dependencies>

creating the deployment. we use client mode testing and reference an artifact created by a maven build.

    @Deployment(testable = false)
    public static EnterpriseArchive accessDeployment() {
            File theEar = MavenDependencyResolver.resolve("myGroupId", "myArtifactId", "1.1.0-SNAPSHOT", "ear");

            EnterpriseArchive archive = ShrinkWrap.createFromZipFile(EnterpriseArchive.class, theEar);

            CoverageInstrument.instrumentArchive(archive);

             return archive;
     }

the class CoverageInstrument is used to instrument some of the archives within an EAR. we only instrument some of the artifacts based on the artifact name.

public class CoverageInstrument {

    public static void instrumentArchive(Archive archive) {
        System.out.println("BEFORE PROCESS ARCHIVE: " + archive.toString());

        Map<ArchivePath, Node> classes = archive.getContent(Filters.include(".*\\.class"));

        for (Map.Entry<ArchivePath, Node> entry : classes.entrySet()) {
            Asset original = entry.getValue().getAsset();

            archive.delete(entry.getKey());
            archive.add(
                    new InstrumenterAsset(original),
                    entry.getKey());
        }

        Map<ArchivePath, Node> jars = archive.getContent();
        for (Map.Entry<ArchivePath, Node> entry : jars.entrySet()) {
            String name = entry.getKey().get();
            String filename = name.substring(name.lastIndexOf("/"));
            Asset asset = entry.getValue().getAsset();

            // only process "our" JARs
            if (name.indexOf("myArtifactId") >= 0 && name.endsWith(".jar")) {
                InputStream in = asset.openStream();
                JavaArchive jar = ShrinkWrap.create(ZipImporter.class, filename).importFrom(in).as(JavaArchive.class);
                instrumentArchive(jar);
                archive.delete(entry.getKey());
                archive.add(jar, entry.getKey().getParent().get(), ZipExporter.class);
            } else if (name.indexOf("myArtifactId") >= 0 && name.endsWith(".war")) {
                InputStream in = asset.openStream();
                WebArchive war = ShrinkWrap.create(ZipImporter.class, filename).importFrom(in).as(WebArchive.class);
                instrumentArchive(war);

                war.addClass(CoverageServlet.class);
                archive.delete(entry.getKey());
                archive.add(war, entry.getKey().getParent().get(), ZipExporter.class);
            }
        }

        if (archive.getName().endsWith(".ear")) {
            System.out.println("ADDING jacoco libs to EAR/lib");
            JavaArchive coverageJar = ShrinkWrap.create(JavaArchive.class);
            coverageJar.addClass(Runtime.class);

            archive.add(coverageJar, "lib", ZipExporter.class);

            addMavenDependencyToArchive(archive, "org.jacoco", "org.jacoco.core");

            File jacocoAgent= MavenDependencyResolver.resolve( "org.jacoco", "org.jacoco.agent", null);
            JavaArchive jacocoAgentJar = ShrinkWrap.createFromZipFile(JavaArchive.class, jacocoAgent);
            JavaArchive nestedjacocoAgentJar = jacocoAgentJar.getAsType(JavaArchive.class, "jacocoagent.jar");

            archive.add(nestedjacocoAgentJar, "lib", ZipExporter.class);

            System.out.println(archive.toString(true));
        }
    }

    private static void addMavenDependencyToArchive(Archive archive, String groupId, String artifactId) {
        File jacocoCore = MavenDependencyResolver.resolve(groupId, artifactId, null);
        JavaArchive jacocoCoreJar = ShrinkWrap.createFromZipFile(JavaArchive.class, jacocoCore);
        archive.add(jacocoCoreJar, "lib", ZipExporter.class);
    }
}

class InstrumenterAsset

public class InstrumenterAsset implements Asset {
    private Asset asset;

    public InstrumenterAsset(Asset asset) {
        this.asset = asset;
    }

    public InputStream openStream() {
        try {
            Instrumenter instrumenter = new Instrumenter(new OfflineInstrumentationAccessGenerator());
            InputStream in = asset.openStream();

            try {
                byte[] instrumented = instrumenter.instrument(in);

                return new ByteArrayInputStream(instrumented);
            } finally {
                in.close();
            }
        } catch (Exception e) {
            throw new RuntimeException("Could not instrument Asset " + asset, e);
        }
    }
}

the CoverageServlet

@WebServlet(value = "/jacoco")
public class CoverageServlet extends GenericServlet {

    private static final long serialVersionUID = 1L;

    @Override
    public void service(ServletRequest req, ServletResponse res) throws ServletException, IOException {
        System.out.println("INVOKE COVERAGE SERVLET");

        try {
            MBeanServer mBeanServer = ManagementFactory.getPlatformMBeanServer();
            ObjectName jacoco = new ObjectName("org.jacoco:type=Runtime");
            printVersion(mBeanServer, jacoco);
            dumpCoverage(mBeanServer, jacoco);

            res.getWriter().println("COVERAGE DUMPED");
        } catch (Throwable e) {
            e.printStackTrace();

            res.getWriter().println("ERROR"); 

            throw new RuntimeException(e);
        }
    }

    private void dumpCoverage(MBeanServer mBeanServer, ObjectName jacoco) throws InstanceNotFoundException, MBeanException, ReflectionException {
        mBeanServer.invoke(jacoco, "dump", new Object[] {Boolean.TRUE}, new String[] {boolean.class.getName()});
    }

    private void printVersion(MBeanServer mBeanServer, ObjectName jacoco) throws MBeanException, AttributeNotFoundException, InstanceNotFoundException, ReflectionException {
        Object version = mBeanServer.getAttribute(jacoco, "Version");
        System.out.println("Jacoco JMX version: " + version);
    }
}

start the JBoss server with some additional arguments:

-Djacoco-agent.jmx=true -Djacoco-agent.destfile=target/jacoco-it.exec

dump the coverage data after the test has finished.

 @AfterClass
 public static void afterArquillianTest() throws Exception {
        URL oracle = new URL(TestConfiguration.getJBossUrl() + "jacoco");
        BufferedReader in = new BufferedReader(
                new InputStreamReader(oracle.openStream()));

        try {
            String inputLine;
            while ((inputLine = in.readLine()) != null) {
                assertEquals("COVERAGE DUMPED", inputLine);
            }
        } finally {
            in.close();
        }
}

after the test has finished the coverage data is dumped into the file target/jacoco-it.exec

[cmcmillen1]

I'm not sure if this is what you already tried but this is working for us:

pom.xml

{code:xml}
<profile>
  <id>jbossas-managed-7</id>
  <dependencies>
    <dependency>
      <groupId>org.jboss.as</groupId>
      <artifactId>jboss-as-arquillian-container-managed</artifactId>
      <version>${jboss.as7.version}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.jacoco</groupId>
      <artifactId>org.jacoco.core</artifactId>
      <version>${jacoco.version}</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
  <build>
    <testResources>
      <testResource>
        <directory>src/test/resources</directory>
        <filtering>true</filtering>
      </testResource>
    </testResources>
    <plugins>
      <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <version>${jacoco.version}</version>
        <configuration>
          <dumpOnExit>true</dumpOnExit>
        </configuration>
        <executions>
          <execution>
            <goals>
              <goal>prepare-agent</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <argLine>-Xms256m -Xmx768m -XX:MaxPermSize=512m</argLine>
          <systemPropertyVariables>
            <arquillian.launch>jbossas-managed</arquillian.launch>
          </systemPropertyVariables>
        </configuration>
      </plugin>
    </plugins>
  </build>
</profile>
{code}

arquillian.xml

{code:xml}
  <container qualifier="jbossas-managed" default="false">
    <configuration>
      <property name="javaVmArguments">-Xms256m -Xmx512m -XX:MaxPermSize=512m ${argLine}</property>
      <property name="managementPort">10199</property>
      <property name="jbossHome">C:\path\to\jboss</property>
    </configuration>
</container>
{code}