TestNG Groups with Arquillian

[tombee]

Hi,

I'm trying to separate my tests so that I can run my unit tests independent to my Arquillian integration tests.  It works fine if I add <excludes> patterns to the surefire plugin, but what I was hoping is to use TestNG's groups functionality.  I had noticed that the example pom.xml in the Arquillian git repository doesn't use TestNG groups either.

I've tried it, and it seems to pick up that my Arquillian test is not included, or is included.  The issue then is that it encounters issues with the Arquillian class that it has extended.

With Arquillian test excluded via TestNG groups through maven-surefire-plugin:

Running TestSuite

19-Sep-2011 22:32:20 org.jboss.arquillian.container.impl.client.container.ContainerRegistryCreator getActivatedConfiguration

INFO: Could not read active container configuration: null

Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 0.545 sec <<< FAILURE!

Results :

Failed tests:   arquillianBeforeSuite(com.acme.corp): Could not create a new instance of class org.jboss.arquillian.test.impl.EventTestRunnerAdaptor see cause

With test included, and running in container:

Failed tests:   arquillianBeforeClass(com.acme.corp): Could not invoke deployment method: public static org.jboss.shrinkwrap.api.Archive com.acme.corp.createDeployment()

  arquillianAfterTest(com.acme.corp)

  arquillianAfterTest(com.acme.corp)

  arquillianAfterClass(com.acme.corp)

Any hints on getting this working are appreciated.

Cheers.

[aslak]

Do you have more of the stack traces?

The last one seems to simply imply that for what ever reason it can't create the ShrinkWrap archive...

[tombee]

Aslak,

Thanks for the reply.  I'd been battling with this today, Jose in #jbosstesting managed to help me to sort out the last one.  This was done by updating the dependencies (from the kitchensink jboss archetype), from 7.0.0.CR1 to 7.0.1.Final.  Also ensuring that TestNG was at 6.1.1.

There was also an issue that I came across that was covered in another post here, where I had to specify Servlet 3.0 in my arquillian.xml.

The issue I am still seeing is that I have one unit test under group "unit" and an arquillian test under group "integration", when I run with just group "unit".  TestNG is trying to execute the arquillianBeforeSuite() method in the Arquillian class.

Maybe this is a TestNG bug?

[tombee]

It seems this is causing the issue:

In the Arquillian base class for a test:

   @BeforeSuite(alwaysRun = true)

   public void arquillianBeforeSuite() throws Exception

The other Before/After methods have got the alwaysRun parameter also.

The TestNG documentation:

alwaysRun - For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not beforeGroups): If set to true, this configuration method will be run regardless of what groups it belongs to.

[aslak]

yea, they were added so the methods would always be called regardless of failure, e.g. BeforeXX fail, AfterXX should still be called. But they seems to have multiple 'strange' side effects..

[tombee]

OK so I guess this isn't going to get resolved anytime soon

I've raised a JIRA here:

https://issues.jboss.org/browse/ARQ-605

I think the workaround for now is probably to separate tests into a unit-testng.xml and arquillian-testng.xml or so, then use the surefire plugin to run those perhaps?

[aslak]

i'll see if i find a soution that fits all usecases here and keep grouping with non Arq tests in mind.

Either spit out the Arq tests in a separate module or use surefire fiters/suite files seems to be the options.

[aslak]

Which level are you placing the group? pr Method or Class ?

[mazi]

Hi,

I have the same problem: I can't use sureFire's <excludedGroups> to exclude the Arquillian tests. This seems to me a very common use-case; I imagine many people want to put their Arquillian tests in a separate group and run them much less often than simple unit tests since they are slow and may require a server to be running.

I'm seeing the problem with group definition at class level:

@Test(groups = "in-container")

public class LogoutRedirectTest extends Arquillian { ... }

Matija

[tombee]

Tried it at both class and method level.  The issue is that annotation with the 'alwaysRun' parameter.  Has that side-effect that there must always be a container available.

I agree with Matija here, this is a common use-case that would be helpful if resolved soon.

The workaround relying upon exclusion via test suite XML files is very nasty, requires a lot more configuration as test config is done at the parent level on my project.  Therefore I need to dump empty xml files in each of the modules for projects that do not have Arquillian tests.

[mazi]

I've done it now using exclusions by name in pom.xml:

<?xml version="1.0" encoding="UTF-8"?>

<project>

    <properties>

        <testng.excludedTests>**/*ArqTest.java</testng.excludedTests>

    </properties>

    <build>

        <pluginManagement>

            <plugins>

                <plugin>

                    <artifactId>maven-surefire-plugin</artifactId>

                    <configuration>

                        <excludes>

                            <exclude>${testng.excludedTests}</exclude>

                        </excludes>

                    </configuration>

                </plugin>

            </plugins>

        </pluginManagement>

    </build>

    <profiles>

        <profile>

            <id>arq.jbossas-remote-7</id>

            <properties>

                <testng.excludedTests>**/XXX</testng.excludedTests>

            </properties>

        </profile>

    </profiles>

</project>

This seems to work for me but I would prefer the solution with groups because it would be cleaner and wouldn't require the naming convention that this config relies on (that all Arquillian test classes be named *ArqTest). The amount of config one has to write is the same with this as with groups, though.