Possible usage of JUnit categories in test suite
Hi,
we started investing into new test suite in RichFaces 4.3 called "fundamental tests" and Lukas created all necessary configuration in POM files so we can run it in CI, too. However, we are facing at least two major issues:
- since new tests use Selenium, they run much longer than unit tests and it is probable that the number of tests will increase significantly in the future so it won't be possible to run whole test suite for each commit
- when a new test for known (unresolved) issue is created it has to be ignored in test suite which causes that QE loses a lot of precious information because sometimes there are duplicates in JIRA or an issue is fixed with new release of Mojarra/MyFaces, QE wants to have a CI job with all failing tests (in which all tests should fail) so that we find out that some issue is fixed immediately; this is not possible with ignored tests
To solve these two problems we could introduce two JUnit categories (introduced in JUnit 4.8). The configuration would look like this:
public interface FailingTest { }
public interface SmokeTest { }
public class TestUnitB {
@Test
public void test1() { ... }
@Test
@Category(FailingTest.class) // https://issues.jboss.org/browse/RF-xxxxx
public void test2() { ... }
}
@RunWith(Arquillian.class)
public class TestIntegrationArq {
@Deployment
public static JavaArchive createDeployment() { ... }
@Test
@Category(FailingTest.class) // https://issues.jboss.org/browse/RF-xxxxx
public void test1() { ... }
@Test
public void test2() { ... }
}
There is also an option to define a test suite in Java class but IMHO it isn't useful because
- one would have to include new test class into test suite which would be error prone and would contain a lot of classes (atm ~220 tests classes in RF5)
- both dev and QE teams want to run tests from the IDE when developing tests which is well supported so this test suite defined in Java class has no use
- in CI, Maven profiles can be used instead of Java class which solves problem in #1
This is how Maven config would look like (profile for integration tests already exist in RF pom)
<profile> <id>unitTests</id> <!-- the same also for integration tests but with failsafe plugin --> <activation> <property> <name>!integration</name> </property> </activation> <build> <plugins> <plugin> <artifactId>maven-surefire-plugin</artifactId> <configuration> <excludedGroups>com.pitonak.category.FailingTest</excludedGroups> </configuration> </plugin> </plugins> </build> </profile> <profile> <id>failingTests</id> <!-- plus activate integration tests profile --> <build> <plugins> <plugin> <artifactId>maven-surefire-plugin</artifactId> <configuration> <groups>com.pitonak.category.FailingTest</groups> </configuration> </plugin> <plugin> <artifactId>maven-failsafe-plugin</artifactId> <configuration> <groups>com.pitonak.category.FailingTest</groups> </configuration> </plugin> </plugins> </build> </profile>
We would have to agree on rules for using these categories, I suggest this:
- each component would have about 10% of its fundamental tests in SmokeTest category
- majority of smoke tests should interact with component somehow, i.e. not tests for transient attributes such as styleClass nor for standard client-side event handlers such as onclick, onenter in r:tab can be smoke test
- FailingTest category would be used instead of JUnit's @Ignore annotation for all tests that are associated with some known (not yet resolved) issue, it would be removed from FailingTest category as soon as associated issue is fixed
- not a single test can be ignored
I wrote about two categories that in my opinion should be introduced immediately but we could introduce unlimited number of categories, e.g.:
- for JSF implementations - Mojarra, MyFaces, Mojarra2118...
- for runtime containers - EAP61, Tomcat7...
- for browsers - Firefox18, IE10, IOS6
Since I'm not sure how QE would handle so many categories I suggest to define them as needed in the future. There would be probaly one Jenkins job for each category which looks unmaintainable.
