Hi,

We have a number of issues with our tests that could be addressed / simplified by using newer testing frameworks (either JUnit4 or TestNG). So I would like to propose migrating to one of these at some point in the foreseeable future. I will list what we can get from a migration such as this and try to share my thoughts on JUnit4 and TestNG.

FWIW, I know JUnit4 pretty well but I only played with TestNG. In the previous project I worked on, we migrated ~ 35.000 tests from JUnit3 to JUnit4. We had special needs that led us to write custom TestRunners, and we had a lot of trouble with the lack of support for test generation through TestCase.suite(). Compared to that I only used TestNG for 'toy' testing.

[...]

The benefits of using a modern test framework would be:

• Syntactical support to ignore tests, allowing us to disable some tests and easily keep track of all disabled tests. It helps avoiding the situation where tests are marked to be ignored because of an issue, the issue is later marked as "fixed", and tests are forgotten and left ignored. A year later, someone rediscovers the ignored tests and turns out they fail. We had this with HORNETQ-65.

https://issues.jboss.org/browse/HORNETQ-65

TestNG:

@Test(enable="false")

public void testFoo() { ... }

JUnit4:

@Ignore @Test

public void testFoo() { ... }

JUnit4 really has the edge here. As you can add a comment to @Ignore liking it to the Jira case: @Ignore(“HORNETQ-42”), this makes it easy to enforce that all @Ignore are linked to a bug report.

• Test categories support. Right now we emulate test categories (e.g. integration, stress etc) by having different Maven projects carrying different groups of tests. It would be better to use a test framework that supported this instead of having an ad-hoc solution (forcing us into 'extra' maven projects as we now have).

TestNG:

@Test(groups = { "stress" })

public void testFoo() { ... }

(notice that syntax support for class-wide categories in TestNG is annoying but it is there)

JUnit4 has test categories, syntax is a bit different but Maven’s support for it is lacking.

• Setting a test timeout limit. We often have single tests that hang and block the entire build. A while ago we had consecutive Hudson runs where some test in PagingTest was hanging and blocking the build.

JUnit4/TestNG:

@Test(timeout = 1000)

public void testFoo { ... }

Not really game changers but nice to have:

1. Better support for parametric tests.
2. JUnit has this @Rules annotation which is extremelly useful at times. TestNG doens't really have the convenience.