Additional test grouping possibility

Jira: https://issues.jboss.org/browse/AS7-5844

PR: https://github.com/jbossas/jboss-as/pull/3513

Motivation

The tests are categorized into modules and then further to packages.

This forms a tree structure, in which we follow the most important distinctive feature/purpose of the test.

There are cases the tests need to be categorized in multiple ways. E.g. security or management tests. It's more practical to keep them separated by functionality tested.

But that means that when the group is to be run, user needs to provide (and maintain) a long list of packages to run.

There's a solution: JUnit's @Category.  See AS7-5844. Using that, the tests may be tagged, and special groups can be created, instead of creating and maintaining list of packages or even classes.

It's not supposed to replace package-based fine-grained division, but to complement it.

Another example where current model has some flaws are smoke tests: Separating the tests physically to different module is artificial and not much semantic: If any tests starts being considered as smoke test, what we do? Move it to different module and change package name. Does that make sense? Wouldn't it be better to just add `@Category(Smoke.class)` ?

It also collides with the technical solution of separating tests by type of server setup (multinode, ...). If we decided to include a multinode test as smoke, what would we do? Move it to smoke module and create another surefire execution? Doesn't make much sense. Just adding the `Smoke` category and filtering the tests to be run using categories is way better.

Last but not least: These categories are totally optional, no one has to go through the tests and add them.

It's a tool for use cases when this additional categorization is necessary, like, security-related tests, it's subset Common Criteria tests, JPA-related tests, "performance smoke tests", etc.

Other tests would remain untouched. The packages will remain the main manner of categorization.

Summary: Tests have some cross-cutting concerns, facetes, which can't be covered by a tree structure, by principle. Also, we are bound by technical solutions coming from Maven's and Arquillian's limitations. Having additional way to categorize can only help, without doing any harm.

Who needs it?

Mostly QA - when developing tests harness, when testing under various conditions (different DB, IPv6, Common Criteria, security policy, ...)

What will change?

Everything will work as before. People not interested in this won't notice anything.

The only change needed is

• to define the tags (as empty interfaces)
• to add the @Category( <GroupInterface>.class ) to some test's class (and add it to Arquillian's deployment if necessary).

Purpose of each group will be documented in the interface's javadoc.

Tags (groups, if you like)

So far, these are considered:

- CommonCriteria

- Management

- Security

Stuart's note: It would be nice if the annotation could be applied at a package level, as security tests are already isolated into individual packages

- Slow

Needs a definition of what's "slow", possibly in terms of comparison to some quick test.

- Negative

This one is different kind: QA wants to mark the negative tests to track where they have some and where they don't.

So this is rather supposed for quality engineering tools.

- Smoke (maybe)

Stuart's note: Unless we actually move the smoke tests into testsuite/integration/basic then this is pointless.

There are adv and disadv for moving tests into .../basic:

+ It would make adding/removing a test to the smoke group easier.

+ It would allow marking also tests in other modules as smoke.

*  Determining whether a test is smoke test would need to go and look. But does anyone need to know it when not actually managing the set of smoke tests, in which case they would just look for the usages of the Smoke type?

-  To get the current set, you'd need to search for Smoke type appearances, or to run the smoke tests.  On a related note, I have a feature request in Surefire to dry-run tests (just list which would run).

Improve single-test runs ( -Dtest=...)

Running a single test is problematic with current solution.

We use multiple runs of Surefire. Maven ignores the includes and excludes when using -Dtest=... That's not optional. (There's a feature request for it though.)

Therefore, Maven runs the given test(s) in all defined Surefire executions.

That usually results in one of them passing, and the rest failing.

If the failing configuration happens to be prior to the correct one, Surefire fast-fails and the other execution is skipped.

Currently, the only fix for it is to manually edit pom.xml to disable the other executions.

Until the aforementioned RFE is implemented, we need to workaround this by switching to Failsafe plugin.

Until recently, that wasn't possible due to bugs in Failsafe. Now it seems to be ok.

Logging issues, reporting improvements

Several logging issues were reported during test development.

Not sure if it got fixed, but I can't reproduce it right now.

Could have been caused by productization (different logging libs version?).

To be verified.

There's a RFE regarding reporting outside Jenkins.

1) A test runs summary doesn't work in EAP. Work in progress.

2) When running in different configurations, various stakeholders requested to be able to have the results compared in a comprehensible table.

    That's what project JUnitDiff aims to do. It has been around for a while, but now it's a good opportunity to get it integrated into the testsuite.

Another considerable thing is to split AS logs into files by tests being run. Technically, it's doable by using custom appender, which would roll the files on change of a property, which would be set by mgmt API.

Test runs should be more independent of build environment

Currently, some of tests count on being run with the AS built in the /build/target/jboss-as-... directory.

They use some files which are generated during the build.

Therefore, the AS has to be built before running the testsuite.

This causes severe issues when running the testsuite/build against product, especially in conditions like:

• Outside Red Hat VPN
• Offline
• Executing plugins which go deeper into the depencency structure

The simple solution is to create ...-test.jar artifacts during the build and make it available for the testsuite to be run.