XTS Testing

Current Status and Roadmap

This Document describes the current status of the XTS tests and what technology they use. There are currently three sets of tests, Unit, Interop and recovery. Each of these is described in turn, and a list of required improvements is presented.

All improvements are targeted at Narayana 5.x unless specified otherwise.

Unit tests

Each XTS component has a set of unit tests. These tests need to be ran within an instance of JBoss AS.  These tests are fully automated by an Ant script ran by Hudson.

Improvements

High Priority (EAP 6.0)

1. Ensure AS 7 compatibility. See issue JBTM-900.
2. CI with JBossAS. This will ensure that changes to components, we depend upon (like JbossWS), that break XTS are spotted at QE time rather than after release. (EAP 6 requirement). JBQA-5191
3. Additional Testing. XTS Demo tests JBQA-5194https://issues.jboss.org/browse/JBQA-5194

Medium Priority (Narayana 5, EAP 7)

1. Automate Emma and collate results. Emma should be used to obtain coverage stats on the test run. Emma, produces individual reports per test, by default. It would be better to have these reports combined as we are concerned with the overall test coverage, rather than the coverage of each test. Ideally we would combine coverage stats over all sets of tests (unit, interop and recovery). Next step would be to improve the coverage where necessary. JBTM-952

Low Priority (Narayana 5, EAP 7)

1. Migrate to Maven. For consistency with rest of the Narayna project. JBTM-945
2. Update to use Arquillian. This would mean that that they can be ran from anywhere that can run JUnit tests, such as an IDE or maven. It would also automate the app server lifecycle and test deployment. JBTM-954
3. Remove home-brew SOAP stack. In the past these tests used a mock/simple SOAP stack developed specifically for the tests. This stack is no longer used as the tests run within JBoss AS. This code is redundant and should be removed. JBTM-953

Interop Tests

We have two sets of interop tests that live in "XTS/interop". These are built with ant. They are each ran by deploying them as a war to a single JBoss instance which deploys the services needed by the test. This war also exposes a web interface that on request runs the tests and relays the results in the http response.

This process is automated by using ant to deploy the war and then making the http request and validating the response. See here in code for scripts: “XTS/localjunit/run-interop-tests.xml” and “XTS/localjunit/run-tests.xml”

Improvements

High Priority (EAP 6.0)

1. Ensure AS 7 compatibility. See issue JBTM-905.
2. Move into the AS test suite. JBQA-5192

Medium Priority (Narayana 5, EAP 7)

1. Automate Emma and collate results. JBTM-952

Low Priority (Narayana 5, EAP 7)

1. Update to use Arquillian. JBTM-955 JBTM-956
2. Migrate to Maven. JBTM-943

Recovery Tests

The recovery tests are the tricky ones. At the moment we have a set of test scenarios that run in a single JBoss server invoked by a remote client. There are also a set of Byteman scripts that trigger failure at certain points in a scenario. There are many different permutations of scenario and Byteman scripts that each create a particular test.

The test scenarios log their progress through the protocol and recovery. On completion of a test run, a human needs to look over the trace and check that it looks right. This process is hard to automate as the trace produced can have many valid interleavings. This is due to the asynchronous nature of the application.

These tests are automated with a bash script, but the output traces must be verified manually by a human. The other problem is that they currently run in a single JBoss server which simulates the situation where every party in the protocol is located in the same app server and crashes & recovers at the same time (not that realistic!). In order to test with multiple servers. we would need a way of combining the traces from each server when verifying the outcome of the test. This could be done by implementing a Byteman helper class, but it would not be trivial.

Improvements

High Priority (EAP 6.0)

1. Move into AS test suite. The tests are unlikely to be accepted in their current form, due to the  level of manual intervention required. This improvement can not be made until we have sufficiently automated the process.
2. Ensure AS 7 compatibility. JBTM-923
3. Automate running. Use Arquillian to automate the tests and generate trace output for human verification. JBQA-3926 JBTM-817
4. Run as part of CI. JBTM-1029
   

Medium priority (Narayana 5, EAP 7)

1. Automate Emma and collate results. JBTM-952

Low Priority (Narayana 5, EAP 7)

1. Migrate to Maven. JBTM-944
   
2. Automate verification. Remove the human verification steps. JBTM-949
3. Multiple Server Tests. If step 2) is successful, we could be able to build the more complex scenarios where each party runs in its own JBoss server. JBTM-950
4. Additional Tests. Andrew Dinn has provided a set of additional tests for us to consider implementing. These should be considered alongside the Emma coverage data for future work. JBTM-951

Notes

• Arquillian supports multiple JBoss servers:
  • https://docs.jboss.org/author/display/ARQ/Multiple+Containers