Current JRunit requirements

-

Distributed testing:

1. Run multiple test clients and a test server concurrently.

2. Framework should ensure that test server is setup and running before any test clients begin their setup or test runs.

3. Test server should continue to run until all test clients complete test runs or are determined to be hung.

4. Test client and test server should be able to run stand alone so that can run each in a separate process within an IDE for debugging.

5. Test client and test server should mimic (or preferably extend) JUnit framework.

6. Need to be able turn on and retrieve logging from both test clients and test servers.

7. Need to be able to execute and report entire test run (clients and server) via junit ant task.  This is needed so can include jrunit tests within full testsuite run from root ant target.

8. Need to be able to specify both client and server test environment variables (i.e. jvm config) via ant task.

9. Zero configuration for discovery.  This means that if running 10 client tests and one server test, all 11 instances can be discovered and communicated to (in terms of test commands like run tests) without having to change configuration for the test framework (like specifying ports that each instance will run on).

10. Framework must be able to detect/discern if test instance is hung and ensure that all test instances are cleaned up so do not have orphaned processes after test run.

Benchmark testing:

1. Should be able to easily mark within test code when to start and stop benchmark capturing (i.e. ThreadLocalBenchmark.openBench(“GetSocket”) and ThreadLocalBenchmark.closeBench(“GetSocket”)).

2. Need to be able to provide extra metadata about the benchmark being captured, such as test environment properties (e.g. OS, transport type, payload size).

3. Need to be able to gather and report benchmark results to different outputs.

1. Local file (raw data)

2. Central database

3. System output

4. Local file (html report)