JRunit

JRunit is a project to aid in adding benchmarking to JUnit based test cases as well as providing a framework extension to JUnit to allow for distributed client/server based tests (which will be referred to as jrunit in this document).  Most of the contents here will be focused on jrunit. 

How to get JRunit

Checkout the jrunit from CVS (i.e. cvs -d :ext:user@cvs.forge.jboss.com:/cvsroot/jboss checkout jrunit).  There is no distribution for this project yet as is in early development.

Reason for jrunit

With regards to client/server based testing, there are certain limitations of JUnit that make it impractical to use out of the box.  The biggest limitation is that it was designed to have all tests run within a single process within a single class.  It is also designed with the constraint that all tests are atomic and independent of any other code outside the test being run and that the lifecycle for these tests are independent for each test.  This is all great for testing low level units of code, but not for client/server scenario tests.

Due to the specific nature of client/server code, there are some extra requirements for testing.  The first is that the lifecycle, or state, of the client and server tests can be managed.  This is important because will need to make sure that the server is created and initialized before any clients begin their tests which will call on the server and that the server does not shutdown before all the clients have finished their tests.  Also need the ability to consolidate the results from all the client and server tests into a single result format.  Finally, need to be able to have all this driven from a single point so can be included within an automated test run from a build.

jrunit addresses these issues by allowing multiple test cases, client and server, to be run simultaneously in different processes, but controlled through a central driver.  This driver controls the complete lifecycle of all the test cases.  Each of the test cases spawned by the driver are run within a harness that is responsible for reporting all the results back to the driver.  This driver is itself a TestCase, so can be run directly from an IDE or ant build as a JUnit test case.

Requirements satisfied by JRunit

The JRunit project was built to satisfy a set of requirements for both distributed testing and benchmark reporting.

Samples for jrunit

If prefer to read code instead of doc, can just go to the samples section and see how all this works via code examples. 

How jrunit works

The first issue to tackle is the lifecycle of the server tests, since it will need to break from the default behavior of JUnit's lifecycle management. 

To start, need a way to tell the server to start up and initialize.  This should be done using setUp() method.  Once this method returns, it is assumed that the server is ready to receive calls.  Next, need a way to tell the server that it can shutdown and clean up.  This should be done using the tearDown() method.  This method will only be called in the case that all clients are finished making their calls to the server.

This behavior is all provided by having the server test case class extend org.jboss.jrunit.ServerTestCase instead of junit.framework.TestCase.  The ServerTestCase actually extends the TestCase class itself, but overrides a few its methods to get the desired behavior.

The server implementation may also have a test method, which will be called just after the setup(), as in the case of regular JUnit test runs.  This test method can contain asserts and are suggested to be used for validation of server data and metrics. 

A few important points: there can be only ONE test method within a server test case.  This is because JUnit creates a new instance of the test class for each test method run.  Therefore, if had multiple tests, would be multiple instances of the server test case created.  To change this would require a sizable change to the JUnit code, so please just use one test method (or contribute the required change ).

Another important point is that the tearDown() method may be called even while a test method is being run.  This is intentional and allows for the test method to loop until the tearDown() method is called.  So a possible example of where this could be used is:

...
private boolean stop = false;

public void testServerMetrics()
{
     while(!stop)
     {
           // collect data here
     }
}

protected void tearDown()
{
     stop = true; // so will cause testServerMetrics() to break out of loop
     
     // do shutdown and clean up code.
}

For the client test case, there is no requirement for jrunit other than they extend the junit.framework.TestCase class and conform to normal constraints of a JUnit test case.

Lastly, there is the org.jboss.jrunit.TestDriver class, which represents the driver for the client and server tests.  This class will spawn new test harnesses that the client and server tests cases will run within.  The test driver will then communicate to the test harnesses using a JGroups message bus to control the test lifecycle for the server test case and all the client tests cases as well as obtaining the results from those test runs. 

The logical order of a test run as controlled by the test runner is:

1.      Spawn a new test harness process for each client and server test case.

2.      Wait for confirmation that all test harnesses have been created and their message bus has been started.

3.      Once confirmation has been received, wait for server test case to start up (i.e. call the setUp() method on the server test case).  Otherwise, if confirmation not received, kill all the processes and return error to JUnit.

4.      Once the confirmation of the server startup has been received, tell all the test cases to run (client and server).  Otherwise, if confirmation not received, send abort message to all the test harnesses.

5.      Wait for results from all the client test cases.

6.      Once all the client test results are received, tell the server to tear down (i.e. call the tearDown() method on the server test case).  Otherwise, if results not received, kill all the processes and return error to JUnit.

7.      Wait for server test results (if server test case had a test method).

8.      Process all results by adding them to root JUnit TestResult, which will be reported via normal JUnit reporting.

9.      Wait for server torn down message, indicating it has successfully cleaned up.

10.      Shutdown message bus and end root test run, returning JUnit execution thread.

The only user coding required for the test driver is to implement a class that extends the org.jboss.jrunit.TestDriver abstract class and implement the declareTestClasses() method.  Within this method, call the TestDriver’s addTestClasses() method and specify the client test case class, the number of clients to spawn, and the server test case class.

Benchmark Decorators

jrunit can also utilize the benchmark decorators to provide benchmark results as well as regular JUnit test results.  Although this document is not meant to cover the benchmark features in detail, it is important to understand a few points as to how it works.

ThreadLocalDecorator (and all the other decorators that accept number of threads and loops) - for each thread specified in the number of threads, there will be a new instance of the test created.  For the number of loops specified, the test methods will be called on each test instance.  So for example, if specify 3 threads and 10 loops, there will be three instances of the test class created and each instance will have their test methods called 10 times by each of their respective threads.  An example class that demonstrates this would be:

public class SimpleThreadLoopCounter extends TestCase
{
   private static int staticCounter = 0;
   private static int staticMethodCounter = 0;
   private int localCounter = 0;

   public static Test suite()
   {
      return new ThreadLocalDecorator(SimpleThreadLoopCounter.class, 3, 10, 0, true, true);
   }

   public SimpleThreadLoopCounter()
   {
      staticCounter++;
   }

   public void testCounter() throws Exception
   {
      System.out.println("staticCounter = " + staticCounter);
      System.out.println("staticMethodcounter = " + ++staticMethodCounter);
      System.out.println("localCounter = " + ++localCounter);
   }
}

When this is run, the last entry will be:

staticCounter = 3
staticMethodcounter = 30
localCounter = 10

Another issue is the way junit treats test classes with multiple test methods.  For each test method that junit runs within a test class, it will create a new test instance to run that test method.  To illustrate this, if copied the testCounter() method from the example above (calling it testCounter2() for example), the last lines printed for the test run would be:

staticCounter = 6
staticMethodcounter = 60
localCounter = 10

Issues with jrunit and decorators:

The ServerTestHarness will not work if the numberOfThreads is more than 1.  I don't see this as being that big of a problem from the point of view that the numberOfThreads is to simulate multiple clients running at the same time.  If want to do this using the ServerTestHarness, can actually spawn multiple clients through it.  Eventhough they won't be running concurrently with the same processes, they will be running concurrently with seperate processes.  Having the ability to use the loop parameter is needed though so can keep the clients calling on the server for an extended period of time (or iterations).  Therefore, if running remote tests and want to use the ThreadLocalDecorator for benchmark data, use the following ThreadLocalDecorator constructor, which will default the number of threads to one:

public ThreadLocalDecorator(Class testClazz, int loops)