Samples walk-through

This covers the samples include within the jboss-benchmark project.  The main focus of the samples covered here are related to client/server tests.  All the samples illustrate the different features of microbenchmark and jrunit using the same base client and server base code which uses a socket to send a String to the server from the client, which is then returned a static String from the server. 

All the samples can be compiled and run within an IDE, but some can be run via the ant build script included in the root directory of jboss-benchmark.  The reason that only some are included within the ant script is that they can be completely automated (which will be covered below). 

Also note that each section title for the samples below corresponds to the org.jboss.jrunit.sample package that they can be found under. 

basic

The first sample for client/server testing is in the org.jboss.jrunit.sample.basic package, which contains SimpleServerTest and SimpleServerTest.  These are the tests that we will build on for the rest of the test cases that demonstrate the different features.  The SimpleClientTest is a basic JUnit test case with two test methods, testClientCall() and testFailure().  The testClientCall() method will open a socket to the server, send a String and receive a String in return and check that the returned String is the value it expected (using an JUnit assert).  The testFailure() method will just fail a JUnit assertion.

The SimpleServerTest is a basic JUnit test case that sets up a socket server within it's setUp() method (on a separate thread) which will accept requests from clients and respond with a static String.  The SimpleServerTest also has a test method, called testRequestCount(), which will simply print out the number of request its taken every 3 seconds.

This introduces one of the problems with JUnit testing from a client/server standpoint.  The client test needs the server to be available to accept requests for the duration of all it's tests.  The server, since it follows the conventional JUnit model, wants to set up, test, and then tear down. 

When the server and then the client tests are run, the testClientCall() method of SimpleClientTest will pass.  However, the testRequestCount() will never end and the tearDown() method will never be called.  To stop the SimpleServerTest from running, the process will have to be killed.  If the code is changed so the testRequestCount() method of SimpleServerTest does not loop, it is very possible that the test run of the SimpleServerTest would be finished before the SimpleClientTest would have a chance to call on it, thus causing the testClientCall() to fail.

decorated

Although this sample does not use the jrunit features directly, it does show the power of using the benchmark decorator.  This sample uses a slight variation of the basic SimpleClientTest and SimpleServerTest.  The SimpleClientTest was modified to declare a suite() method.  This method will return a new instance of the ThreadLocalDecorator class.  In this particular case, the parameters specify that the decorator should run the test case org.jboss.jrunit.sample.decorated.SimpleClientTest, using 3 separate threads, each looping 10 times, with no delay (see ThreadLocalDecorator section for more information on configuring the ThreadLocalDecorator).  This means that when the SimpleClientTest is run, there will be three instances created that run all the tests 10 times each. 

The testClientCall() method of the SimpleClientTest class also has some lines added to mark the starting and ending of named benchmarks.  In particular, the 'ClientCall' and the 'GotSocket' benchmark will be started at the beginning of the method.  The 'GotSocket' benchmark will be stopped after making the call the getSocket() method and the 'ClientCall' benchmark will be closed at the end of the method.

At this point the SimpleServerTest has not modified.  Therefore, based on the code, it will accept only one client socket connection at a time and process the requests for this connection.  After that connection is closed, it will move onto the next connection.

To run this sample, start the org.jboss.jrunit.sample.basic.decorated.SimpleServerTest and then the org.jboss.jrunit.sample.decorated.SimpleClientTest.  After the client has finished, stop the server and at the bottom of the client console, should see something similar to:

SubBenchmarks:
  Benchmark:ClientCall
  Executions:30
  Time:15311
  SubBenchmarks:
    Benchmark:GetSocket
    Executions:30
    Time:15191

This indicates how long it took for the three threads to run ten iterations of the testClientCall() and getSocket() methods. 

Now, uncomment the for loop in the setUp() method of the SimpleServerTest class.  This will make it so there are three threads available to accept incoming client socket request (please do not send me any complaints about how this is NOT how servers should be coded... it for demonstration purposes, relax ).  Compile and re-run.  You results should now look something like:

SubBenchmarks:
  Benchmark:ClientCall
  Executions:30
  Time:130
  SubBenchmarks:
    Benchmark:GetSocket
    Executions:30
    Time:30

These results indicate that just by changing the server to be multi threaded so that it could accept and process all the client requests concurrently the client calls executed over 100 times faster.   

clientserver

The next example, which is under org.jboss.jrunit.sample.clientserver package addresses how jrunit helps to solve these issues when running a client/server test within a JUnit construct.  The only change required to the original test classes is that the SimpleServerTest extends org.jboss.jrunit.ServerTestCase instead of junit.framework.TestCase, which will allow the behavior change needed within the server test run (see the section on ServerTestCase for more details on how this class changes the test run behavior).  Also changed the test to use log4j instead of System.out.println for logging since output will not be to console, more on why this is in a minute.

The other class within the org.jboss.jrunit.sample.clientserver package is the SampleClientServerTest.  This class extends the TestDriver class which provides the harness for running the client and server in different processes.  The main method of note is the declareTestClasses(), which must be implemented as is an abstract method within TestDriver.  This method then calls the addTestClasses() method from the TestDriver.  The parameters to the addTestClasses() method, in order, are the client test class to run, the number of clients processes to spawn, and the server test class to run.  Notice that the client specified is the original org.jboss.jrunit.sample.basic.SimpleClientTest class and the new org.jboss.jrunit.sample.clientserver.SimpleServerTest, which extends ServerTestCase.  There is also the getTestHarnessLogLevel() method, which tells the driver what the log level should be set to for the test harness code, for debugging purposes.

To run this sample, go to the root directory and run the ant target 'run-SampleClientServerTest'.  The console output should be similar to:

run-SampleClientServerTest:
    [junit] Running org.jboss.jrunit.sample.clientserver.SampleClientServerTest
    [junit] Tests run: 1, Failures: 2, Errors: 0, Time elapsed: 10.045 sec
    [junit] Test org.jboss.jrunit.sample.clientserver.SampleClientServerTest FAILED
[junitreport] Transform time: 471ms

This ant task will also create an html report under the output/test-report/ directory. 

Reviewing the results should indicate that there were two failures.  The first being the failure from the testFailure() method of the org.jboss.jrunit.sample.basic.SimpleClientTest class and the other being from the testRequestCount() method of the org.jboss.jrunit.sample.clientserver.SimpleServerTest (because only received one client call instead of 30).

multipleclientserver

This next sample is basically the as the previous sample except will run three client processes to call on the server instead just the one.  The only new class for this example is the org.jboss.jrunit.sample.multipleclientserver.SampleMultipleClientServerTest.  The only difference in this class and the SampleClientServerTest from before is that change the second parameter to addTestClassses() method from 1 to 3.

To run this sample, go to the root directory and run the ant target 'run-SampleMultipleClientServerTest'.  The console output should be similar to:

run-SampleMultipleClientServerTest:
    [junit] Running org.jboss.jrunit.sample.multipleclientserver.SampleMultipleClientServerTest
    [junit] Tests run: 1, Failures: 4, Errors: 0, Time elapsed: 11.577 sec
    [junit] Test org.jboss.jrunit.sample.multipleclientserver.SampleMultipleClientServerTest FAILED
[junitreport] Transform time: 511ms

This ant task will also create an html report under the output/test-report/ directory. 

Reviewing the results should indicate that there were now four failures.  The first three being the failure from the testFailure() method of the org.jboss.jrunit.sample.basic.SimpleClientTest class and the fourth being from the testRequestCount() method of the org.jboss.jrunit.sample.clientserver.SimpleServerTest (because only received one client call instead of 30).  This also means that each of the three SimpleClientTest's testClientCall() method executed successfully.

clientonly

It is also possible to just use the jrunit framework for running the client side only in the case where a server is already running.  The way to accomplish this, as shown in org.jboss.jrunit.sample.clientonly.SampleClientOnlyTest is to change the last parameter to the addTestClasses() method to be null.  To test this, run the org.jboss.jrunit.sample.basic.SimpleServerTest, then the org.jboss.jrunit.sample.clientonly.SampleClientOnlyTest.  Remember that the SimpleServerTest will have to be shutdown manually in this case.  The results should be the same as the previous example, except without the failure from the SimpleServerTest.

decoratedclientserver

This sample is exactly like the previous client/server examples from before, except the client is org.jboss.jrunit.sample.decoratedclientserver.SimpleDecoratedClientTest.  This class has the same code as the SimpleClientTest in the previous examples, but has added code for obtaining benchmark results in addition to the basic JUnit test results.  The main differences in code is a new suite() method has been added which creates a new ThreadLocalDecorator, specifing that client should loop running it's test 10 times.  There are also lines within the testClientCall() which specify when particular benchmark metrics, via static calls to ThreadLocalBenchmark,  should be started and stopped.  For more details on ThreadLocalBenchmark, please refer to the section on ThreadLocalBenchmark.

The SampleDecoratedClientServerTest class extends BenchmarkTestDriver instead of TestDriver.  This sets up the root test driver so that it will listen and report the remote benchmark results to the console and to a local file based on the test's class name (i.e. SampleDecoratedClientServerTest_benchmark.txt).