How test cases should work

[tom.elrod]

To start, since this is probably going to be a long thread, will state how the test cases work using the benchmark decorators.

ThreadLocalDecorator (and all the other decorators that accept number of threads and loops) - for each thread specified in the number of threads, there will be a new instance of the test created. For the number of loops specified, the test methods will be called on each test instance. So for example, if specify 3 threads and 10 loops, there will be three instances of the test class created and each instance will have their test methods called 10 times by each of their respective threads. An example class that demonstrates this would be:

public class SimpleThreadLoopCounter extends TestCase
{
private static int staticCounter = 0;
private static int staticMethodCounter = 0;
private int localCounter = 0;

public static Test suite()
{
return new ThreadLocalDecorator(SimpleThreadLoopCounter.class, 3, 10, 0, true, true);
}

public SimpleThreadLoopCounter()
{
staticCounter++;
}

public void testCounter() throws Exception
{
System.out.println("staticCounter = " + staticCounter);
System.out.println("staticMethodcounter = " + ++staticMethodCounter);
System.out.println("localCounter = " + ++localCounter);
}
}

When this is run, the last entry will be:

staticCounter = 3
staticMethodcounter = 30
localCounter = 10

Also, important to note that JUnit will create a new instance of the TestCase for each test method it calls. So using the same sample above, if copied the testCounter method and called it testCounter2, the output would be:

staticCounter = 6
staticMethodcounter = 60
localCounter = 10

This means that eventhough local variables can be reused for each loop, they will be thrown away for each new test method run.

Having said all this will follow up with another post of where, if anywhere, this behavior should be changed.

[tom.elrod]

I think that they way things work in the previous post should remain and would be a natural extension of the base JUnit TestCase.

However, I think there should be some changes to this behavior if running a TestCase that is specifically a server TestCase (when talking about having client/server tests where there are two process running for a test case, which is what I want for the remoting tests).

So, for server test cases...

- Need a way to tell the server to start up and initialize. This should be done using setUp() method. Once this method returns, it is assumed that the server is ready to receive calls.
- Need a way to tell the server that it can shutdown and clean up. This should be done using the tearDown() method. This method will only be called in the case that all clients are finished making their calls to the server.

The server may have test methods, which will be called just as in the case of regular JUnit test runs. However, the setUp() and tearDown() will not be called prior and after each test method call; it will be called only once each per test run. These test methods can contain asserts and are suggested to be used for validation of server data and metrics.

IMPORTANT - the tearDown() method may be called even while a test method is being run. This is intentional and allows for the test method to loop until the tearDown() method is called. So a possible example of where this could be used is:

...
private boolean stop = false;

public void testServerMetrics()
{
while(!stop)
{
// collect data here
}
}

protected void tearDown()
{
stop = true; // so will cause testServerMetrics() to break out of loop

// do shutdown and clean up code.
}

Also, another change to the default JUnit test case behavior is that there will NOT be a new instance of the server test created when it moves to the next test method. This is due to not knowing where the clients will be in their test run, so can't tear down a server test case instance and build up a new one just to run a new test method.

All this is just design talk at this point and I have to actually write the code to make it work this way (if possible). Wanted to post this so can field any objections before I go through the effort.

-Tom

[tom.elrod]

Also, another change to the default JUnit test case behavior is that there will NOT be a new instance of the server test created when it moves to the next test method. This is due to not knowing where the clients will be in their test run, so can't tear down a server test case instance and build up a new one just to run a new test method.

So strike this. Going to be close to impossible to do this without rewriting JUnit. The TestSuite's private method addTestMethods() calls on public static method createTest() which will contruct a new TestCase instance and set the test method on it and puts in a collection for each test method. This collection is iterated at a later point and runs each TestCase (as a seperate instance).

So will just doument the hell out of server test case saying that if have more than one test method, will have more than one instance of the server created. Imagine this will reduce mis-use by about 10%. ;)