Basic questions on JRUnit

[timfox]

Hello chaps

I'm looking at JRUnit with the view to seeing how it can help us with jms perf testing (as you know).

I have some very basic questions so pls be patient :)

If I have 3 machines, A, B C. From machine A, I want to say "run test T1 on machine B", "run test T2 on machine C". The tests should run and return some results to machine A. I then want to compute a benchmark based on those results and store it in persistent storage.

From the docs/samples I couldn't work out how to do this.

Where do I specify the hostnames for the machines?

Do I have to start some kind of daemon process on machines B and C so it can spawn test processes when required? If so, how.

Finally, how do I return results from the test method?

A simple end to end example would be great, if you have one.

As you gather I am clearly missing a big piece of the puzzle here, hopefully you can correct this....

Tim

[clebert.suconic]

We were talking about adding that capability on JRunit during JBoss World last week.

We have the infra structure for adding it, as we have a MessageBus in top of JGroups, and we also have the framework for showing values.


IMO, if you need to create that feature, you could create into JRunit as we have the basic framework. ;-)


Clebert

[timfox]

Clebert -

I'm confused - what functionality were you thinking of adding?

Are you saying that JRUnit does not currently have the ability to run tests on remote machines?

[clebert.suconic]

JRunit was designed thinking about remoting tests but we only implemented what we needed for MicroBenchmarks.

It's the goal of the framework having the hability of synchronize remote agents, but we don't have that feature as for now.

It shouldn't be hard to implement it, but we don't have it now.


Clebert

[timfox]

Hi Clebert-

Now I'm even more confused..

My understanding was that JRUnit is an extension to JUnit to allow it run remote tests.

If it doesn't actually run remote tests then how does this differ from straightforward JUnit?

Sorry for being slow, but I'm trying to get my head around what it does :(

-Tim

[tom.elrod]

So there are a couple of features that you need Tim that we don't currently have in JRunit. As Clebert mentioned, the current framework was built with a basic client/server test model in mind (but think we can modify to meet your needs).

So one thing that you need is some sort of agent running in a separate process that be controlled by a central driver. We don't have this, but would not be too difficult to create. Just need to talk through what this agent would be responsible for. My thinking is it would be a process that sits idle, then the driver would sent message for it to load a JUnit test case and execute it using the JUnit test runner (at least this would be the most convenient way since would product the JUnit test results that could be reported back to the driver).

The second thing would be a way for the driver to control multiple test instances. Currently the JRunit driver is only setup to control multiple instances of the same client type (same class, multiple instances) and a server instance. This would be easy enough to change, but would need to define a distinction between type of test (client or server) because will impact how the driver controls the test instance's life cycle.

The third thing, which we talked to Ovidiu about last week, is the ability to receive testing data (benchmarks, etc.) during runtime, so can monitor how the test instance is performing while being run. Also need a way to push configuration information back to the running test instance so it can be tuned based on that information. This is a more complicated issue and will require more effort (but think would be worth the investment).

Given all this, think would be worth chatting about in person (maybe via skype?) and then posting the results of the conversation here.

[tom.elrod]

Oh, and to answer you question about client/server, JRunit actually creates the new client & server instances for each test run. Then shuts those process down when test runs are completed. This is way the agent feature is required.

Technically, can have a server already running in a different process outside of JRunit, but is up to the client tests that get created to know how to access and no test results will be received by the driver for the server (nor will the driver attempt to controll, or even communicate with the server... is just there running).

[timfox]

So there are a couple of features that you need Tim that we don't currently have in JRunit. As Clebert mentioned, the current framework was built with a basic client/server test model in mind (but think we can modify to meet your needs).

So one thing that you need is some sort of agent running in a separate process that be controlled by a central driver. We don't have this, but would not be too difficult to create. Just need to talk through what this agent would be responsible for. My thinking is it would be a process that sits idle, then the driver would sent message for it to load a JUnit test case and execute it using the JUnit test runner (at least this would be the most convenient way since would product the JUnit test results that could be reported back to the driver).

Yes, this is pretty much how I currently do it. It's pretty much trivial (thanks to jboss remoting :) )

The second thing would be a way for the driver to control multiple test instances. Currently the JRunit driver is only setup to control multiple instances of the same client type (same class, multiple instances) and a server instance. This would be easy enough to change, but would need to define a distinction between type of test (client or server) because will impact how the driver controls the test instance's life cycle.



The third thing, which we talked to Ovidiu about last week, is the ability to receive testing data (benchmarks, etc.) during runtime, so can monitor how the test instance is performing while being run. Also need a way to push configuration information back to the running test instance so it can be tuned based on that information. This is a more complicated issue and will require more effort (but think would be worth the investment).

Actually I'm not sure I need this feature any more :)

Given all this, think would be worth chatting about in person (maybe via skype?) and then posting the results of the conversation here.

Yes. Given that I've already implemented the remote stuff without JRUnit, let's get together when I'm done and we can evaluate how to bring those features into JRUnit (if appropriate)

[tom.elrod]

Ok. Just let us know when you are ready.