remoting vs raw transport performance

[tom.elrod]

There was a jira issue logged (JBREM-195) that pointed out the performance difference between raw rmi calls and rmi calls using remoting. Although the performance is better now, it spurred me to write some tests to compare the difference (so far all my performance test have been comparing difference of transports *within* remoting). Can find the tests at http://anoncvs.forge.jboss.com:8080/changelog/JBoss/JBossRemoting?cs=MAIN:telrod:20051021163552 (the readme.txt explains them in more detail).

The thing is that the results do not make sense to me:

1st run

JBoss Remoting (rmi & java serialization) - 74116 milliseconds
JBoss Remoting (socket & java serialization) - 74888 milliseconds
JBoss Remoting (socket & jboss serialization) - 70041 milliseconds
RMI - 79064 milliseconds
Socket - 82869 milliseconds

2nd run

JBoss Remoting (rmi & java serialization) - 77141 milliseconds
JBoss Remoting (socket & java serialization) - 75078 milliseconds
JBoss Remoting (socket & jboss serialization) - 70772 milliseconds
RMI - 78563 milliseconds
Socket - 84491 milliseconds

Maybe I am too close to the code to write objective tests like this. Wondering if anyone would be willing to take a look and see where I might have missed something.

Thanks.

-Tom

[adrian.brock]

The test is too simple and lacking in diagnostics.
Where was the bottleneck (90% in the OS network layer?)
Is ~1 minute enough time to make "random" full GCs insignificant,
has the JIT even kicked in, etc.?

Making performance decisions on these kind of simple tests (which are subject
to arbitrary implementation changes across versions) is stupid.

It is also worthless without looking at concurrency.

I know from the JBossMQ tests that UIL2 is significantly faster than RMI, can't remember
the numbers off the top of my head? But that is from running unit tests (real world
jms usage), not simple benchmarks.

[tom.elrod]

OMG, who invided Adrian? ;)

As usual, you are correct. However, I don't want to take the time to write the raw transport counterpart tests for my remoting performance tests until I am sure I can write ones that are correctly represented. So I wrote these simple ones to verify my approach was valid, as would be a lot more wasted time if do the full performance testsuite for raw rmi and socket and they are flawed.

[starksm64]

So run for an hour on a loopback interface to make a better first pass on diffs, or was this run over a local loopback.

[tom.elrod]

Will just write the full fledge raw transport tests the duplicate the remoting performance tests (so can capture in nightly testsuite run via cruisecontrol and have jrunit benchmark reporting). Already did the rmi one.

Plan on running different scenarios (number of calls, payload size, etc.) on a reserved qa box to get some real numbers next week.