serialization performance across network

[timfox]

Moved here from private email conversation:

From: Tim Fox Sent: Thursday, January 26, 2006 2:11 PM
>>> To: Tom Elrod; Clebert Suconic
>>> Cc: Ovidiu Feodorov
>>> Subject: JBoss serialization issue
>>>
>>> Hi all-
>>>
>>> I've been trying out JBoss Serialization with jboss messaging and
>>> getting some strange results.
>>>
>>> When running the client and server on the same machine (different
>>> VMs), I get about 10% performance increase over standard
>>> serialization, which is great :)
>>>
>>> However when I run the client and server on different machines with a
>>> 100MB/s network between them, the performance using jboss
>>> serialization plummets to around 1% (one hundredth) of the
>>> performance using standard serialization.
>>>
>>> This is all running JDK5, and I am using both remoting and serialization
>>>
>>> from their own CVS modules.
>>>
>>> I have verified the same behaviour on my home network and also in the QA
>>>
>>> lab.
>>>
>>> A further interesting observation is that if I use JDK1.4.2 I get the
>>> same very low performance even when I use standard serialization as
>>> well.
>>>
>>> The low performance only occurs if the traffic is over an actual
>>> physical network.
>>>
>>> Any ideas?

[timfox]

>> Clebert Suconic wrote:
>>
>>> How can I run your testcase?
>>>
>>> I would like to look at it.

[timfox]

> Tim Fox wrote:
>
>> Currently this is observed with a jboss messaging client talking to a
>> jboss messaging server on a remote machine.
>>
>> The client sends invocations via remoting using the socket transport
>> and serializationtype=jboss.
>>
>> When I switch to serializationtype=java the performance increases by
>> about 10000%
>>
>> I haven't actually isolated this yet in a smaller "test case" although
>> I don't think I will be able to create a junit test that replicates
>> this since the client and server must be on different boxes.
>>
>> One thing I would like to try is to start a remoting connector using
>> the socket transport on box A, and fire invocations at it from box B
>> and see if I can replicate it this way.
>>
>> If you already have a test setup that does this, it might save me some
>> time.

[timfox]

Tom Elrod wrote:

Nothing out of the box for doing this. Can either take one of the samples and modify it (since will have a separate server and client class), or build off of org.jboss.test.remoting.performance.synchronous.PerformanceServerTest and org.jboss.test.remoting.performance.synchronous.PerformanceClientTest, but these will require some work as well to get setup.

[clebert.suconic]

Default serialization (java.io.ObjectInputStream/ObjectOutputStream) have an internal Buffer, different from what JBossObjectOutputStream has.

Maybe when using it over the network, some other behavior is happening. (just wondering)


I would like to personally run your JMS tests. If you (Tim) could tell me how to do it, or to appoint me some documentation about it.


Clebert

[timfox]

Ok I have managed to replicate this in a trivial set-up.

On machine A, I have a simple main class which creates and starts two connectors:

socket://192.168.1.10:1234/?serializationtype=java
socket://192.168.1.10:1235/?serializationtype=jboss

Then, on machine B, we simply send instances of a very simple Serializable object (it just contains a string) of size approx. 1000 bytes, in a loop, using client.invoke.

We repeat this with client instances for each connector.

Results show:

$ ./runtestclient
FieldsManager in use = org.jboss.serial.classmetamodel.ReflectionFieldsManager
Invoking using java serialization
Rate: 201.8978396931153 msgs/sec
Invoking using jboss serialization
Rate: 4.976609933313427 msgs/sec

I have sent the test code offline to Clebert and Tom.

I am using JDK5.

[timfox]

FWIW-

Looking at the code of JBossObjectOutputStream I notice it flushes the underlying stream after each writeObject.

If the underlying stream represents the output stream of a socket (or even of a bufferedoutputstream over a socket output stream), then it's my understanding that this will cause whatever's been written to be sent over the network.

This would mean we're sending at least one tcp packet per flush which could be very inefficient if we haven't written enough to fill a packet. ??

I would have though that calling flush should be the responsibility of the user of the OutputStream not the output stream itself. (I think this is the case with the standard Java ObjectOutputStream).

I'm not sure if this is the reason behing the performance problem, but it's a thought...

[clebert.suconic]

You were right about the flush.

But on the process, I profiled JBoss Serialization with JBoss Profiler and saw some improvements I could make with writeString.

I could get good number then just by removing flush, and I could get 40% more after enabling my writeUTF improvements.

This is actually a good thing for your consideration. The old version of JBossMQ used some writeUTF, and this version I'm writing is going to be much faster.


I didn't commit any code yet, as I need to consider some circular buffers and fine tune my testcases. I will have it done/ committed this week. Will ask you to test it then.


Thanks for your input. This thread was great, as it could produce nice improvements.


Clebert

[timfox]

Clebert-

Sounds great :)

We use writeUTF quite heavily so any improvements there would be great.

I look forward to trying it out.

[clebert.suconic]

I commited the code on CVS.


And I also attached your test (with some improvements, like build.xml and other facilities) at http://jira.jboss.org/jira/browse/JBSER-38

So, if you could validate if this fixes your problem.

BTW: the package I attached to JBSER-38 already has the latest version compiled from CVS for JBossSerialization.

[timfox]

Clebert-

I've updated from CVS, but I'm still getting the NPE (http://www.jboss.com/index.html?module=bb&op=viewtopic&t=76374) which prevents me from testing your latest optimisations.

[clebert.suconic]

I thought the NPE was a separated issue for you.

Then, I created separated issues. One for the NPE, and another one for the performance problem over the network. So, in summary I fixed only the flush thing.

I will see if I can fix it real quick. Will update the thread then.

[timfox]

Ok np.

I've fixed the NPE in my local copy (I just made sure the DataOutputStream is created in the JBossObjectOutputStream constructor) and everything looks fine.

Message rates across the network are looking good now :)

BTW Do you know when the changes will make it into CVS HEAD? We'd rather not have our own copies of dependent modules in CVS for too long.

Thx

[clebert.suconic]

Can you commit your NPE fix associated to JBSER-39 on the CVS comment? I will take a look then.

I need to do some performance tests with AS, and I can release another release on repository.jboss.com. It shouldn't take long.

[timfox]

Done