1 2 Previous Next 16 Replies Latest reply on Feb 20, 2006 8:34 PM by Clebert Suconic

    serialization performance across network

    Tim Fox Master

      Moved here from private email conversation:

      From: Tim Fox Sent: Thursday, January 26, 2006 2:11 PM
      >>> To: Tom Elrod; Clebert Suconic
      >>> Cc: Ovidiu Feodorov
      >>> Subject: JBoss serialization issue
      >>>
      >>> Hi all-
      >>>
      >>> I've been trying out JBoss Serialization with jboss messaging and
      >>> getting some strange results.
      >>>
      >>> When running the client and server on the same machine (different
      >>> VMs), I get about 10% performance increase over standard
      >>> serialization, which is great :)
      >>>
      >>> However when I run the client and server on different machines with a
      >>> 100MB/s network between them, the performance using jboss
      >>> serialization plummets to around 1% (one hundredth) of the
      >>> performance using standard serialization.
      >>>
      >>> This is all running JDK5, and I am using both remoting and serialization
      >>>
      >>> from their own CVS modules.
      >>>
      >>> I have verified the same behaviour on my home network and also in the QA
      >>>
      >>> lab.
      >>>
      >>> A further interesting observation is that if I use JDK1.4.2 I get the
      >>> same very low performance even when I use standard serialization as
      >>> well.
      >>>
      >>> The low performance only occurs if the traffic is over an actual
      >>> physical network.
      >>>
      >>> Any ideas?

        • 1. Re: serialization performance across network
          Tim Fox Master

          >> Clebert Suconic wrote:
          >>
          >>> How can I run your testcase?
          >>>
          >>> I would like to look at it.

          • 2. Re: serialization performance across network
            Tim Fox Master

            > Tim Fox wrote:
            >
            >> Currently this is observed with a jboss messaging client talking to a
            >> jboss messaging server on a remote machine.
            >>
            >> The client sends invocations via remoting using the socket transport
            >> and serializationtype=jboss.
            >>
            >> When I switch to serializationtype=java the performance increases by
            >> about 10000%
            >>
            >> I haven't actually isolated this yet in a smaller "test case" although
            >> I don't think I will be able to create a junit test that replicates
            >> this since the client and server must be on different boxes.
            >>
            >> One thing I would like to try is to start a remoting connector using
            >> the socket transport on box A, and fire invocations at it from box B
            >> and see if I can replicate it this way.
            >>
            >> If you already have a test setup that does this, it might save me some
            >> time.

            • 3. Re: serialization performance across network
              Tim Fox Master

              Tom Elrod wrote:

              Nothing out of the box for doing this. Can either take one of the samples and modify it (since will have a separate server and client class), or build off of org.jboss.test.remoting.performance.synchronous.PerformanceServerTest and org.jboss.test.remoting.performance.synchronous.PerformanceClientTest, but these will require some work as well to get setup.

              • 4. Re: serialization performance across network
                Clebert Suconic Master

                Default serialization (java.io.ObjectInputStream/ObjectOutputStream) have an internal Buffer, different from what JBossObjectOutputStream has.

                Maybe when using it over the network, some other behavior is happening. (just wondering)


                I would like to personally run your JMS tests. If you (Tim) could tell me how to do it, or to appoint me some documentation about it.


                Clebert

                • 5. Re: serialization performance across network
                  Tim Fox Master

                  Ok I have managed to replicate this in a trivial set-up.

                  On machine A, I have a simple main class which creates and starts two connectors:

                  socket://192.168.1.10:1234/?serializationtype=java
                  socket://192.168.1.10:1235/?serializationtype=jboss

                  Then, on machine B, we simply send instances of a very simple Serializable object (it just contains a string) of size approx. 1000 bytes, in a loop, using client.invoke.

                  We repeat this with client instances for each connector.

                  Results show:

                  $ ./runtestclient
                  FieldsManager in use = org.jboss.serial.classmetamodel.ReflectionFieldsManager
                  Invoking using java serialization
                  Rate: 201.8978396931153 msgs/sec
                  Invoking using jboss serialization
                  Rate: 4.976609933313427 msgs/sec
                  


                  I have sent the test code offline to Clebert and Tom.

                  I am using JDK5.

                  • 6. Re: serialization performance across network
                    Tim Fox Master

                    FWIW-

                    Looking at the code of JBossObjectOutputStream I notice it flushes the underlying stream after each writeObject.

                    If the underlying stream represents the output stream of a socket (or even of a bufferedoutputstream over a socket output stream), then it's my understanding that this will cause whatever's been written to be sent over the network.

                    This would mean we're sending at least one tcp packet per flush which could be very inefficient if we haven't written enough to fill a packet. ??

                    I would have though that calling flush should be the responsibility of the user of the OutputStream not the output stream itself. (I think this is the case with the standard Java ObjectOutputStream).

                    I'm not sure if this is the reason behing the performance problem, but it's a thought...

                    • 7. Re: serialization performance across network
                      Clebert Suconic Master

                      You were right about the flush.

                      But on the process, I profiled JBoss Serialization with JBoss Profiler and saw some improvements I could make with writeString.

                      I could get good number then just by removing flush, and I could get 40% more after enabling my writeUTF improvements.

                      This is actually a good thing for your consideration. The old version of JBossMQ used some writeUTF, and this version I'm writing is going to be much faster.


                      I didn't commit any code yet, as I need to consider some circular buffers and fine tune my testcases. I will have it done/ committed this week. Will ask you to test it then.


                      Thanks for your input. This thread was great, as it could produce nice improvements.


                      Clebert

                      • 8. Re: serialization performance across network
                        Tim Fox Master

                        Clebert-

                        Sounds great :)

                        We use writeUTF quite heavily so any improvements there would be great.

                        I look forward to trying it out.

                        • 9. Re: serialization performance across network
                          Clebert Suconic Master

                          I commited the code on CVS.


                          And I also attached your test (with some improvements, like build.xml and other facilities) at http://jira.jboss.org/jira/browse/JBSER-38

                          So, if you could validate if this fixes your problem.

                          BTW: the package I attached to JBSER-38 already has the latest version compiled from CVS for JBossSerialization.

                          • 10. Re: serialization performance across network
                            Tim Fox Master

                            Clebert-

                            I've updated from CVS, but I'm still getting the NPE (http://www.jboss.com/index.html?module=bb&op=viewtopic&t=76374) which prevents me from testing your latest optimisations.

                            • 11. Re: serialization performance across network
                              Clebert Suconic Master

                              I thought the NPE was a separated issue for you.

                              Then, I created separated issues. One for the NPE, and another one for the performance problem over the network. So, in summary I fixed only the flush thing.

                              I will see if I can fix it real quick. Will update the thread then.

                              • 12. Re: serialization performance across network
                                Tim Fox Master

                                Ok np.

                                I've fixed the NPE in my local copy (I just made sure the DataOutputStream is created in the JBossObjectOutputStream constructor) and everything looks fine.

                                Message rates across the network are looking good now :)

                                BTW Do you know when the changes will make it into CVS HEAD? We'd rather not have our own copies of dependent modules in CVS for too long.

                                Thx

                                • 13. Re: serialization performance across network
                                  Clebert Suconic Master

                                  Can you commit your NPE fix associated to JBSER-39 on the CVS comment? I will take a look then.

                                  I need to do some performance tests with AS, and I can release another release on repository.jboss.com. It shouldn't take long.

                                  1 2 Previous Next