1 2 3 Previous Next 36 Replies Latest reply on Oct 18, 2009 9:02 AM by steffi

    HornetQ - low performance in real world scenarion?

    mreasy

      Our aim was to compare the performance of HornetQ against JBossMQ. To check, if we get at least the same performance as JBossMQ without the need to flood the database with short term data (Oracle: massive redolog / archivelog generation).

      We tested the following two combinations:

      1.) HornetQ (JBoss 4.05 with embedded HornetQ)
      2.) JBossMQ (JBoss 4.05 with embedded JBossMQ (connected to remote Oracle 11 database)

      We deployed a simple MBean that acts as MessageProducer.
      The following happens during execution:
      a.) load an external file (10KB)
      b.) setup jms connection/session (non-transacted, auto_acknowledge) + look-up queue
      c.) for each iteration:
      createObjectMessage
      .setObject(filecontent)
      send()
      d.) close jms resources

      We deployed a MessageDrivenBean that acts as MessageConsumer - plain simple onMessage method. Used only to clean up the message queue.

      We used the following HornetQ config:

       <journal-sync-transactional>true</journal-sync-transactional>
       <journal-sync-non-transactional>true</journal-sync-non-transactional>
       <journal-min-files>10</journal-min-files>
       <journal-compact-min-files>15</journal-compact-min-files>
      


      For reasons of transactional safety, we set "journal-sync=true". We assume, that "journal-sync=false" cannot be used on a production system.
      We only measured the throughput (message/sec) of the message sending and decided to use HornetQ from trunk as of 2009-10-14.
      Each run was sending 10.000 messages with about 10KB each.

      Platform: HornetQ (msg/sec) JBossMQ (msg/sec)
      ********************************************************************************
      WinXP 64bit 20 125
      - write cache is on
      HornetQ data store on local HD:
      [Hitachi Ultrastar 15K300
      (300GB, SAS, 3.5", 15000rpm, 16MB)]
      
      Solaris SPARC 35 125
      
      WinXP 64bit 98 125
      - write cache is on
      HornetQ data store on network drive
      
      Redhat Enterprise Linux 1200-1600 125
      AIO configured
      HornetQ data store on SAN
      
      Redhat with local data store I will post when ready
      


      We have to support Windows and various Unix platforms with our enterprise solution. It should perform well on all these platforms.
      Our results on the Windows platform with NIO are not acceptable for us, we expected them to be better using the file journal for persistence.
      The question is, if our test was invalid for some reason or if there is someone with better results and could explain how to achieve them.

      Doing some micro-benchmarking, revealed that waiting for the response using
      sendCondition.await(toWait, TimeUnit.MILLISECONDS);
      in org.hornetq.core.remoting.impl.ChannelImpl.sendBlocking(Packet) is taking a significant amount of time. Also it seems that this needs to get called multiple times per send. Maybe this may serve as a hint.
      Index: src/main/org/hornetq/core/remoting/impl/ChannelImpl.java
      ===================================================================
      --- src/main/org/hornetq/core/remoting/impl/ChannelImpl.java (revision 8107)
      +++ src/main/org/hornetq/core/remoting/impl/ChannelImpl.java (working copy)
      @@ -262,6 +262,7 @@
      
       long start = System.currentTimeMillis();
      
      + final long responseStart = System.nanoTime();
       while (response == null && toWait > 0)
       {
       try
      @@ -283,6 +284,12 @@
      
       start = now;
       }
      + final long responseDuration = System.nanoTime() - responseStart;
      + if (responseDuration > 50 * 1000000)
      + {
      + System.out.println("Response:\t" + responseDuration);
      + }
      +
      
       if (response == null)
       {
      


      --
      Rico Neubauer

        • 1. Re: HornetQ - low performance in real world scenarion?
          mreasy

          addendum: tested also with latest BETA and some trunk-versions in-between - no difference.

          • 2. Re: HornetQ - low performance in real world scenarion?
            ataylor

            How are you embedding HornetQ with jboss 4.x, are you using InVM connectors instead of Netty Connectors. If not that may be the cause.

            Theres currently an outstanding JIRA for this, see https://jira.jboss.org/jira/browse/HORNETQ-146.

            • 3. Re: HornetQ - low performance in real world scenarion?
              mreasy

               

              "ataylor" wrote:
              How are you embedding HornetQ with jboss 4.x, are you using InVM connectors instead of Netty Connectors. If not that may be the cause.
              (...)


              InVM is in use.
              Integration is similar to that one, which is currently in progress in the HornetQ project - there was already some discussion about this integration between Tom, Clebert, a colleague and me. In my opinion 1.) it shouldn't matter, important is that invm-connector is used 2.) yes it's done correctly.

              • 4. Re: HornetQ - low performance in real world scenarion?
                mreasy

                err: maybe it was you (Andy) instead of Clebert - sorry don't remember

                • 5. Re: HornetQ - low performance in real world scenarion?
                  timfox

                  I will take a look this, I suspect there is something wrong in your test program / config.

                  But also, please bear in mind that benchmarking of TRUNK is pretty meaningless. TRUNK can be in any kind of crazy state.

                  Benchmarking a beta is also pretty meaningless since we don't do performance optimisations until GA.

                  But, in any case, the windows results need to be investigated.

                  Other observations-

                  Don't use ObjectMessage - ObjectMessage is the slowest JMS message type you can use - see performance section in user manual

                  If you are sending persistent messages and blocking on send, then your code will be spending most of it's time waiting for the response from the send to come back from the server.

                  This is what it is dong here:

                  sendCondition.await(toWait, TimeUnit.MILLISECONDS);
                  


                  It's completely normal.

                  So summary, please don't make performance expectations from TRUNK, or even a beta.

                  For a GA release, yes, it would be an issue. But thanks for bringing it to our attention.

                  • 6. Re: HornetQ - low performance in real world scenarion?
                    timfox

                    Ah, so you were the guy "solar_sea" who did his own integration into AS 4.0?

                    That explains a lot ;)

                    Like we mentioned before, we have a developer currently doing this integration, and that's the one we'll be shipping with HornetQ.

                    We have no idea how you have done this, or if you have done it in the correct way.

                    • 7. Re: HornetQ - low performance in real world scenarion?
                      mreasy

                      Thanks for your replies.
                      No, that's not me, but a colleague. - Yes, I mentioned that discussion above.

                      From my pov our integration is perfectly ok - don't insist of you believing me of course ;)

                      We will re-test with your integration, but as said above, I don't see this as relevant.

                      TRUNK is undefined, that's clear; beta is old and no more defined, GA is not yet available - anything else we could use?

                      Usages of ObjectMessage and the waits for the responses are clear - but that's the pre-defined use-case.
                      Do you have benchmarks which show another trend regarding the comparison of sync vs. async? Would be interested in you configuration then.

                      thx, Regards,
                      Rico Neubauer

                      • 8. Re: HornetQ - low performance in real world scenarion?
                        timfox

                         

                        "MrEasy" wrote:

                        TRUNK is undefined, that's clear; beta is old and no more defined, GA is not yet available - anything else we could use?


                        No, I'm not really sure what you're expecting.. a beta is a beta, and after betas come a GA. Seems like you're expecting a beta to be production ready - that's a major misunderstanding of what a beta is.

                        No one will take a benchmark using a beta seriously.

                        But as I say, we will look at the performance on windows when we get around to performance optimisation which will occur sometime soon before we ship the GA.

                        • 9. Re: HornetQ - low performance in real world scenarion?
                          mreasy

                          Don't get me wrong, neither do I not HornetQ to be ready for production, nor do I want to gripe about it, but in case there are totally different results from you, please share your setup.
                          If you say that this has not been looked into yet, then that's also fine with me, I'll wait.

                          • 10. Re: HornetQ - low performance in real world scenarion?
                            timfox

                            I just did a very simple test using the core perf example in the distro (you could repeat this too).

                            I set it to send 1K messages with block-on-persistent-send and sync-non-transactional both set to true.

                            So basically for every message it's sent, it gets persisted on the server, it gets sync'd to disk, then a response is sent back and when the client gets the response it sends the next one.

                            On my cheap dell laptop (core 2 duo, 2GHz with 2MiB RAM) I can do about 800 msgs/sec with NIO and about 1050 with AIO, this is running on Fedora 11.

                            This sounds about right, since my crappy disk can probably do no more than about 1000 syncs per sec. A very fast server disk might be able to do about 2000.

                            If you set everything to blocking then your limit (with NIO) is how many syncs per sec your disk can do.

                            • 11. Re: HornetQ - low performance in real world scenarion?
                              mreasy

                              This sounds good - a 20% decrease when not using AIO is reasonable and acceptable.
                              Will do the tests as discussed in IRC and have a look at the core perf example tomorrow.
                              --
                              Rico Neubauer

                              • 12. Re: HornetQ - low performance in real world scenarion?
                                mreasy

                                Hi, ran some tests with the core perf example, the same way you did (I hope, see configuration extract below).
                                Nevertheless my results are way worse than yours.

                                block-ack=true
                                average: 71.85 msg/s (10000 messages in 139.18s)
                                block-ack=false
                                average: 71.50 msg/s (10000 messages in 139.87s)

                                Also tested the same using the in-VM-acceptor by fumbling the tests, which increased throughput by about 7%.

                                Could you cross-check my configuration stated below with yours for any differences and in case there are none: any idea why my throughput is at about 10% of your 'cheap dell laptop'?

                                Environment: Sun Fire V440, Solaris, local hard-disk.
                                Did some add. tests on an office PC with Win, which showed about 40% of the above throughput (expected due to hardware)
                                My configuration: default settings, besides the following:

                                hornetq-configuration.xml: <journal-sync-non-transactional>true
                                num-messages=10000
                                num-warmup-messages=100
                                block-ack=true or false depending upon test
                                block-persistent=true
                                


                                • 13. Re: HornetQ - low performance in real world scenarion?
                                  timfox

                                  Please post you entire properties file, also your hornetq-configuration.xml file

                                  Also please can you specify which example you run, exactly how you ran it, and what extra setup you did.

                                  • 14. Re: HornetQ - low performance in real world scenarion?
                                    timfox

                                    I'd like to know the exact, step by step, set of actions you did - it may sound pedantic but it's often in the details where the errors creep in.

                                    1 2 3 Previous Next