7 Replies Latest reply on Feb 7, 2010 3:04 AM by Tim Fox

    Perf relative to JBM?

    Zemian Deng Apprentice

      Hi list,

       

      Is there any expectation or reports on HornetQ 2.0.0GA performance in relative to JBoss 5.1.0.GA Messing?

       

      Would the following program produce any meaningful data to compare the two systems?

      1. Create a simple java JMS client

      2. In the client, create and start a thread with a JMS consumer on a ExampleQueue. Measure msg/sec on receive rate per certain interval.

      3. After consumer started, create a JMS procducer and send burst of a msgs (eg: 1000). Measure msg/sec on send rate per certain interval.

      4. Run client against standalone hornetq server, then on jboss default server. Collect the average/max rate handled by client on two systems for evaluation.

       

      Thanks,

      -Z

        • 1. Re: Perf relative to JBM?
          Zemian Deng Apprentice

          Sorry, I just see that similar perf questions have been asked already.

          http://community.jboss.org/thread/147037?tstart=0


          • 2. Re: Perf relative to JBM?
            Zemian Deng Apprentice

            Hi,

             

            I understand that simple test like one mentioned above or other posting are not a good measure of the messaging server, and it really just testing my own network bandwidth. Despite that, if I were to run it against the JBossing Messaging (in JBoss-5.1.0) vs HornetQ (with default settings), I ought to get similar throughput in respect to each other right?

             

            I am little confused when I observed that JBM has x10-13 times higher throughput rate than HornetQ-2.0.0.GA. Is there some obvious things that I should know when running a non-cluster standalone HornetQ vs the jboss default server?

             

            Much Thanks,

            -Z

            PS I read the user manual mentioned flow control, but default hornetq is not enabled though right. I also see the buffering window is default to 1MB, and that should only help in simple test with tiny text messages.

            • 3. Re: Perf relative to JBM?
              Tim Fox Master

              HornetQ is way faster than JBoss Messaging 1.x

               

              In fact, HornetQ is faster than any other messaging system we have tested with so far.

               

              We will be producing performance results in the next few weeks, and the methodology will be public so you can reproduce.

               

              I won't comment on your results as it's likely to be a problem with set-up, configuration or in your test itself.

              • 4. Re: Perf relative to JBM?
                Tim Fox Master

                BTW, if you compare "out of the box" JBM with "out of the box" HornetQ you will probably find JBM appears faster on a naive test that sends persistent messages one after another.

                 

                Why? Because JBM uses a database for persistence, and the default one is HSQL - which *does not sync to disk*.

                 

                HornetQ out of the box *does* make sure data is synched to disk.

                 

                Moreover, people often do silly things like run the benchmark on a laptop or cheap desktop which normally have disk write cache enabled. (See user manual perf tuning section)

                 

                Simple, ad hoc tests rarely give significant or reliable results. A good benchmark takes a lot of thought to setup and configure.

                 

                If you wanted to do a better comparison of JBM to Hornetq you would need to

                 

                a) run the benchmark on real server machines

                b) make sure disk write cache is disabled

                c) Make sure JBM is configured to use a *real* database (not HSQL), e.g. MySQL, Oracle etc

                d) Make sure the database itself is configured to sync to disk (some database *do not* by default do this)

                e) Design a good benchmark which does not just send one message after another

                f) Make sure servers and clients are tuned appropriately

                g) When taking the benchmark, make sure there is a long warmup period so JIT can warm up properly

                h) Run the benchmark for a significant amount of time to minimise non determinacy due to thread scheduling and garbage collection

                i) etc

                 

                You are much better off just waiting for us to release the results with corresponding framework that you can try yourself.

                • 5. Re: Perf relative to JBM?
                  Zemian Deng Apprentice

                  Ahh, write to disk! That is the likely cause then. Okay I see the reason now.

                   

                  Thank you much for the helpful hint.

                   

                  Yes I look forward to the benchmark result and test procedure you have.

                  • 6. Re: Perf relative to JBM?
                    Zemian Deng Apprentice

                    Tim,

                    I turned delivery mode to NON_PERSIST and see the expected result now. Thanks again.

                     

                    So I have a question on the suggestion you made on item e). Do you mean not to send one msg right after another in a loop? What is the down fall of sending a burst of msg during benchmarking (measuring) a msg flow rate?

                     

                    Thanks,

                    -Z

                    • 7. Re: Perf relative to JBM?
                      Tim Fox Master

                      saltnlight5 wrote:

                       

                      Tim,

                      I turned delivery mode to NON_PERSIST and see the expected result now. Thanks again.

                       

                      So I have a question on the suggestion you made on item e). Do you mean not to send one msg right after another in a loop? What is the down fall of sending a burst of msg during benchmarking (measuring) a msg flow rate?

                       

                      Thanks,

                      -Z

                      That kind of test isn't very useful. If you measure the performance of a single sender and consumer on a messaging system, then you might come to the conclusion that messaging system A is faster than messaging system B.


                      But that would be an invalid conclusion. Most set-ups have many connections, 100s, 1000s to the server not one. You may find that messaging system B is actually a lot more performant with large numbers of connections, i.e. it scales better and can sustain a higher overall throughput.