6 Replies Latest reply on Jun 23, 2008 2:48 PM by clebert.suconic

    Interesting performance numbers with 2.0 alpha, 1.4 & AMQ 5.

    seattle.golfer

      So I'm pretty disenchanted with AMQ (4.1.1 & 5.1.0) at the moment, thanks to a number of pretty glaring bugs that we keep running into.

      While trying to figure out the best way to proceed, I ran the JBM 2.0 alpha performance tests (http://www.jboss.org/file-access/default/members/jbossmessaging/freezone/docs/userguide-2.0.0.alpha1/html/performance.html) against
      *JBM 1.4.0.SP3
      *JBM 2.0.0.alpha1
      *AMQ 5.1 with the amqPersistenceAdapter
      *AMQ 5.1 with the journaledJDBCPersistenceAdapter.

      Since all we use are topics with multiple durable subscriptions, my only change to the performance test was to use topics with durable subs instead of queues.

      I ran the tests with the options shown below because they most closely mimic our application's JMS usage. All of these are on my primary dev machine, a 2.2 GHz C2 Duo with 4 ghz of ram, 10.5.3, JDK 1.5.0_13. I terminated both AMQ tests after a while cause I got tired of waiting on this run.

      I was expecting to see JBM 2.0 top the list, but instead JBM 1.4 kicked some serious ass! Any developers care to speculate on why 1.4 would be so much better in this case?

      Just getting this out there cause it has me flummoxed :).

      Jboss 1.4
      ant perfSender -Dsess.trans=true -Dsess.trans.size=1 -Ddelivery.mode=PERSISTENT average: 1332.09 msg/s (200000 messages in 150.14s)
      ant perfListener -Dclient.id=one -Ddrain.queue=false -Dsess.trans=true -Dsess.trans.size=1 average: 1332.06 msg/s (200000 messages in 150.14s)
      ant perfListener -Dclient.id=two -Ddrain.queue=false -Dsess.trans=true -Dsess.trans.size=1 average: 1332.10 msg/s (200000 messages in 150.14s)

      Jboss 2.0 Alpha
      ant perfSender -Dsess.trans=true -Dsess.trans.size=1 -Ddelivery.mode=PERSISTENT average: 862.35 msg/s (200000 messages in 231.92s)
      ant perfListener -Dclient.id=one -Ddrain.queue=false -Dsess.trans=true -Dsess.trans.size=1 average: 850.65 msg/s (200000 messages in 235.11s)
      ant perfListener -Dclient.id=two -Ddrain.queue=false -Dsess.trans=true -Dsess.trans.size=1 average: 851.58 msg/s (200000 messages in 234.86s)

      AMQ 5.1 stock config with amqPersistenceStore
      ant perfSender -Dsess.trans=true -Dsess.trans.size=1 -Ddelivery.mode=PERSISTENT average: 400.74 msg/s (102000 messages in 254.53s)
      ant perfListener -Dclient.id=one -Ddrain.queue=false -Dsess.trans=true -Dsess.trans.size=1 average: 400.74 msg/s (102000 messages in 254.53s)
      ant perfListener -Dclient.id=two -Ddrain.queue=false -Dsess.trans=true -Dsess.trans.size=1 average: 400.74 msg/s (102000 messages in 254.53s)

      AMQ 5.1 our config with journaledJDBC
      ant perfSender -Dsess.trans=true -Dsess.trans.size=1 -Ddelivery.mode=PERSISTENT average: 13.04 msgs/s (8000 messages in 613.11s)
      ant perfListener -Dclient.id=one -Ddrain.queue=false -Dsess.trans=true -Dsess.trans.size=1 average: 13.05 msgs/s (8000 messages in 613.04s)
      ant perfListener -Dclient.id=two -Ddrain.queue=false -Dsess.trans=true -Dsess.trans.size=1 average: 13.05 msgs/s (8000 messages in 613.05s)

      Regards,
      Deepak.

        • 1. Re: Interesting performance numbers with 2.0 alpha, 1.4 & AM
          timfox

          Glad to hear you are looking at JBM 2.0.

          A couple of thoughts:

          1) I'd strongly recommend against running all tests on the same box. You'll probably find the results won't give you a real picture of performance since you really need a network in play (minimum 1GB network).

          2) For JBM 1.4 - are you using out of the box config? If so JBM 1.4 out of the box in the app server uses HSQLDB which isn't a "real" database - has not tx, and is in memory. It may give high perf figures because it's not actually persisting to disk. If you replace HSQLDB by a real db like Oracle or MySQL you'll probably see a difference in the perf.

          3) JBM 2.0 - I'd strongly recommend running the server on linux if you're not already and installing libaio (see userguide) - this will make the journal even faster.

          4) AMQ by default does not by default sync tx to disk - make sure you switch this on.

          5) I'd also try with larger transaction sizes. If you're using tx size of 1, there's little point you may as well just send a single persistent message outside a tx. The JBM journal will shine even more with larger or more real world transactions.



          • 2. Re: Interesting performance numbers with 2.0 alpha, 1.4 & AM
            clebert.suconic

            Is this Linux? Are you using libaio? What disk you have in your config?

            If you are using linux/libaio, when you add more clients you should scale even higher. I mean.

            Our target for BETA is to maximize your disk... say.. if your disk can do up to 20K messages a second on the disk, you would be able to add 20 more producers and you would still have 800/1K messages per second on each producer.

            • 3. Re: Interesting performance numbers with 2.0 alpha, 1.4 & AM
              clebert.suconic

               

              Our target for BETA is to maximize your disk... say.. if your disk can do up to 20K messages a second on the disk, you would be able to add 20 more producers and you would still have 800/1K messages per second on each producer.


              Of course I meant... 20 producers at the small transaction size.

              If you use higher transaction sizes, you would be pretty impressed with the rates you would achieve with a single producer.

              • 4. Re: Interesting performance numbers with 2.0 alpha, 1.4 & AM
                seattle.golfer

                 

                "timfox" wrote:


                1) I'd strongly recommend against running all tests on the same box. You'll probably find the results won't give you a real picture of performance since you really need a network in play (minimum 1GB network).



                We are designing each node in this role so it is horizontally scalable. What that means is that the JMS Broker runs on each node, and all message producers only publish messages to their local JMS broker. Message consumers can live on other nodes, and use a piece of custom (spring) code to perform the subscription multiplexing to all the topics they are interested in. (*)

                So, that is a long winded way of saying, on the producing side, network latency & bandwidth shouldn't matter, and on the receiving side, I'm not overly concerned because the producers are the only ones in the real time code flow.

                "timfox" wrote:


                2) For JBM 1.4 - are you using out of the box config? If so JBM 1.4 out of the box in the app server uses HSQLDB which isn't a "real" database - has not tx, and is in memory. It may give high perf figures because it's not actually persisting to disk. If you replace HSQLDB by a real db like Oracle or MySQL you'll probably see a difference in the perf.



                Yup, out of the box JBM 1.4 config. HSQL is fine with us because of the architecture I mentioned above, as long as HSQL is configured to persist to disk for durable storage across restarts (which it is by default it appears).

                "timfox" wrote:

                3) JBM 2.0 - I'd strongly recommend running the server on linux if you're not already and installing libaio (see userguide) - this will make the journal even faster.

                4) AMQ by default does not by default sync tx to disk - make sure you switch this on.

                5) I'd also try with larger transaction sizes. If you're using tx size of 1, there's little point you may as well just send a single persistent message outside a tx. The JBM journal will shine even more with larger or more real world transactions.


                Good info to have, thanks!


                "clebert.suconic@jboss.com" wrote:

                Is this Linux? Are you using libaio? What disk you have in your config?

                If you are using linux/libaio, when you add more clients you should scale even higher. I mean.

                This was on a MacBook, but all our servers run linux with a recent 2.6 kernel. So we should be able to utilize all the performance the ASYNCIO provides!

                "clebert.suconic@jboss.com" wrote:

                Our target for BETA is to maximize your disk... say.. if your disk can do up to 20K messages a second on the disk, you would be able to add 20 more producers and you would still have 800/1K messages per second on each producer.


                Is there a tentative date for the Beta or the Release?


                Thanks for the responses guys, much appreciated.

                I'm going to recommend we switch to JBM 1.4 from AMQ for now, and plan on upgrading to 2.0 once it is released. I



                (*) We designed things this way because AMQ's network of brokers feature didn't work as advertised, and JBM 1.4's clustering still required a central database. When we are ready to adopt 2.0, we'll look at using a distributed destination/clustering within JBM.



                • 5. Re: Interesting performance numbers with 2.0 alpha, 1.4 & AM
                  ataylor

                   

                  Yup, out of the box JBM 1.4 config. HSQL is fine with us because of the architecture I mentioned above, as long as HSQL is configured to persist to disk for durable storage across restarts (which it is by default it appears).


                  JBM 2.0.0 GA (and probably the Beta )will have a JDBC Journal implementation so you will be able to use HSQL if thats what you want to do.



                  Is there a tentative date for the Beta or the Release?


                  October time for Beta wilth the GA probably early next year.

                  • 6. Re: Interesting performance numbers with 2.0 alpha, 1.4 & AM
                    clebert.suconic

                     

                    Yup, out of the box JBM 1.4 config. HSQL is fine with us because of the architecture I mentioned above, as long as HSQL is configured to persist to disk for durable storage across restarts (which it is by default it appears).



                    I'm not sure if it's memory or disk, but on HSQL as transactions won't work, you won't have the same level of guarantee. Commit is not forcing a sync on HSQL files... so it is probably faster but it is also probably broken.


                    This was on a MacBook


                    I have heard the JVM on Mac has few performance issues with sockets.