11 Replies Latest reply on Jun 22, 2008 11:25 PM by gary.c.chen

    I got some performance problem while sending a lot of messag

    gary.c.chen

      Hi,

      I am now attempting to use JBM as a JMS Server on my application.
      I did a performance test on it using the JBossJMSNewPerformanceBenchmark framework,
      but the result was so difference from the performance result that we saw on the development wiki.

      My test environment and executors setup were as follow :

      1. I was uesing two machines to achieve this test, one had a JBoss AS running on it (machine A),
      another was deployed a MySql database server (machine B), I was using it as the JBM persistence store.
      I deployed the PerformanceBenchmark framework JBM executor (the rmi-executor-scoped.sar file) as a MBean on JBoss AS (machine A).
      And I runned the control & report tool (using 'ant run') on machine B.And all the executor and JBoss AS configurations were using
      default parameters except the JVM option using for starting JBoss AS.I setted min and max VM memory to 1024m.
      The two machines were all using Sun JDK 1.5_15.

      2. My first test config was as follow :

       <default-executor-name>REMOTE</default-executor-name>
       <destination>/queue/testQueue</destination>
       <connection-factory>/ConnectionFactory</connection-factory>
       <acknowledgment-mode>AUTO_ACKNOWLEDGE</acknowledgment-mode>
       <delivery-mode>NON_PERSISTENT</delivery-mode>
       <message-factory-class>org.jboss.jms.perf.framework.factories.MessageMessageFactory</message-factory-class>
       <message-size>100</message-size>
      
       <!-- safeguard for receive jobs that block waiting messages will never arrive -->
       <duration>600000</duration>
      
       <providers>
       <provider name="JBossMessaging">
       <factory>org.jnp.interfaces.NamingContextFactory</factory>
       <url>jnp://150.236.80.206:1099</url>
       <pkg>org.jboss.naming:org.jnp.interfaces</pkg>
       <executor name="REMOTE" url="rmi://remote-messaging"/>
       <executor name="REMOTE2" url="rmi://remote-messaging2"/>
       <executor name="COLOCATED" url="rmi://local-messaging"/>
       <executor name="COLOCATED2" url="rmi://local-messaging2"/>
       <color>255,0,0</color>
       </provider>
       </providers>
      
       <performance-tests>
       <performance-test name="Throughput-INVM">
      
       <graph>
       <x job="send" result="yes" metric="rate" label="measured send rate (msg/s)"/>
       <y job="receive" result="yes" metric="rate" label="measured receive rate (msg/s)" />
       </graph>
      
       <destination>/queue/simpleQueue</destination>
       <delivery-mode>NON_PERSISTENT</delivery-mode>
       <message-size>200</message-size>
       <duration>3600000</duration>
      
       <drain/>
       <parallel>
       <send rate="1000" executor="COLOCATED"/>
       <receive executor="COLOCATED2"/>
       </parallel>
      
       <drain/>
      
       <parallel>
       <send rate="2000" executor="COLOCATED"/>
       <receive executor="COLOCATED2"/>
       </parallel>
      
       <drain/>
      
       <parallel>
       <send rate="3000" executor="COLOCATED"/>
       <receive executor="COLOCATED2"/>
       </parallel>
      
       <drain/>
      
       <parallel>
       <send rate="4000" executor="COLOCATED"/>
       <receive executor="COLOCATED2"/>
       </parallel>
      
       <drain/>
      
       <parallel>
       <send rate="5000" executor="COLOCATED"/>
       <receive executor="COLOCATED2"/>
       </parallel>
      
       <drain/>
      
       <parallel>
       <send rate="6000" executor="COLOCATED"/>
       <receive executor="COLOCATED2"/>
       </parallel>
      
      
       <drain/>
       <execution provider="JBossMessaging"/>
       <!--<execution provider="JBossMQ"/>-->
      
       </performance-test>
       </performance-tests>
      
      


      The test runned an hour per SEND JOB so I thought I could see the performance after JBM went to
      a steady status and the result was as follow :

      
      Execution 1 (provider JBossMessaging)
      0. DRAIN JOB drained 0 messages
      1. PARALLEL
       SEND JOB sent 3360753 messages in 3601210 ms with a target rate 1000 messages/sec, real rate 933.23 messages/sec
       RECEIVE JOB received 50753 messages in 3600007 ms with a target rate 0 messages/sec, real rate 14.1 messages/sec
      2. DRAIN JOB drained 2000 messages
      3. PARALLEL
       SEND JOB sent 1550066 messages in 3601180 ms with a target rate 2000 messages/sec, real rate 430.43 messages/sec
       RECEIVE JOB received 2276000 messages in 3600007 ms with a target rate 0 messages/sec, real rate 632.22 messages/sec
      4. DRAIN JOB drained 2000 messages
      5. PARALLEL
       SEND JOB sent 1556184 messages in 3603317 ms with a target rate 3000 messages/sec, real rate 431.88 messages/sec
       RECEIVE JOB received 2278000 messages in 3600001 ms with a target rate 0 messages/sec, real rate 632.78 messages/sec
      6. DRAIN JOB drained 2000 messages
      7. PARALLEL
       SEND JOB sent 1732609 messages in 3601484 ms with a target rate 4000 messages/sec, real rate 481.08 messages/sec
       RECEIVE JOB received 2026000 messages in 3600001 ms with a target rate 0 messages/sec, real rate 562.78 messages/sec
      8. DRAIN JOB drained 2000 messages
      9. PARALLEL
       SEND JOB sent 1601025 messages in 3600575 ms with a target rate 5000 messages/sec, real rate 444.66 messages/sec
       RECEIVE JOB received 2240000 messages in 3600010 ms with a target rate 0 messages/sec, real rate 622.22 messages/sec
      10. DRAIN JOB drained 2000 messages
      11. PARALLEL
       SEND JOB sent 1593108 messages in 3601466 ms with a target rate 6000 messages/sec, real rate 442.35 messages/sec
       RECEIVE JOB received 2282000 messages in 3600001 ms with a target rate 0 messages/sec, real rate 633.89 messages/sec
      12. DRAIN JOB drained 2000 messages
      
      


      So I think the performance under this case was so poor.

      And then I run a couple of testes using a config as follow :

       <duration>30000</duration>
      


      and the others parameter were the same as above.
      But I got the results which send rate VS receive rate were so unsteady.
      for example some time I got a value as follow :

      
      SEND JOB sent 119970 messages in 30002 ms with a target rate 4000 messages/sec, real rate 3998.73 messages/sec
      RECEIVE JOB received 322885 messages in 30014 ms with a target rate 0 messages/sec, real rate 10757.81 messages/sec
      
      


      and some time I got a value as follow :

      
      SEND JOB sent 119978 messages in 30004 ms with a target rate 4000 messages/sec, real rate 3998.73 messages/sec
      RECEIVE JOB received 119978 messages in 30016 ms with a target rate 0 messages/sec, real rate 3997.13 messages/sec
      
      


      and event the recevie rate change trend was so difference between testes,
      some time the recevie rate got down at the point of send rate 3998.73 messages/sec, and went to 4998.43 messages/sec at the point of
      send rate 4997.1 messages/sec and 5995.9 messages/sec at the point of send rate 5998.3 messages/sec,
      some time it still went up and never got down.

      So I have some question :

      1. Why the performance was so poor in the first test, am I having any problem ?
      2. If the duration parameter set to 30 sec, can I get a steady performance result ?
      3. Why the receive rate was so unsteady in the second test ?

      Thanks.

        • 1. Re: I got some performance problem while sending a lot of me
          timfox

          I can't really tell why you get performance without looking at your setup.

          Take a look at these though - you can reproduce them too.

          http://www.jboss.org/file-access/default/members/jbossmessaging/freezone/docs/userguide-2.0.0.alpha1/html/performance.html#performance.results

          • 2. Re: I got some performance problem while sending a lot of me
            gary.c.chen

            JBM version: JBM 1.4.0.SP3

            Machine A setup :

            CPU: 2 * Dual-Core AMD Opteron(tm) Processor 2220
            Memory: 8,388,148K

            Filesystem size used avail capacity Mounted on
            /dev/dsk/c3t0d0s0 61G 16G 44G 27% /
            /devices 0K 0K 0K 0% /devices
            ctfs 0K 0K 0K 0% /system/contract
            proc 0K 0K 0K 0% /proc
            mnttab 0K 0K 0K 0% /etc/mnttab
            swap 11G 860K 11G 1% /etc/svc/volatile
            objfs 0K 0K 0K 0% /system/object
            /usr/lib/libc/libc_hwcap2.so.1
            61G 16G 44G 27% /lib/libc.so.1
            fd 0K 0K 0K 0% /dev/fd
            swap 11G 104K 11G 1% /tmp
            swap 11G 28K 11G 1% /var/run
            /dev/dsk/c3t2d0s0 67G 2.5G 64G 4% /opt/drutt

            JBoss AS starting argument:

            bash-3.00# pargs 25925
            25925: /opt/drutt/local/java/bin/java -Dprogram.name=run.sh -server -verbose:gc -Xms10
            argv[0]: /opt/drutt/local/java/bin/java
            argv[1]: -Dprogram.name=run.sh
            argv[2]: -server
            argv[3]: -verbose:gc
            argv[4]: -Xms1024m
            argv[5]: -Xmx1024m
            argv[6]: -Dsun.rmi.dgc.client.gcInterval=3600000
            argv[7]: -Dsun.rmi.dgc.server.gcInterval=3600000
            argv[8]: -Djava.endorsed.dirs=/export/home/drutt/jboss-4.2.2.GA-4Linux/lib/endorsed
            argv[9]: -classpath
            argv[10]: /export/home/drutt/jboss-4.2.2.GA-4Linux/bin/run.jar:/opt/drutt/local/java/lib/tools.jar
            argv[11]: org.jboss.Main
            argv[12]: -b
            argv[13]: 150.236.80.206
            argv[14]: -c
            argv[15]: messaging

            Network setup:

            100M network

            Machine B setup:

            CPU: 1 * Intel Core 2 CPU T7200 2.0GHz
            Memory: 3,538,284K

            All JBM and JBoss AS config parameter using default value.

            • 3. Re: I got some performance problem while sending a lot of me
              timfox

              Two obvious things:

              1) Database is not colocated with server
              2) 100Mb network is *slow* - remember you have all your database traffic going over that too. I would recommend a 1GB network and that's just for the client<->server traffic (not db traffic)
              3) The old perf framework is deprecated - we don't use it any more.

              • 4. Re: I got some performance problem while sending a lot of me
                gary.c.chen

                As you said, Database that persistent manager processes is bottleneck. Is there any method can improve performance of the traffic between jms queue and persistent db? Exactly, we tried to use batch mode for persistent manager. But I don't find any option to alternate the batch size. Finally, I want to know if the persistent action blocks jms queue receiving and writing?

                • 5. Re: I got some performance problem while sending a lot of me
                  gary.c.chen

                  Another question :
                  How can I turn off the persistent while sending and receiving messages ? That means all the messages sended to the queues are just store in memory. regardless PERSISTENT message or NON_PERSISTENT message.

                  • 6. Re: I got some performance problem while sending a lot of me
                    timfox

                     

                    "gary.c.chen" wrote:
                    As you said, Database that persistent manager processes is bottleneck.


                    Actually I said the *network* would be a bottleneck, but yes the database would probably be a bottleneck too.


                    Is there any method can improve performance of the traffic between jms queue and persistent db?


                    The connection to the database is handled by the database product not by JBM.


                    • 7. Re: I got some performance problem while sending a lot of me
                      timfox

                       

                      "gary.c.chen" wrote:
                      Another question :
                      How can I turn off the persistent while sending and receiving messages ? That means all the messages sended to the queues are just store in memory. regardless PERSISTENT message or NON_PERSISTENT message.


                      There has been a recent thread on this very subject, please have a search of the forums :)

                      • 8. Re: I got some performance problem while sending a lot of me
                        gary.c.chen

                        Yes, I got it, thanks fox.

                        • 9. Re: I got some performance problem while sending a lot of me
                          gary.c.chen

                          Afterward we setted up another test, this time we setted the parameter to ten minutes,
                          and we setted the FullSize parameter, PageSize parameter, DownCacheSize parameter of the queue to be tested
                          to 3355444, 10000, and 10000, and the messages we sended in this test were all NON_PERSISTENT messages, so this
                          time all the messages that sended to the queue were all stored in the memory. We also checked that the
                          persistent database had never been writed or readed.

                          In this test case we saw that messages send rate was very fast, almost equaled to it's
                          target send rate, but the messages received by the consumer were so a little, almost equaled to 1/10
                          of the messages sended to the queue, so the mesages been buffered in the queue while receiving message
                          were several hundred thousand or several million. And along with the send rate to be increased the buffered number
                          was increased, so the time to complete the receving job increased too. And the most important point was that the
                          receive rate was so poor, almost equaled to 1/10 of it's send rate per parallel test.

                          Why the messages receive rate so poor?
                          Does the queue must scan some of (or all) the message handles (or something like that) while a consumer receive a message
                          because of the message selector mechanism ?
                          Does the queue will copy all the message handles or synchronize all the message handles before it do this scan
                          , so the message receiving time of one message receiving trip will increase along with the messages buffered
                          in the queue increased ?
                          Does message receive rate is computed like this :
                          all messages received / message job spended time ?

                          • 10. Re: I got some performance problem while sending a lot of me
                            gary.c.chen

                            the duration parameter to ten minutes

                            • 11. Re: I got some performance problem while sending a lot of me
                              gary.c.chen

                              timfox, could you please give me a reply ? thanks.