13 Replies Latest reply on Sep 28, 2006 8:28 AM by omerlin

    Poor performance after replacing JBossMQ

    omerlin

      Hello,
      Trying to improve the performance on our application, i tried to replace the plumbering - as i saw the performance should be signifantly better with JBossMessaging.
      We use the JBoss4.0.4GA .
      I followed the How-to wiki and had to change some lines of code (I should open a minor bug on the queue.send function ) and achieved functional tests.
      But when i tried to stress my application with our homemade stress tool l was very deceived - all was very slow.

      I would like to know if our architecture is not responsible of this poor performance :
      We use 1 queue with many selector (20 in my test) based on a hash on the selector identifier. We have so 20 sequencer scaning for message in one queue.

      I read this is a not-performant way of processing message (JBMessaging-275) - but is it normal that we have a so obvious difference with JBossMQ ?

      Olivier

        • 1. Re: Poor performance after replacing JBossMQ
          timfox

          Having multiple consumers with selectors on a queue is normally an anti-pattern (http://jira.jboss.com/jira/browse/JBAS-1348), and will always have poor performance compare to using selectors with topics.

          This will be true for any JMS implementation.

          However I am surprised there is such a difference in performance between JBoss Messaging and JBoss MQ since they do pretty much the same thing. What version of messaging are you using?

          • 2. Re: Poor performance after replacing JBossMQ
            omerlin

            I'm using the 1.0.1CR4.

            • 3. Re: Poor performance after replacing JBossMQ
              ovidiu.feodorov

              What is the difference in performance?

              I am interested in replicating this locally, could you please describe your experiment in such a way it can be replicated?

              A working test would be best, if you have the time ...

              • 4. Re: Poor performance after replacing JBossMQ
                omerlin

                Hello,

                It's very difficult to treproduce as this is a big telecom project - Phonbook3G synchronization - (6 month with 12 people) - each sequencer that use a JMS selector on the shared queue launch a complex SYNCML synchronisation with an external server

                My idea was just to replace the plumbing because :
                1) i found that the persistence implementation in Oracle is costly - and the implementation seems different & optimized with JBossMessaging
                2) One key point on JBoss Messaging is the performance improvment

                My performance test use a home-made injector (due to proprietary protocol).
                -With JBossMQ i achieve around 2 req/s
                - With JBossMessaging less than 0,5 req/s ( and i get some timeout )
                For information 2req/s means around 50 open session on JBoss.

                I will try to give you more precise information using OptimizeIt

                Is it possible that i did some configuration mistakes that could explain this difference ? (I don't see what)

                rgds,

                • 5. Re: Poor performance after replacing JBossMQ
                  timfox

                  Omerlin-

                  I'm still trying to understand why you are using multiple selectors on a queue - as already mentioned this is usually an antipattern (poor design) - there is one case where it is not, so if you could explain to me more details about how/what/why you use these selectors then I can explain more.

                  • 6. Re: Poor performance after replacing JBossMQ
                    ovidiu.feodorov
                    • 7. Re: Poor performance after replacing JBossMQ
                      omerlin

                      Thank you for your interrest !

                      Messages sent by the client to the shared Queue are group by a hash code.
                      The message unique identifier is named MSISDN.
                      For example, each message with attribute MSISDN (a long) finishing by 0 will be carried to the sequencer 0
                      each message with attribute MSISDN (a long) finishing by 1 will be carried to the sequencer 1
                      (...)
                      each message with attribute MSISDN (a long) finishing by N will be carried to the sequencer N


                      We have N sequencer Staleless SB - each sequencer drives a protocol exchange and read a message based on the hash code (between 0 and N)

                      I hope i'm clear : each sequence get its batch of message for processing.
                      The objective is to do batch processing.

                      I understand that this is an anti-pattern as we have to scan the shared queue (in fact there is 2 queues like this) and this is perhaps even worse as many threads do the same scan.

                      One solution we imagine is to fix the batching size to 10 for instance and to create 10 destination queues and 10 sequencers - we can do this configuration quite quickly

                      Do you have any suggestion of another pattern that may help to solve our problem ?




                      • 8. Re: Poor performance after replacing JBossMQ
                        timfox

                        Perhaps you can send to a topic rather than a queue?

                        Each sequencer creates a (durable?) subscription with the selector you mentioned?

                        If a message only ever matches the selector of one of the sequencers then this should work for you.

                        Selectors on topic subscriptions are much more performant than selectors on queues since the selector is evaluateed *before* the message is put in the subscription.

                        • 9. Re: Poor performance after replacing JBossMQ
                          omerlin

                          We did not use topics because there is only one destination but if i understand, even if you have only one destination (one topic) it's much more better to use selector on topics than on queues.

                          So in our case we can imagine to have many consumers on the same topic with different selector
                          Is it correct ?

                          • 10. Re: Poor performance after replacing JBossMQ
                            timfox

                            Yes.

                            When you create a consumer on a topic, it creates what is called a "subscription", this can either be durable or non durable (non durable survives a failure like a jms queue).

                            Each subscription is basically a queue itself, but the difference from a jms queue is that if you create the subscription with a selector then it will only receive messages that match the selector.

                            Therefore when you consume from it, there is no need to scan it since you already know that all messages match, hence it is much faster.

                            This is the "correct" pattern, as long as a message only ever matches one of the selectors.

                            • 11. Re: Poor performance after replacing JBossMQ
                              omerlin

                              I see a Sun article http://developers.sun.com/sw/docs/articles/integration/tuning_tradeoffs.html
                              where they say the same thing :
                              Selectors are more performant on non durable subscribers

                              But the real performance gain would be to have many destination ... right ?

                              • 12. Re: Poor performance after replacing JBossMQ
                                timfox

                                The most performant solution would be to have one topic and multiple subscriptions - one subscription per selector.

                                I can see no performance gain in having multiple topics, and having multiple queues will give you worst performance.

                                • 13. Re: Poor performance after replacing JBossMQ
                                  omerlin


                                  We will try the Durable subscribers with selector solution on our product- and i will stress our application with both JBossMQ & JBossMessaging.

                                  I'll give you a feedback asap