11 Replies Latest reply on Jul 31, 2007 2:28 AM by mzeijen

    Sequential processing of messages with the same Key

    mzeijen

      I have this queue that receives messages with changes to certain customer records. Messages of the same customer should be processed in a absolute sequential order. Messages of different customers can be processed parallel.
      The whole thing needs to work in a clustered environment.

      How can I achieve this?

        • 1. Re: Sequential processing of messages with the same Key

          FAQ: (Not actually in the JBoss FAQ, but is probably in Sun's JMS FAQ, it's in the spec).
          FAQ == Asked and answered many times before.

          • 2. Re: Sequential processing of messages with the same Key
            mzeijen

            I did search and I found something specific for Websphere but nothing in general or specific for JBoss.

            In the Sun FAQ, the Java EE 5 Tutorial and on Google I didn't find the solution.

            That is why I ask here, because I didn't find it...

            • 3. Re: Sequential processing of messages with the same Key

              Ok, since I haven't answered this question this month, I'll answer it again,
              but only in the short version (well actually with the full caveats in case
              somebody else figures out where the search engine is - its in the top right by the way :-).

              What you are asking for is the default behaviour, but you need to think about it.

              Messages are delivered in the order they are sent.

              But with the following caveats.
              1) They all have the same priority
              2) The receiver is not filtering messages (i.e. ignoring some messages)
              3) There is only one sender (senders can race with each other when sending)
              4) There is only one receiver on the queue (messages could go to different receivers
              if there are multiple) - not relevant for topics
              5) A message is not NACKed (negatively acknowledged) - if a message is Nacked
              and redelivered then all bets are off
              6) You don't use scheduled delivery

              So the answer to your question is (partially) to create topic subscriptions with a selector
              for each customer, but be prepared for redelivery not respecting the order.

              MessageListener instances (client side) can be attached to multiple subscriptions.
              And JBoss supports multiple subscriptions for an MDB (not in the spec)
              by configurating multiple invoker-proxy-(bindings) in jboss.xml

              How you avoid the senders racing with each other is something you'll have to
              resolve yourself (if that is relevant), probably with some kind of shared lock
              on who can send messages.

              • 4. Re: Sequential processing of messages with the same Key

                 

                "adrian@jboss.org" wrote:

                And JBoss supports multiple subscriptions for an MDB (not in the spec)
                by configurating multiple invoker-proxy-(bindings) in jboss.xml


                And before you ask, you'll need to configure the MDB singleton otherwise it
                behave like (4 - multiple receivers - even for topics :-)

                • 5. Re: Sequential processing of messages with the same Key
                  mzeijen

                  Thanks for taking the time for the explanation. Maybe I was looking for the wrong words (like: sequential jms) because I did use the search engine... I searched for an hour without finding anything useful. Please don't think that I easily post such questions on this forum because I know that you JBoss guys have a lot to do.

                  I need to study you solution a bit more because I don't understand several things about it. Like:

                  - Why do I need a topic? Isn't a queue enough because a message only needs to be processed by one receiver?
                  - Do I need to configure a invoke-proxy-binding for every customer? I do have 700.000 customers, and those can be added dynamically...

                  I thought that the solution would be like a cluster wide locking mechanism. This is what I mean:
                  - A message comes in to the message queue with a certain customer id property
                  - For this id a lock is acquired via the cluster wide singleton locking mechanism by the MB
                  - The next message for this id is received while the other is still being processed
                  - The process for the second message tries to acquire a lock, but needs to wait for the other process to finish for this id.
                  - The moment the lock is released the next process in line receives the lock and can start processing. In this case the second process.

                  Is this a stupid idea? Am I missing something about the whole JMS technology?


                  • 6. Re: Sequential processing of messages with the same Key

                     

                    "mzeijen" wrote:

                    - Why do I need a topic? Isn't a queue enough because a message only needs to be processed by one receiver?


                    Because then you need multiple receivers and selectors to get concurrency.
                    Contested queues with selectors are an anti-pattern for any reasonable
                    volume of messages (again see other posts).


                    - Do I need to configure a invoke-proxy-binding for every customer? I do have 700.000 customers, and those can be added dynamically...


                    You don't create 700,000 subscriptions that would be a management nightmare. ;-)
                    It would also require every send of a message checking 700,000 selectors to see
                    which subscription matches the message.

                    You use a "like", e.g. all customers beginning with A, B, C, etc.
                    to divide them into groups according to how much concurreny you need.

                    Or better yet, you use a "hash" of the customer id if you want a better spread.


                    I thought that the solution would be like a cluster wide locking mechanism. This is what I mean:
                    - A message comes in to the message queue with a certain customer id property
                    - For this id a lock is acquired via the cluster wide singleton locking mechanism by the MB
                    - The next message for this id is received while the other is still being processed
                    - The process for the second message tries to acquire a lock, but needs to wait for the other process to finish for this id.
                    - The moment the lock is released the next process in line receives the lock and can start processing. In this case the second process.

                    Is this a stupid idea? Am I missing something about the whole JMS technology?


                    Sounds stupid to me. :-) If I only because I doubt it would scale.
                    JMS is about asynchronicity, so you need to consider an asynchronous design
                    (if JMS is even relevant for what you are trying to achieve).

                    Think of JMS as more like a set of pipes and valves that let you control
                    where messages flow from and to. You want to direct all messages of a particular
                    type to a particular destination so they flow "sequentially" down that pipe.

                    • 7. Re: Sequential processing of messages with the same Key
                      mzeijen


                      You don't create 700,000 subscriptions that would be a management nightmare. ;-)
                      It would also require every send of a message checking 700,000 selectors to see
                      which subscription matches the message.

                      You use a "like", e.g. all customers beginning with A, B, C, etc.
                      to divide them into groups according to how much concurreny you need.

                      Or better yet, you use a "hash" of the customer id if you want a better spread.


                      I am missing something totally. A queue can process multiple messages at once, depending on the size of the thread pool, right?. Doesn't this create concurrency? If I understood the whole JMS clustering thing then the cluster should make sure that those processes get distributed one the cluster. If I want to make it more scalable then I only make sure that the queue can be processed by more threads and that more machines are available on the cluster. Lets forget about the DB for now, because that will become the bottleneck eventually (if you don't let it scale ;)).

                      But I see where you are going. Doesn't your solution also mean that a subscription can only have one thread to make sure that it processes all the messages sequentially? Because that would indeed be the solution.


                      • 8. Re: Sequential processing of messages with the same Key

                         

                        "mzeijen" wrote:

                        Doesn't your solution also mean that a subscription can only have one thread to make sure that it processes all the messages sequentially? Because that would indeed be the solution.


                        That's what I said above in the "before you ask" post about MDB singletons.

                        Let's say you have 26 durable topic subscriptions one for A, B, C, D, etc.
                        Then you create 26 MDBs to process those messages (single threadedly for each).

                        In a cluster you put the MDB in deploy-hasingleton or use some other
                        mechanism to distribute them across the cluster but making sure the MDB
                        for clients beginning with "A" only starts on one node.

                        • 9. Re: Sequential processing of messages with the same Key
                          mzeijen

                          Yeah, sorry you are right. I should have remembered that.

                          Is this solution only based on JBoss MQ or is this the general solution? Does the JBoss Messaging give me more possibilities?

                          • 10. Re: Sequential processing of messages with the same Key

                             

                            "mzeijen" wrote:

                            Is this solution only based on JBoss MQ or is this the general solution? Does the JBoss Messaging give me more possibilities?


                            It's just the JMS spec, at no point are any features specific to JBossMQ used.

                            I know JBoss Messaging tries harder than JBossMQ to solve the ordering problem
                            for NACKed messages.

                            • 11. Re: Sequential processing of messages with the same Key
                              mzeijen

                              Thank you for your help and your patience.