1 2 Previous Next 24 Replies Latest reply on Aug 19, 2008 12:36 PM by timfox

    Message redistribution in JBM 2.0 - request for feedback

    timfox

      As you know, JBM 1.4 contains message redistribution with clustered queues.

      This means, if you have a clustered queue over several nodes in a cluster, and consumers on one node are faster than consumers on another node, then messages will be pulled from one node to another transparently to satisfy demand.

      The idea here is it should give optimal use of server resources for processing messages.

      Problem is this behaviour can confuse users. The classic case is they set up a cluster of N nodes with a clustered queue, send messages to a single node and wonder why all their consumers on different nodes aren't processing the messages. This is because the local node still has spare cpu cycles so there is no point in allowing other nodes to consume, since that would involve unnecessary network round tripping. So it's actually the optimal use of resources.

      Regarding JBM 2.0 I'd appreciate any feedback on whether you think this feature is of value. Or the customer confusion value outweighs its value, or if you think it should behave in some other way.

        • 1. Re: Message redistribution in JBM 2.0 - request for feedback
          clebert.suconic

          IMO the user should have an option.

          In some cases where you don't have more than few messages/second in a queue, the Singleton queue would make more sense for the user, but in cases where you would flood the servers the clustered queue is necessary.

          Just a brain storm: Maybe we could change the behavior based in some statistics? Say after a message start flooding one server we change the distribution schema from single to partial queue?

          • 2. Re: Message redistribution in JBM 2.0 - request for feedback
            timfox

            When you say Singleton I think you really mean "non clustered queue". Singleton is something quite different - it's when you have the queue on only one node of the cluster and is not currently supported by JBM.

            Regarding your brain storm - I don't think this is a good idea. The idea was to simplify, not make things more complex. KISS principle again. ;)

            • 3. Re: Message redistribution in JBM 2.0 - request for feedback
              jay.howell

              Yeah Singleton has a loaded meaning and we should probably not use it jbm terms because it's confused with the jboss terms.

              If we didn't have partial queue on each machine, what would be the alternative for load balancing messages in a queue. Would we(I'm just throwing out some thoughts)...

              1. Only allow one(it an it's buddy) node on the cluster to hold the actual queue(I think this is what Clebert was talking about). Then when a consumer hits a node in the cluster, it's directed to the one that has the actual queue on it. This would allow a consumer to go to anynode in the cluster and ask for a queue, where by it would be redirected to the machine that has the real queue, but the actual queue would be on one node.

              2. Drop load balancing support all together for queues and make consumers go to the machine that the queue is on(either that or it's backup node). So the clients would have to have some knowledge of what queues are on what boxes in the cluster.

              3. Replicate the entire queue to one or more buddies. Performance would go down, but uses could choose to have the queue completely replicated on each node. We spoke about making the number of buddies in the cluster configurable, so you could choose, just one for failover or multiple. In this same paradigm users could choose which nodes in the cluster will have the queue.


              I do think that the idea of pulling messages around the ring of nodes was a little confusing to users(especially when they start using selectors), but I do think that the ability to be able to load balance consumers on different boxes, doing local access(without having to hop to a different cluster member) is a definite need. So I'd start off with what I would believe the requirements to be from a customer stand point.

              • 4. Re: Message redistribution in JBM 2.0 - request for feedback
                jeffdelong

                I teach classes on the JBoss ESB / SOA Platform. ESB uses JBM clustered queues to achieve high availability and scalability. I describe the feature exactly as Tim has described it; and students seem to accept this as reasonable / preferable behavior. I would not want to see this change in the future. Customers just need to be better educated.

                Jeff

                • 5. Re: Message redistribution in JBM 2.0 - request for feedback
                  clebert.suconic

                   

                  "clebert.suconic@jboss.com" wrote:
                  IMO the user should have an option.

                  In some cases where you don't have more than few messages/second in a queue, the Singleton queue would make more sense for the user, but in cases where you would flood the servers the clustered queue is necessary.


                  What I meant to say is... as we discussed in the past.. for some cases a single Queue on the Cluster (That's why I said Singleton... the naming here should be correct) would make more sense than the Partial Queue, for cases where you don't need performance. (There is a JIRA for that .. I know).

                  If we had that option.. I would vote to just keep the Partial Queue the way it is on 1.4, and implement the Single queue on the cluster.

                  That's my vote.. but I'm not necessarily an interface with the user... so I value more the vote of folks who do the interface. (Like JeffDelong who is a trainer or Jay who is on support)

                  • 6. Re: Message redistribution in JBM 2.0 - request for feedback
                    timfox

                    Guys, please stick to the topic, I asked whether we should keep message redistribution or not (or change it), not whether we have clustered queues or how we implement clustered queues - that's not changing.

                    • 7. Re: Message redistribution in JBM 2.0 - request for feedback
                      timfox

                      +1 to Jeff's point that things need to be documented / explained better for JBM 2.0. I lose track of the amount of times I've had to explain this feature to confused users on the forums.

                      • 8. Re: Message redistribution in JBM 2.0 - request for feedback
                        timfox

                        Just to recap on what my original question was.

                        We have partial queues on each machine, we're not talking about whether we should have partial queues or not. We're talking about whether we should have _message redistribution_ in its current form ,or in a different form or got rid of altogether.

                        Let me define message redistribution as "The moving of messages between partial queues. This occurs after the message has reached its first partial queue. Messages are pulled from one node to another based on whether the remote node is active and the local node is busy".

                        Don't confuse message redistribution with load balancing of connections across nodes - that's something different.

                        • 9. Re: Message redistribution in JBM 2.0 - request for feedback
                          clebert.suconic

                          My vote is for get rid of Message redistribution.

                          As I said I haven't been dealing with customers... so I would value more the opinion of people who do it.

                          • 10. Re: Message redistribution in JBM 2.0 - request for feedback
                            timfox

                             

                            "clebert.suconic@jboss.com" wrote:
                            My vote is for get rid of Message redistribution.


                            If we did that, how would we deal with stranded messages (messages stuck on nodes with no consumers).

                            • 11. Re: Message redistribution in JBM 2.0 - request for feedback
                              clebert.suconic

                              IMO we just document customers will need to guarantee a consumer on each node.

                              But if you tell me that's not an option.. than the answer is probably simple.. we probably need to guarantee message redistribution as it was done on JBM 1.4.

                              • 12. Re: Message redistribution in JBM 2.0 - request for feedback
                                timfox

                                 

                                "clebert.suconic@jboss.com" wrote:
                                IMO we just document customers will need to guarantee a consumer on each node.

                                But if you tell me that's not an option.. than the answer is probably simple.. we probably need to guarantee message redistribution as it was done on JBM 1.4.


                                I think we need to prevent stranded messages, but that doesn't mean we have to do it as it was done in 1.4.

                                • 13. Re: Message redistribution in JBM 2.0 - request for feedback
                                  jeffdelong

                                  My apologies for not understanding the "re-distribution" feature. Tim's original post said

                                  ... contains message redistribution with clustered queues.

                                  This means, if you have a clustered queue over several nodes in a cluster, and consumers on one node are faster than consumers on another node, then messages will be pulled from one node to another transparently to satisfy demand.


                                  I thought this was how a clustered queue worked. That is, a message is put on a "local queue", and the messagePeer sends the message to each of its peers in the cluster. The first available consumer de-queues the message, and it is removed from the other message peers' instance of the clustered queue.

                                  What does the message re-distribution feature provide?

                                  • 14. Re: Message redistribution in JBM 2.0 - request for feedback
                                    timfox

                                     

                                    "jeffdelong" wrote:

                                    What does the message re-distribution feature provide?


                                    Message redistribution in 1.4 moves messages between partial queues to satisfy consumption demand.

                                    1 2 Previous Next