1 Reply Latest reply on Jun 26, 2008 8:20 AM by fl_

    Messages distribution across cluster.

    fl_

      Hello,

      We are thinking about switching from JBossMQ to JBossMessaging. I have read user guide for JBM 1.4.0 SP3 and few messages on forum but I think I still need clarification about how messages are distributed across nodes in cluster configuration. I read a following topic but still don't know if I understand everything correctly.
      http://www.jboss.com/index.html?module=bb&op=viewtopic&t=133762&postdays=0&postorder=asc&start=0

      We have three application servers, called AS1, AS2, and AS3. Our application is assembled from:
      Producer A (singleton) - deployed only on AS1
      Queue A (clustered)
      Consumer A - deployed on all nodes
      Producer B - deployed on all nodes
      Queue B (clustered)
      Consumer B - deployed on all nodes

      Deployment and messages flow:

      (AS1) Producer A ---> Queue A ---> Consumer A/Producer B ---> Queue B ---> Consumer B
      (AS2) Queue A ---> Consumer A/Producer B ---> Queue B ---> Consumer B
      (AS3) Queue A ---> Consumer A/Producer B ---> Queue B ---> Consumer B
      

      As you can guess, I wish that load related with queue A will be balanced across cluster. If I understand correctly, if non-loadBalanced ConnectionFactory will be used in Producer A only Consumer A on AS1 will be used to handle whole load, right? (according to the rule: 'local consumer is preferred')
      Have it got any matter how many elements will be in queueA? Is there any starvation-protecting mechanism which will sends part of messages across cluster if 'number of messages' or 'time in queue' exceed some limit?
      In other words, is there any situation where rule: 'local consumer is preferred' will be ignored.

      If I understand correctly the best I can do is to use loadBalanced ConnectionFactory in Producer A. Then different connections created via this ConnectionFactory will be attached to various nodes. Then load will be balanced. Am I right?


      Second issue is about rule: 'local consumer is preffered'. If I understand correctly in cluster configuration every node has its own 'local queue' for every queue.
      In fact it may be very useful feature for us, if part of the flow stay on one node. It means behavior where: Consumer A on AS2 gets a message, process it and send new message to queue B (via Producer B on the same node: AS2), then Consumer B on the same node (AS2) will be used to process this message. So part of the flow: 'Consumer A/Producer B ---> Consumer B' will be processed on the same node.
      If I understand correctly this will happen if non-loadBalanced ConnectionFactory will be used in Consumer A, Producer B and Consumer B, right?
      Will it be ok, if Producer A sends message via loadBalanced ConnectionFactory to QueueA and Consumer A use non-loadBalanced ConnectionFactory to consume such message?

      Most important question in this approach is about failover. Lets say that there is processing and all queues (especially QueueB) keeps messages (each node has its own messages in its own 'local queue') and then, lets say, AS2 go down. Will messages related with AS2 be seamless distributed to other nodes?

      Best regards.

        • 1. Re: Messages distribution across cluster.
          fl_

          Hello,

          I made few tests and found out strange behavior.
          Lets say that we have deployment layout like following:

          (Node0) Producer ---> @ <--- Consumer (dest = 'moduleA')
           |
          (Node1) @ <--- Consumer (dest = 'moduleB')
           |
          (Node2) @ <--- Consumer (dest = 'moduleC')
           \---- Consumer (dest = 'singleton')
          


          All producers and consumer are standalone applications. So Producer and Consumer with selector dest = 'moduleA' are connected to Node0, Consumer with selector dest = 'moduleB' is connected to Node1 and two consumers are connected to Node2 with selectors like above.

          So now, Producer send message with stringProperty set to dest='singleton' and this message stuck on Node0. It is not distributed to Node2 (in database it is assigned to Node0) and what is more important, it can't be received with Consumer connected to Node2.
          We can send even few hundreds of such messages and they all stuck on Node0.

          It is the same if we send message to moduleC, it just stuck on Node0.
          Messages to moduleA can be received with consumer connected to Node0.

          Now we close consumer connected to Node0, so now deployment looks like follow:
          (Node0) Producer ---> @
           |
          (Node1) @ <--- Consumer (dest = 'moduleB')
           |
          (Node2) @ <--- Consumer (dest = 'moduleC')
           \---- Consumer (dest = 'singleton')
          


          Messages are not switched to another node, they stay attached to Node0. So Consumer with selector dest = 'singleton' is still not able to receive its messages (that are currently in queue).

          Now, we send one message to module 'singleton'. And this trigger some kind of action, messages are switched to Node2, and now we can receive them by consumer (dest='singleton) connected to Node2.

          I think that even after sending this one message, there is still some mess in database (and in behavior), because I had had messages with desc='moduleB' in queue before, and now they were transfered to Node2 instead of to Node1. So they stuck there to ;)

          JBoss 4.2.1 GA
          JBoss Messaging 1.4.0 SP3 CP02

          I don't know it is a bug? I look inside JMS specification, but there is very little about situations where few consumers with different selectors are connected to one queue especially in clustered environment.
          Nevertheless I think it is expected that above situation will be working "correctly" ;) it means each message will be received by corresponding consumer.

          There are two other things that I noticed:
          1) a lot of SQL updates
          When messages are switching across cluster, there is a lot of database UPDATE operation. It can impact performance, can't it?
          2) message order
          This 'message switching' breaks any order in queue. I don't want start discussion about that order should be kept or not.
          My question is: Is there chance that one of message will be extremely unlucky and will be never consumed ?
          In situation where we have constantly a lot of messages in queue: producers are adding new messages and consumers are receiving them little slower; we expect that old messages will be generally consumed before new one.
          I think that in current approach, it is possible that one message will be on some "switching mode" state every time when consumers request message to consume. And it stuck in queue.

          BR