4 Replies Latest reply on Mar 6, 2013 8:37 AM by ataylor

    Load balancing across cluster nodes

    zohar_melamed

      Hi

       

      We have a hornetq cluster with a number of worker consumers connected directly to specific cluster nodes.

       

      Work requests are sent to a queue.

       

      Occasionally we have a slow cluster node = a node where we lost a  worker or the wrokers are slow due to various reasons.

       

      The system is not load balanced dynamically and so slows down considerbly when any workers are slow - even if all other workers are idle.

       

      Had a look at the docs but could not see an obvious solution. I would like to re route the messages from the slow cluster node to the other nodes , or change the way messages are deliverd so the slow node is sent less work.

       

       

      Any suggestions on how we can tackle this issue?

       

      Cheers

      Zohar

        • 1. Re: Load balancing across cluster nodes
          gaohoward

          You can set consumer-window-size to 0 (or a small value depends on the actual situation of yours) to make your slow consumer not cache any messages.

           

          Howard

          • 2. Re: Load balancing across cluster nodes
            zohar_melamed

            Thanks Howard,

             

            The problem is not the consumer-window-size - thats already set to 1 (or very small).

            The problem is at the broker load-balancing level.

             

            We are using a queue as a means to balance load across workers. The problem we have is that messages dont end up getting balanced 'blind' across nodes in the cluster.

            Let say there are 3 brokers A, B & C and each have 10 consumers connected to each one - but for some reason node C has very slow consumers (or ended up with a lot of expensive tasks, or lost some consumers).

             

            Under normal circumstances the messages to a given destination (queue) are balanced across the nodes in the cluster. But what we are seeing is that all consumers connected to nodes A & B are idle - all their messages are consumed. But node C still has a load of messages and its consumers are slowly working through them.

             

            In our case, we have 11 brokers in a cluster - and the problem we have is that 10 brokers and their consumers sitting idle - and the whole system waiting for the handfull of consumers to work through the backlog of messages on the 11th node.

             

            What Id like to know is if there is some way of tackling this problem?

             

             

             

            Cheers,

            Zohar

            • 3. Re: Load balancing across cluster nodes
              mookythemook

              The load-balancing mechanism seems a bit flawed.

              Queue semantics lend themselves to pull-based (ie natural) load balancing - yet HornetQ doesnt use that internally - and instead uses some push-based load balancing (round-robin by default)?

              • 4. Re: Load balancing across cluster nodes
                ataylor

                The load-balancing mechanism seems a bit flawed.

                Queue semantics lend themselves to pull-based (ie natural) load balancing - yet HornetQ doesnt use that internally - and instead uses some push-based load balancing (round-robin by default)?

                I'm not sure I would agree with you that it is flawed or that one way is better than another, both have disadvantages. However, we do need to make it more dynamic which we have planned for future releases.