-
1. Re: Load balancing across cluster nodes
gaohoward Mar 3, 2013 9:34 PM (in response to zohar_melamed)You can set consumer-window-size to 0 (or a small value depends on the actual situation of yours) to make your slow consumer not cache any messages.
Howard
-
2. Re: Load balancing across cluster nodes
zohar_melamed Mar 4, 2013 5:54 AM (in response to gaohoward)Thanks Howard,
The problem is not the consumer-window-size - thats already set to 1 (or very small).
The problem is at the broker load-balancing level.
We are using a queue as a means to balance load across workers. The problem we have is that messages dont end up getting balanced 'blind' across nodes in the cluster.
Let say there are 3 brokers A, B & C and each have 10 consumers connected to each one - but for some reason node C has very slow consumers (or ended up with a lot of expensive tasks, or lost some consumers).
Under normal circumstances the messages to a given destination (queue) are balanced across the nodes in the cluster. But what we are seeing is that all consumers connected to nodes A & B are idle - all their messages are consumed. But node C still has a load of messages and its consumers are slowly working through them.
In our case, we have 11 brokers in a cluster - and the problem we have is that 10 brokers and their consumers sitting idle - and the whole system waiting for the handfull of consumers to work through the backlog of messages on the 11th node.
What Id like to know is if there is some way of tackling this problem?
Cheers,
Zohar
-
3. Re: Load balancing across cluster nodes
mookythemook Mar 6, 2013 6:44 AM (in response to zohar_melamed)The load-balancing mechanism seems a bit flawed.
Queue semantics lend themselves to pull-based (ie natural) load balancing - yet HornetQ doesnt use that internally - and instead uses some push-based load balancing (round-robin by default)?
-
4. Re: Load balancing across cluster nodes
ataylor Mar 6, 2013 8:37 AM (in response to mookythemook)The load-balancing mechanism seems a bit flawed.
Queue semantics lend themselves to pull-based (ie natural) load balancing - yet HornetQ doesnt use that internally - and instead uses some push-based load balancing (round-robin by default)?
I'm not sure I would agree with you that it is flawed or that one way is better than another, both have disadvantages. However, we do need to make it more dynamic which we have planned for future releases.