Automatic cluster rebalancing?

[mikkeljb]

I am trying to setup a symmetric HornetQ  cluster, with a clustered queue.

All nodes have one or more JBoss AS server with MDB's as registred consumers.

When the Messages arrived they are Balanced using round robin.

Sometimes the consumer rate varies between the nodes in the HornetQ cluster, so some nodes emties there queue when others have many messages in their queue.

Is their someway to configure the cluster to automaticlly redistrube messages already in queues, even though all cluster nodes have registred consumers?

Thanks.

[markslater]

It would be really helpful for the server-side load balancing to be a) reflective of capacity of the node, and b) dynamic.

At present, if I have a cluster of two nodes, A and B, and for a given queue, I have one consumer on node A, and four on node B, if I put 1000 messages on the queue, node A gets 500, and node B gets 500.  That single consumer on node A has four times as much work to do as those on node B.

As time progresses, and the node B consumers consume all the messages on node B, they end up idle, while poor old node A still has a big stack of messages to process.  At this point, if I terminate the consumer on A, HornetQ can redistribute all of node A's messages to node B, and all four node B consumers can chew through those remaining messages.

It would be great if HornetQ could be aware of in imbalance in consumption capacity - for example

1. The number of messages sent to a particular node could be proportional to the number of the queue's consumers on that node - in the example above, node A would get 200 messages for its one consumer, and node B would get 800 for its four consumers
2. The number of messages sent to a particular node could be adjusted according to the backlog of messages on the queue by node - if node A has a huge backlog, but node B is idle, it makes sends to send more messages to B than A
3. *BEST OPTION* when a node has no messages for a given queue, but another node does, messages should be redistributed to the idle node.  In other words, if node B delivers its last message for a given queue, but node A still has a large number of messages for that queue, a number of those messages should be taken from node A and redistributed to node B.

Note that this situation can also be encountered due to individual consumers having different performance characteristics (for example if they run on hosts with different specifications), or if some messages on a given queue are much more expensive to process than others.

[ataylor]

1. this was on the dev forum which is probably why there was no response to it, all discussions should go on to the user forum.

2. Mark, you should try not to Hijack other peoples threads for your questions, alays open your own thread.

3. There is a JIRA already open for the feature/s you suggest.

[markslater]

Sorry - I figured my requirement was the same as the OP's.  The JIRA you're referring to is [HORNETQ-721] Message Redistribution based on consumer statistics - JBoss Issue Tracker, right?

[ataylor]

correct