JMS-based client-side load balancing: doc unclear?

[kennardconsulting]

Hi guys,

I have a cluster of 3 nodes. Each node has different performance, so for fastest results I want a single queue in a single node that all MDBs consume off as fast as possible. This way the slow nodes do not hold up the fast nodes. To achieve this under JBoss Messaging I set 'PrefetchSize' to 0. Under HornetQ I set 'ConsumerWindowSize' (in /server/all/deploy/jms-ra.rar/META-INF/ra.xml) to 0.

However I am still seeing the fast node process all its messages first, then wait for the other nodes to finish.

I think this is because HornetQ has *two* load balancers: client side and server side. So I disabled server side by setting <max-hops>0</max-hops>. Now all messages are put into a single node. But they are also consumed by a single node!

I am unclear how to balance the other nodes so they feed off this single node's queue? The documentation says:

"With client-side load balancing, subsequent sessions created using a single session factory can be connected to different nodes of the cluster"

Apparently this is the default? But nothing seems to be happening. What does 'subsequent sessions created using a single session factory' mean in the context of JMS and ra.xml? I currently have maxSession (under the MDB's activation-config) limited to 4 (fast node), 2 (medium node) and 1 (slow node) to throttle message processing.

Regards,

Richard.

[clebert.suconic]

We used to have sessions load balancing on the nodes, but on 2.2.2 the sessions are bound to where the connection is. I'm not sure we made some mistake on the configuration here (so we may need to fix it).

max-hops is about server's side load balancing. Which means a message is round robbed. it doesn't have anything to do with client load balancing.

[kennardconsulting]

Clebert,

Thanks for your reply!

Yes I understand <max-hops>0</max-hops> just stops server-side balancing. Now I need a way to make the other nodes in my cluster consume off the first node's queue.

I tried putting <redistribution-delay>1000</redistribution-delay> in hornetq-configuration.xml, but it didn't do anything. It doesn't really sound like the right setting, because it talks about 'consumers being closed'. In my case, the MDB consumer won't be closed - it will be busy. I want the other messages in the local queue redistributed to the idle nodes.

From a high-level, my question is: how can I configure JBoss 6 so that, in an cluster where the nodes have varying processing speeds, message processing is completed as quickly as possible (ie. no node is ever left idle).

Richard.

[ataylor]

ok, MDB's do not client side load balance. a pool of MDB's share a session factory and load balancing is done at a server locator level (in jms this would be connection and connection factory respectively). The only way you could get client side load balancing would be to deploy multiple mdb's.

[kennardconsulting]

Andy,

But isn't 'some of my MDB nodes are slower than others' a common use case? It is mentioned in the manual but the advice there just says 'use consumer-window-size=0'. It doesn't sound like this is going to work for MDBs?

Are you saying HornetQ does not support slow MDBs? Can't I get the equivalent to the JBoss Messaging functionality? Deploying multiple MDBs doesn't seem right, because presumably I would need to hard-code the name (and how many) cluster nodes I was talking to?

Richard.

[ataylor]

if the wondow size for your consumer is 0 then there should be no starved MDB's as each buffer is always <1.

[kennardconsulting]

But the MDBs are on different nodes. It does not make much sense to talk of 'fast' MDBs and 'slow' MDBs all on the one node, right? Because 'fast' and 'slow' is usually governed by the node itself (processor speed, disk speed, bandwidth etc).

So we have different MDBs on different nodes. 'Consumer window size' does not help, because it only applies to the local queue (for MDBs at least). And so, yes, the MDB will consume all messages off its local queue. But it won't then go looking at remote queues. It will just sit idle.

In summary: if my producer produces 300 messages, and they get round-robined 100 to each node, then the overall time of processing is always going to be limited to the time it takes the slowest node to process its 100 messages. But if I send all 300 messages to just one node, the other nodes sit there idle.

Is there a recommended solution to this? What I want is, say, the fast nodes do 125 messages each and the slow node does 50. If "MDB's do not client side load balance", as you say, what I need is a session factory that can steal from remote queues once its local queue is empty?

[ataylor]

But the MDBs are on different nodes. It does not make much sense to talk of 'fast' MDBs and 'slow' MDBs all on the one node, right? Because 'fast' and 'slow' is usually governed by the node itself (processor speed, disk speed, bandwidth etc).

Ahh, ok, I assumed you only had MDB's on one node.

So we have different MDBs on different nodes. 'Consumer window size' does not help, because it only applies to the local queue (for MDBs at least). And so, yes, the MDB will consume all messages off its local queue. But it won't then go looking at remote queues. It will just sit idle.

In summary: if my producer produces 300 messages, and they get round-robined 100 to each node, then the overall time of processing is always going to be limited to the time it takes the slowest node to process its 100 messages. But if I send all 300 messages to just one node, the other nodes sit there idle.

Is there a recommended solution to this? What I want is, say, the fast nodes do 125 messages each and the slow node does 50. If "MDB's do not client side load balance", as you say, what I need is a session factory that can steal from remote queues once its local queue is empty?

Ok, I understand what your topology is now, what you really want is to dynamically server side load balance depending on what load a particular node is under which we currently don't do, and to be honest, the overhead of doing this would probably negate the time lost by your MDB sitting idle as there is a lot of info that would need to be tracked. Also remember that once we start re distributing messages around nodes ordering can be broken.

[kennardconsulting]

Andy,

> the overhead of doing this would probably negate the time lost by your MDB sitting idle

I think that depends on a number of things. In my case, where each message takes around 6 seconds to process on the slow node and 3 seconds to process on the fast nodes, it certainly would not.

So this sounds like a use case that used to be supported by JBoss Messaging but is no longer supported by HornetQ. Could you help me find a solution? For example:

1. What would be involved in writing a server-side ConnectionLoadBalancingPolicy to send 2/3 of the messages to the fast nodes and 1/3 to the slow node? This wouldn't be dynamic, but it would be good enough for my needs

2. Could I put the journal folder on a shared drive, then point each of the nodes at the same journal folder? This is most similar to JBoss Messaging I think (ie. a shared database)?

3. What would be involved in writing a client-side ConnectorFactory to steal messages off remote queues if the local queue is empty?

Regards,

Richard.

[ataylor]

I think that depends on a number of things. In my case, where each message takes around 6 seconds to process on the slow node and 3 seconds to process on the fast nodes, it certainly would not.

Yes in your case, remember the hornetq server doesnt know that a consumer is slow , how many messages it has in its buffer, etc etc. im not saying its not possible, im just saying that its not simplistic.

So this sounds like a use case that used to be supported by JBoss Messaging but is no longer supported by HornetQ. Could you help me find a solution? For example:

well i wouldnt say it was supported in JBM, in fact iirc JBM always ditributes to a local node if there is one, so know message redistribution.

1. What would be involved in writing a server-side ConnectionLoadBalancingPolicy to send 2/3 of the messages to the fast nodes and 1/3 to the slow node? This wouldn't be dynamic, but it would be good enough for my needs

currently this isnt pluggable like the client side load balancing

2. Could I put the journal folder on a shared drive, then point each of the nodes at the same journal folder? This is most similar to JBoss Messaging I think (ie. a shared database)?

No, a journal can only be written to by one node

3. What would be involved in writing a client-side ConnectorFactory to steal messages off remote queues if the local queue is empty?

Im not really sure what you mean by this.

you could configure your MDb's to all connect to the same messaging server? this is typically how this is done if you want to load balance, maybe even have a remote HQ server but thats up to you

[ataylor]

you could configure your MDb's to all connect to the same messaging server? this is typically how this is done if you want to load balance, maybe even have a remote HQ server but thats up to you

Actually, if you think about it, if you have many producers load balanced across the nodes then basically nodes will just be swapping messages.

[kennardconsulting]

Andy,

> well i wouldnt say it was supported in JBM, in fact iirc JBM always ditributes to a local node if there is one, so no message redistribution.

That's true, but JBM could point all its MDBs to a 'remote queue' (ie. a relational database) so 'redistribution' was not necessary.

> You could configure your MDb's to all connect to the same messaging server?... Actually, if you think

> about it, if you have many producers load balanced across the nodes then basically nodes will just be swapping messages.

That sound promising. Could you please explain how I could set this up inside JBoss AS 6?

Regards,

Richard.

[clebert.suconic]

That's true, but JBM could point all its MDBs to a 'remote queue' (ie. a relational database) so 'redistribution' was not necessary.

It's  no different on HornetQ. You can still point your MDB to a remote node.

On JBM you could point to a remote node of JBM as well.

The fact that JBM used a Database doesn't make the DB a remote queue. The DB is meant to be used by a single instance.

[kennardconsulting]

Clebert,

Okay fair point. I have lodged an RFE: https://issues.jboss.org/browse/HORNETQ-721

Thanks,

Richard.

[ataylor]

simply configure the ra.xml to look up either a standalone instance of HormetQ or the AS6 node you weant to use. theres a jca rremote example that does something similar