Jboss WildFly JMS cluster - messages are sent to only one node during each run

[tv_padmanabhan]

Hi,

We are trying to setup a JBoss JMS cluster for load-balancing. There is a reported issue on the use of MDBs for load-balancing. I am not sure if there is a workaround/fix for this, but we do want to send message to a cluster and have MDBs listening for messages addressed to specific destinations. Does anyone know a solution for this?

The other main issue is that I have configured a 2-node cluster but when I try to send messages, all messages are redirected to one node only at any given time, i.e. during another run all messages go to another node. Is this a configuration issue? If so which config value needs to be changed so that the messages are properly distributed? I tried specifying the

org.hornetq.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy

and also tried specifying the same while configuring the ConnectionFactory on the client-side as below:

HornetQConnectionFactory connectionFactoryWithHA =
  HornetQJMSClient.createConnectionFactoryWithHA(configuration, JMSFactoryType.CF);
connectionFactoryWithHA.setConnectionLoadBalancingPolicyClassName(RandomConnectionLoadBalancingPolicy.class.getName());

Attached is my standalone-full-ha.xml and I start the nodes as follows:

standalone.bat -c standalone-full-ha.xml -b <ip> -Djboss.node.name=<jboss1 | jboss2>

Thanks,
Padmanabhan

[jbertram]

We are trying to setup a JBoss JMS cluster for load-balancing. There is a reported issue on the use of MDBs for load-balancing. I am not sure if there is a workaround/fix for this, but we do want to send message to a cluster and have MDBs listening for messages addressed to specific destinations. Does anyone know a solution for this?

The forum thread you cited here is a few years old and is about JBoss AS 5.1 with JBoss Messaging.  It is completely unrelated to Wildfly with HornetQ.

The other main issue is that I have configured a 2-node cluster but when I try to send messages, all messages are redirected to one node only at any given time, i.e. during another run all messages go to another node. Is this a configuration issue? If so which config value needs to be changed so that the messages are properly distributed? I tried specifying the

I recommend is reading the HornetQ documentation on clustering (which includes load balancing).  Pay special attention to the <forward-when-no-consumers> configuration element on <cluster-connection>.

[tv_padmanabhan]

Hi Justin,

     Thanks a lot. It worked like a charm. Sorry for using this discussion for asking one more clarification (or let me know if I should open another one). I am planning to make the server (read message producer) send message to the cluster without knowledge of the consumer but the messages should be distributed (load-balanced) based on consumer into respective queues. Is this possible - it looks like use of message filters to me but I could not find an article for the same? Or is it a better idea to use something like a sticky connection factory at the producer end itself to send to specific node in the cluster based on the consumer - is it even possible?

    In the load-balancing configuration, do we also  get any failover as well, let's say node A goes down, how do consumers currently reading from node A queue get failover or how to achieve this?

Thanks,

Padmanabhan (Paddy)

[jbertram]

I am planning to make the server (read message producer) send message to the cluster without knowledge of the consumer but the messages should be distributed (load-balanced) based on consumer into respective queues. Is this possible - it looks like use of message filters to me but I could not find an article for the same?

Messages sent to any node in the cluster will be distributed to any matching consumers on any other node in the cluster.  Did you not read the documentation I cited previously?  It specifically discusses server-side message load-balancing.

It's not clear to me where message filters enter the picture.  Are your consumers using message filters?

In the load-balancing configuration, do we also  get any failover as well, let's say node A goes down, how do consumers currently reading from node A queue get failover or how to achieve this?

Load-balancing and fail-over are mutually exclusive.  If you want more details on how to configure fail-over I suggest you read the documentation on fail-over.

In general, I think you'll find the documentation answers a lot of questions and will save you (and others) time spent on the forum.

[tv_padmanabhan]

I am not sure if i am missing something here, but I see this line regarding server-side load-balancing:

"but they can be a bit cleverer than that and also be configured to only distribute to other nodes if they have matching consumers. We'll look at both these cases in turn with some examples"

but the examples do not seem to be in that page and the next page seems to cover a different topic.

[jbertram]

I don't see any specific examples in that section of the documentation.  However, the server-side load-balancing behavior referenced here is controlled by the <forward-when-no-consumers> configuration element on <cluster-connection> that I previously told you to pay special attention to.

If you want specific examples, take a look at the Clustered Queue example and/or the Symmetric Cluster example.  The actual examples are available in the HornetQ distribution.

[tv_padmanabhan]

http://docs.jboss.org/hornetq/2.4.0.Final/docs/user-manual/html/examples.html#examples.queue-message-selector seems to match my needs. I need a consumer that listens on specific properties. But I am not sure if load-balancing will still happen across them. For example if node A is listening on some properties set P1 and node B is listening on some properties P2, and if I send 6 messages with P1 and 4 with P2, will node A properly receive only the expected 6 and node B the expected 4 messages?

[jbertram]

For example if node A is listening on some properties set P1 and node B is listening on some properties P2, and if I send 6 messages with P1 and 4 with P2, will node A properly receive only the expected 6 and node B the expected 4 messages?

I'm not 100% clear on what you mean here.  You talk about "node A" and "node B" listening, but nodes in and of themselves don't listen.  I assume by "node A" you mean "a consumer running on node A".

To answer your question - yes.  If the consumers are connected when the messages are sent then they I would expect them to get messages that match their respective filters.  I recommend you try this out yourself.

[tv_padmanabhan]

The Queue selector worked as per my requirements but with a problem. I tried starting one of the consumers after the producers have sent messages (Thread.sleep). The JMS provider ended up sending all messages to the other node in the cluster. Now when the late-arriving consumer starts up, is there a way ti can pull the messages intended for itself from the other node where all messages were sent?

[jbertram]

I tried starting one of the consumers after the producers have sent messages (Thread.sleep). The JMS provider ended up sending all messages to the other node in the cluster. Now when the late-arriving consumer starts up, is there a way ti can pull the messages intended for itself from the other node where all messages were sent?

Load-balancing based on selectors is supported (i.e. consumer connects before the message is sent), but redistribution based on selectors is not supported (i.e. consumer connects after the message is sent).

[tv_padmanabhan]

Hi Justin,

     Thank you for sharing that information. I searched for the sentence you have mentioned but could not get any matching result. Would you be able to post the documentation that explains this, so that I can try to see if there is any other option? One idea which I am not sure is ideal is to have some sort of message timeout listener (if that is possible to have), so that when consumer at node A is not available and all messages intended for this consumer are sent to the consumer at node B. Then may be our application instance running in node B can have a message timeout listener and remove all messages waiting in its queue and then resend them (in the role of a producer) (so if the consumer at node A appears within that time, it will pick up). May be there is a better option.

Regards,

Paddy

[jbertram]

My comment wasn't quoting any specific documentation.  I was simply explaining why the behavior your observed was expected.

It's hard to recommend any alternatives without knowing more about your use-case and why you're consuming from a queue with selectors.  In general, consuming from a queue with selectors is discouraged due to the performance penalty that is inherently imposed (due to repeatedly scanning the queue for selector matches).  In general, alternatives include:

• Using a topic with subscriptions that use a selector.
• Don't use a cluster.  A single HornetQ instance can deal with millions of messages per second (verified with Spec JMS) so it's likely you don't need load-balancing to handle your message volume.  Most users simply assume they need load-balancing without doing the actual work to bench-mark and verify.