Problem with load balancing mdbs in hornetq cluster

[sv_srinivaas]

Hi, I have two issues with MDB and Hornetq cluster. It’ll be of great help if you can spare few mins of your valuable time as I’ve not been able to resolve the following issues.

I have gone through the cluster and mdb related portions of Hornetq user manual but still not able achieve load balancing with mdbs in cluster. Also I’m able to successfully execute the mdb and clustered a queue example that comes with Hornetq.

Version: Hornetq CR2OS:

Windows XP (SP2)

JBoss: 5.0.0. GA

1.     First issue is with the load balancing of messages in Hornetq cluster with (EJB3) MDBs where I have only the queue servers clustered and the mdbs deployed in third node which is NOT clustered.

Steps to reproduce the issue

a. Created a cluster of two queue nodes nodeA and nodeB with a queue defined (as sampleQueue).

b. Then I deployed two mdbs MDB1 and MDB2 both in the third node C (which is not part of the above cluster as I had set the clustered param to false for this node in hornetq-configuration.xml) where MDB1 consumes messages from sampleQueue of node A and MDB2 from sampleQueue of node B.

Also I could see that the consumer count for sampleQueue as 15 in both nodeA and B (15 is default pool size for mdb).

c. Then from a client application (hosted in tomcat) I sent 30 messages to sampleQueue in node A

What I could see from jmx console is that all the 30 messages were sent to node A and none was sent to node B and all the 30 messages were consumed by MDB1 (of nodeA) itself.

Is this the expected behavior? I was expecting 15 messages to be consumed by MDB1 from sampleQueue of nodeA and 15 messages to be consumed by MDB2 from sampleQueue of nodeB.

I’ve attached the xmls for all the 3 nodes.

2. Now the second issue is, when I set the clustered param to true in node C (in hornetq-configuration.xml) where the MDBs are hosted (because in PROD the MDBs will also be deployed in a separate cluster) I get the below exception during the nodeC startup.

ERROR [ClusterConnectionImpl] Failed to handle messagejava.lang.IllegalStateException: Cannot find binding for jms.queue.sampleQueue96bf7979-f9f8-11de-bf1c-001a4d1349b0       at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerCreated(ClusterConnectionImpl.java:758)            at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:552)            at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:767) at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:45)           at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:885)           at java.lang.Thread.run(Thread.java:619)

Also I have gone through the below url and made all the necessary changes to avoid any overlapping of Queue and MDB clusters (as they are in the same network). Pls let me know if i'm missing any configuration to run multiple clusters in the same network.

http://community.jboss.org/wiki/TwoClustersontheSameNetwork

[ataylor]

It looks like you havent balanced your sending client. take a look at the client-side-load-balancing-example. If you are only creating one connection in your client all messages will be sent to the same node.

[sv_srinivaas]

Andy, I was under the assumption that if I have the below code in hornetq-configuration.xml (as mentioned in the clustered-queue example) for both the queue nodes then Hornetq would automatically load balance the messages across all the nodes in cluster.

Also i've checked both the nodes and both are having consumers attached to the queue. Please correct me if i'm wrong

<cluster-connection name="my-cluster">
        <address>jms</address>
        <retry-interval>500</retry-interval>
        <use-duplicate-detection>true</use-duplicate-detection>
        <forward-when-no-consumers>true</forward-when-no-consumers>
        <max-hops>1</max-hops>
        <discovery-group-ref discovery-group-name="my-discovery-group"/>
     </cluster-connection>

Also do you have any inputs regarding my second issue with mdbs deplyed in a cluster?

Thanks

[ataylor]

yeah, sorry, i misunderstood you when you talked about load balancing, what you are talking about is message distribution.

Firstly i would check that your cluster nodes are discovering each other. you can do this by checking that the distributes queues exist. You can do this by using jconsole and connecting to each server and checking that the clustered queues are there.

after that start your MDB's and make sur ethat each queue has consumers.

It could be that the second MDB's remote consumer isnt being added, hence the error you see.

[sv_srinivaas]

I'm able to see the queue in both the nodes A and B through JMX console and I also see the consumer count as 15 in both the nodes queue. I could also see the message ALL Members : 2  in node A server console that confirms that the cluster has been created successfully.

But I somehow not able to see the messages being distributed to both the queues on a round robin fashion when sent from the client application. It always goes to nodeA and I'm not sure why.

Again instead of using MDBs if I run the example code that comes with Hornetq where they create two consumers and receive messages alternatively then it works fine.

[timfox]

This should work without you having to load balance at the client, since it should round robin on the server side.

Can you invoke the JMX method getNodes() on the ClusterConnection control via the jmx-console. This will tell you how many nodes in the cluster each node thinks there is, so you can tell whether the nodes have formed a cluster properly.

[timfox]

Also try not deploying the MDBs, then send the message then use the jmx-console to see how many messages are in the queue(s) on nodes A and B

[ataylor]

In the console check under address/core/jms.queue.sampleQueue/Attributes/queueNames there should be 2!

[sv_srinivaas]

Yes I could see the below two entries against queueNames under address/core/sampleQueue in both the nodes

jms.queue.sampleQueue6aef5796-faad-11de-ba2f-001a4d1349b0
jms.queue.sampleQueue

And for nodeA->ClusterConnections->Attribute name->Nodes I see the below entry

     {a12adaa5-faad-11de-9642-001c23457f19=/172.18.10.229:5445}

And for nodeB->ClusterConnections->Attribute name->Nodes I see the below entry

     {6aef5796-faad-11de-ba2f-001a4d1349b0=/172.18.10.229:5445}

Node id of nodeA: 6aef5796-faad-11de-ba2f-001a4d1349b0 

Node id of nodeB: a12adaa5-faad-11de-9642-001c23457f19

Ip of NodeA: 172.18.10.229

Ip of NodeB: 172.18.10.63

[sv_srinivaas]

Tried sending 30 messages without deploying the MDBs and all 30 messages went to nodeA and remained unconsumed.

[timfox]

That doesn't look right to me.

According to the output node A has made a connection to the node with address:

172.18.10.229

And node B also has made a connection to the node with address

172.18.10.229

So it hasn't formed a cluster correctly.

I'd check your config and that you've specified your IP addresses correctly and that they correspond to the ip addresses of the cards on those boxes.

[timfox]

Ok I spotted the error in your config:

in node B you have:

  <connector name="netty">
         <factory-class>org.hornetq.integration.transports.netty.NettyConnectorFactory</factory-class>
         <param key="host"  value="${hornetq.remoting.netty.host:172.18.10.229}"/>
         <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
      </connector>

but it should be

  <connector name="netty">
         <factory-class>org.hornetq.integration.transports.netty.NettyConnectorFactory</factory-class>
         <param key="host"  value="${hornetq.remoting.netty.host:172.18.10.63}"/>
         <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
      </connector>

I guess this is a copy and paste error

[sv_srinivaas]

Thanks, you are awesome, problem resolved. I know I can count on you guys.