12 Replies Latest reply on Jan 6, 2010 11:53 PM by srinivaas venkat

    Problem with load balancing mdbs in hornetq cluster

    srinivaas venkat Novice

      Hi, I have two issues with MDB and Hornetq cluster. It’ll be of great help if you can spare few mins of your valuable time as I’ve not been able to resolve the following issues.

       

       

      I have gone through the cluster and mdb related portions of Hornetq user manual but still not able achieve load balancing with mdbs in cluster. Also I’m able to successfully execute the mdb and clustered a queue example that comes with Hornetq.

        

      Version: Hornetq CR2OS:

      Windows XP (SP2)

      JBoss: 5.0.0. GA

       

      1.     First issue is with the load balancing of messages in Hornetq cluster with (EJB3) MDBs where I have only the queue servers clustered and the mdbs deployed in third node which is NOT clustered.

       

       

      Steps to reproduce the issue

       

       

      a. Created a cluster of two queue nodes nodeA and nodeB with a queue defined (as sampleQueue).

       

       

      b. Then I deployed two mdbs MDB1 and MDB2 both in the third node C (which is not part of the above cluster as I had set the clustered param to false for this node in hornetq-configuration.xml) where MDB1 consumes messages from sampleQueue of node A and MDB2 from sampleQueue of node B.

       

      Also I could see that the consumer count for sampleQueue as 15 in both nodeA and B (15 is default pool size for mdb).

       

      c. Then from a client application (hosted in tomcat) I sent 30 messages to sampleQueue in node A

       

      What I could see from jmx console is that all the 30 messages were sent to node A and none was sent to node B and all the 30 messages were consumed by MDB1 (of nodeA) itself.

       

      Is this the expected behavior? I was expecting 15 messages to be consumed by MDB1 from sampleQueue of nodeA and 15 messages to be consumed by MDB2 from sampleQueue of nodeB.

       

      I’ve attached the xmls for all the 3 nodes.

       

      2. Now the second issue is, when I set the clustered param to true in node C (in hornetq-configuration.xml) where the MDBs are hosted (because in PROD the MDBs will also be deployed in a separate cluster) I get the below exception during the nodeC startup.

       

      ERROR [ClusterConnectionImpl] Failed to handle messagejava.lang.IllegalStateException: Cannot find binding for jms.queue.sampleQueue96bf7979-f9f8-11de-bf1c-001a4d1349b0       at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerCreated(ClusterConnectionImpl.java:758)            at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:552)            at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:767) at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:45)           at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:885)           at java.lang.Thread.run(Thread.java:619)

       

      Also I have gone through the below url and made all the necessary changes to avoid any overlapping of Queue and MDB clusters (as they are in the same network). Pls let me know if i'm missing any configuration to run multiple clusters in the same network.

       

       

      http://community.jboss.org/wiki/TwoClustersontheSameNetwork

       

       

        • 1. Re: Problem with load balancing mdbs in hornetq cluster
          Andy Taylor Master
          It looks like you havent balanced your sending client. take a look at the client-side-load-balancing-example. If you are only creating one connection in your client all messages will be sent to the same node.
          • 2. Re: Problem with load balancing mdbs in hornetq cluster
            srinivaas venkat Novice

            Andy, I was under the assumption that if I have the below code in hornetq-configuration.xml (as mentioned in the clustered-queue example) for both the queue nodes then Hornetq would automatically load balance the messages across all the nodes in cluster.

             

            Also i've checked both the nodes and both are having consumers attached to the queue. Please correct me if i'm wrong

             

            <cluster-connection name="my-cluster">
                   
            <address>jms</address>
                   
            <retry-interval>500</retry-interval>
                   
            <use-duplicate-detection>true</use-duplicate-detection>
                   
            <forward-when-no-consumers>true</forward-when-no-consumers>
                   
            <max-hops>1</max-hops>
                   
            <discovery-group-ref discovery-group-name="my-discovery-group"/>
                
            </cluster-connection>

             

            Also do you have any inputs regarding my second issue with mdbs deplyed in a cluster?

             

            Thanks

            • 3. Re: Problem with load balancing mdbs in hornetq cluster
              Andy Taylor Master

              yeah, sorry, i misunderstood you when you talked about load balancing, what you are talking about is message distribution.

               

              Firstly i would check that your cluster nodes are discovering each other. you can do this by checking that the distributes queues exist. You can do this by using jconsole and connecting to each server and checking that the clustered queues are there.

               

              after that start your MDB's and make sur ethat each queue has consumers.

               

              It could be that the second MDB's remote consumer isnt being added, hence the error you see.

              • 4. Re: Problem with load balancing mdbs in hornetq cluster
                srinivaas venkat Novice

                I'm able to see the queue in both the nodes A and B through JMX console and I also see the consumer count as 15 in both the nodes queue. I could also see the message ALL Members : 2  in node A server console that confirms that the cluster has been created successfully.

                 

                But I somehow not able to see the messages being distributed to both the queues on a round robin fashion when sent from the client application. It always goes to nodeA and I'm not sure why.

                 

                Again instead of using MDBs if I run the example code that comes with Hornetq where they create two consumers and receive messages alternatively then it works fine.

                • 5. Re: Problem with load balancing mdbs in hornetq cluster
                  Tim Fox Master

                  This should work without you having to load balance at the client, since it should round robin on the server side.

                   

                  Can you invoke the JMX method getNodes() on the ClusterConnection control via the jmx-console. This will tell you how many nodes in the cluster each node thinks there is, so you can tell whether the nodes have formed a cluster properly.

                  • 6. Re: Problem with load balancing mdbs in hornetq cluster
                    Tim Fox Master
                    Also try not deploying the MDBs, then send the message then use the jmx-console to see how many messages are in the queue(s) on nodes A and B
                    • 7. Re: Problem with load balancing mdbs in hornetq cluster
                      Andy Taylor Master

                      In the console check under address/core/jms.queue.sampleQueue/Attributes/queueNames there should be 2!

                      • 8. Re: Problem with load balancing mdbs in hornetq cluster
                        srinivaas venkat Novice

                        Yes I could see the below two entries against queueNames under address/core/sampleQueue in both the nodes

                         

                        jms.queue.sampleQueue6aef5796-faad-11de-ba2f-001a4d1349b0
                        jms.queue.sampleQueue

                         

                        And for nodeA->ClusterConnections->Attribute name->Nodes I see the below entry

                             {a12adaa5-faad-11de-9642-001c23457f19=/172.18.10.229:5445}

                         

                        And for nodeB->ClusterConnections->Attribute name->Nodes I see the below entry

                             {6aef5796-faad-11de-ba2f-001a4d1349b0=/172.18.10.229:5445}

                         

                        Node id of nodeA: 6aef5796-faad-11de-ba2f-001a4d1349b0 

                         

                        Node id of nodeB: a12adaa5-faad-11de-9642-001c23457f19

                         

                        Ip of NodeA: 172.18.10.229

                         

                        Ip of NodeB: 172.18.10.63

                        • 9. Re: Problem with load balancing mdbs in hornetq cluster
                          srinivaas venkat Novice

                          Tried sending 30 messages without deploying the MDBs and all 30 messages went to nodeA and remained unconsumed.

                          • 10. Re: Problem with load balancing mdbs in hornetq cluster
                            Tim Fox Master

                            That doesn't look right to me.

                             

                            According to the output node A has made a connection to the node with address:

                             

                            172.18.10.229

                             

                            And node B also has made a connection to the node with address

                             

                            172.18.10.229

                             

                            So it hasn't formed a cluster correctly.

                             

                            I'd check your config and that you've specified your IP addresses correctly and that they correspond to the ip addresses of the cards on those boxes.

                            • 11. Re: Problem with load balancing mdbs in hornetq cluster
                              Tim Fox Master

                              Ok I spotted the error in your config:

                               

                              in node B you have:

                               

                                <connector name="netty">
                                       <factory-class>org.hornetq.integration.transports.netty.NettyConnectorFactory</factory-class>
                                       <param key="host"  value="${hornetq.remoting.netty.host:172.18.10.229}"/>
                                       <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
                                    </connector>

                               

                              but it should be

                               

                                <connector name="netty">
                                       <factory-class>org.hornetq.integration.transports.netty.NettyConnectorFactory</factory-class>
                                       <param key="host"  value="${hornetq.remoting.netty.host:172.18.10.63}"/>
                                       <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
                                    </connector>

                               

                              I guess this is a copy and paste error

                              • 12. Re: Problem with load balancing mdbs in hornetq cluster
                                srinivaas venkat Novice

                                Thanks, you are awesome, problem resolved. I know I can count on you guys.