4 Replies Latest reply on Aug 7, 2013 5:25 AM by Simon Harvey

    HornetQ cluster message distribution

    Simon Harvey Newbie

      Hi,

       

      I have a clustered environment with 2 live nodes (node1 and node2). Both node1 and node2 have a queue configured ("sampleQueue") and I am connecting my clients (producer and consumer) by instantiating the JMS connection factory directly and specifying a list of connectors

       

      Currently I am using one producer and one consumer. I would like to be able to have the messages distributed between the nodes in a round-robin fashion. So, the producer should be able to place a message on the queue and HornetQ will then handle distribution between the nodes. The consumer has a message-driven-bean listening on the queue and should be able to receive messages regardless of which node the message was allocated to.

       

      I am having trouble getting message distribution to work correctly. When the producer is started it gets a connection to one of the nodes and messages are only placed on the queue for that that node. Messages are not being distributed between node 1 and node 2. The consumer is also getting a connection to one node and only consuming off that same node's queue. So, if the producer connects to node 1 and the consumer connects to node 2, messages just sit on the queue on node 1 with nothing to pick them up.

       

      In the start-up logs I see the following:

       

      Node 1 (192.168.100-12)

      -----------------------

      Aug 06, 2013 5:49:25 PM org.hornetq.core.logging.impl.JULLogDelegate info

      INFO: Bridge ClusterConnectionBridge@25699176 [name=sf.mycluster.b540fe13-fe6a-11e2-949b-ed0487da14bd, queue=QueueImpl[name=sf.mycluster.b540fe13-fe6a-11e2-949b-ed0487da14bd, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=9f3dc4f0-fe6a-11e2-9782-e99149a726a6]]@35cced2a targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@25699176 [name=sf.mycluster.b540fe13-fe6a-11e2-949b-ed0487da14bd, queue=QueueImpl[name=sf.mycluster.b540fe13-fe6a-11e2-949b-ed0487da14bd, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=9f3dc4f0-fe6a-11e2-9782-e99149a726a6]]@35cced2a targetConnector=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192.168.100-13], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@647970853[nodeUUID=9f3dc4f0-fe6a-11e2-9782-e99149a726a6, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192.168.100-12, address=jms, server=HornetQServerImpl::serverUUID=9f3dc4f0-fe6a-11e2-9782-e99149a726a6])) [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192.168.100-13], discoveryGroupConfiguration=null]] is connected

       

       

      Node 2 (192.168.100-13)

      -----------------------

      Aug 06, 2013 5:49:25 PM org.hornetq.core.logging.impl.JULLogDelegate info

      INFO: Bridge ClusterConnectionBridge@30578ed2 [name=sf.mycluster.9f3dc4f0-fe6a-11e2-9782-e99149a726a6, queue=QueueImpl[name=sf.mycluster.9f3dc4f0-fe6a-11e2-9782-e99149a726a6, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=b540fe13-fe6a-11e2-949b-ed0487da14bd]]@446195c9 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@30578ed2 [name=sf.mycluster.9f3dc4f0-fe6a-11e2-9782-e99149a726a6, queue=QueueImpl[name=sf.mycluster.9f3dc4f0-fe6a-11e2-9782-e99149a726a6, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=b540fe13-fe6a-11e2-949b-ed0487da14bd]]@446195c9 targetConnector=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192.168.100-12], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1628186119[nodeUUID=b540fe13-fe6a-11e2-949b-ed0487da14bd, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=0-0-0-0, address=jms, server=HornetQServerImpl::serverUUID=b540fe13-fe6a-11e2-949b-ed0487da14bd])) [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192.168.100-12], discoveryGroupConfiguration=null]] is connected

       

       

      However, on one node (I have seen this on both nodes but haven't tied down the scenario when this happens yet) I get the following:

       

      Aug 06, 2013 5:49:47 PM org.hornetq.core.loggstartuping.impl.JULLogDelegate warn

      WARNING: Connection failure has been detected: Did not receive data from /192.168.100.13:48703. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

       

       

      1. From my understanding of HornetQ I think my requirements should work. Could you confirm this is correct?

       

      2. Are you able to make any suggestions as to why message distribution is not working? I have added some details form my configuration at the end of this message.

       

       

      Many thanks,

       

      Simon

       

       

      Configuration:

       

      hornetq-configuration.xml

      ------------------------------------

       

         <clustered>true</clustered>

       

         <connectors>

            <connector name="netty">

               <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

               <param key="host"  value="0.0.0.0"/>

               <param key="port"  value="5445"/>

            </connector>

           

            <connector name="netty-throughput">

               <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

               <param key="host"  value="0.0.0.0"/>

               <param key="port"  value="5446"/>

               <param key="batch-delay" value="50"/>

            </connector>

         </connectors>

       

         <acceptors>

            <acceptor name="netty">

               <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

               <param key="host"  value="0.0.0.0"/>

               <param key="port"  value="5445"/>

            </acceptor>

           

            <acceptor name="netty-throughput">

               <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

               <param key="host"  value="0.0.0.0"/>

               <param key="port"  value="5446"/>

               <param key="batch-delay" value="50"/>

               <param key="direct-deliver" value="false"/>

            </acceptor>

         </acceptors>

       

         <cluster-user>platform-cluster</cluster-user>

         <cluster-password>platform-cluster</cluster-password>

       

         <broadcast-groups>

            <broadcast-group name="platform-bg">

               <group-address>239.195.50.10</group-address>

               <group-port>9876</group-port>

               <broadcast-period>5000</broadcast-period>

               <connector-ref>netty</connector-ref>

            </broadcast-group>

         </broadcast-groups>

       

         <discovery-groups>

            <discovery-group name="platform-dg">

               <group-address>239.195.50.10</group-address>

               <group-port>9876</group-port>

               <refresh-timeout>10000</refresh-timeout>

            </discovery-group>

         </discovery-groups>

        

         <cluster-connections>

            <cluster-connection name="mycluster">

               <address>jms</address>          

               <connector-ref>netty</connector-ref>

                      <discovery-group-ref discovery-group-name="platform-dg"/>

            </cluster-connection>

         </cluster-connections>

       

         <address-settings>

                <!--default for catch all -->

                <address-setting match="#">

                          <max-delivery-attempts>6</max-delivery-attempts>

                          <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                          <expiry-address>jms.queue.ExpiryQueue</expiry-address>

                          <redelivery-delay>10000</redelivery-delay>

                          <max-size-bytes>104857600</max-size-bytes>

                          <message-counter-history-day-limit>10</message-counter-history-day-limit>

                          <address-full-policy>PAGE</address-full-policy>

                          <redistribution-delay>0</redistribution-delay>

                </address-setting>

         </address-settings>

       

       

      hornetq-jms.xml

      -----------------------

       

         <connection-factory name="NettyConnectionFactory">

            <xa>true</xa>

            <connectors>

               <connector-ref connector-name="netty"/>

            </connectors>

            <entries>

               <entry name="/XAConnectionFactory"/>

            </entries>

         </connection-factory>

        

         <connection-factory name="NettyConnectionFactory">

            <xa>false</xa>

            <connectors>

               <connector-ref connector-name="netty"/>

            </connectors>

            <entries>

               <entry name="/ConnectionFactory"/>

            </entries>

         </connection-factory>

        

         <connection-factory name="NettyThroughputConnectionFactory">

            <xa>true</xa>

            <connectors>

               <connector-ref connector-name="netty-throughput"/>

            </connectors>

            <entries>

               <entry name="/XAThroughputConnectionFactory"/>

            </entries>

         </connection-factory>

        

         <connection-factory name="NettyThroughputConnectionFactory">

            <xa>false</xa>

            <connectors>

               <connector-ref connector-name="netty-throughput"/>

            </connectors>

            <entries>

               <entry name="/ThroughputConnectionFactory"/>

            </entries>

         </connection-factory>

       

         <queue name="DLQ">

                <entry name="/queue/DLQ" />

         </queue>

       

         <queue name="CallbackQueue">

                <entry name="/queue/CallbackQueue" />

         </queue>

       

         <queue name="ExpiryQueue">

                <entry name="/queue/ExpiryQueue" />

         </queue>

        • 1. Re: HornetQ cluster message distribution
          Simon Harvey Newbie

          Apologies here is the configuration:

           

          Configuration:

           

           

          hornetq-configuration.xml

          -------------------------

           

           

             <clustered>true</clustered>

           

           

             <connectors>

                <connector name="netty">

                   <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                   <param key="host"  value="0.0.0.0"/>

                   <param key="port"  value="5445"/>

                </connector>

               

                <connector name="netty-throughput">

                   <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                   <param key="host"  value="0.0.0.0"/>

                   <param key="port"  value="5446"/>

                   <param key="batch-delay" value="50"/>

                </connector>

             </connectors>

           

           

             <acceptors>

                <acceptor name="netty">

                   <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

                   <param key="host"  value="0.0.0.0"/>

                   <param key="port"  value="5445"/>

                </acceptor>

               

                <acceptor name="netty-throughput">

                   <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

                   <param key="host"  value="0.0.0.0"/>

                   <param key="port"  value="5446"/>

                   <param key="batch-delay" value="50"/>

                   <param key="direct-deliver" value="false"/>

                </acceptor>

             </acceptors>

           

           

             <cluster-user>platform-cluster</cluster-user>

             <cluster-password>platform-cluster</cluster-password>

           

             <broadcast-groups>

                <broadcast-group name="platform-bg">

                   <group-address>239.195.50.10</group-address>

                   <group-port>9876</group-port>

                   <broadcast-period>5000</broadcast-period>

                   <connector-ref>netty</connector-ref>

                </broadcast-group>

             </broadcast-groups>

           

           

             <discovery-groups>

                <discovery-group name="platform-dg">

                   <group-address>239.195.50.10</group-address>

                   <group-port>9876</group-port>

                   <refresh-timeout>10000</refresh-timeout>

                </discovery-group>

             </discovery-groups>

            

             <cluster-connections>

                <cluster-connection name="mycluster">

                   <address>jms</address>          

                   <connector-ref>netty</connector-ref>

                          <discovery-group-ref discovery-group-name="platform-dg"/>

                </cluster-connection>

             </cluster-connections>

           

           

             <address-settings>

                    <!--default for catch all -->

                    <address-setting match="#">

                              <max-delivery-attempts>6</max-delivery-attempts>

                              <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                              <expiry-address>jms.queue.ExpiryQueue</expiry-address>

                              <redelivery-delay>10000</redelivery-delay>

                              <max-size-bytes>104857600</max-size-bytes>

                              <message-counter-history-day-limit>10</message-counter-history-day-limit>

                              <address-full-policy>PAGE</address-full-policy>

                              <redistribution-delay>0</redistribution-delay>

                    </address-setting>

             </address-settings>

           

           

           

           

          hornetq-jms.xml

          ---------------

           

           

             <connection-factory name="NettyConnectionFactory">

                <xa>true</xa>

                <connectors>

                   <connector-ref connector-name="netty"/>

                </connectors>

                <entries>

                   <entry name="/XAConnectionFactory"/>

                </entries>

             </connection-factory>

            

             <connection-factory name="NettyConnectionFactory">

                <xa>false</xa>

                <connectors>

                   <connector-ref connector-name="netty"/>

                </connectors>

                <entries>

                   <entry name="/ConnectionFactory"/>

                </entries>

             </connection-factory>

            

             <connection-factory name="NettyThroughputConnectionFactory">

                <xa>true</xa>

                <connectors>

                   <connector-ref connector-name="netty-throughput"/>

                </connectors>

                <entries>

                   <entry name="/XAThroughputConnectionFactory"/>

                </entries>

             </connection-factory>

            

             <connection-factory name="NettyThroughputConnectionFactory">

                <xa>false</xa>

                <connectors>

                   <connector-ref connector-name="netty-throughput"/>

                </connectors>

                <entries>

                   <entry name="/ThroughputConnectionFactory"/>

                </entries>

             </connection-factory>

           

           

             <queue name="DLQ">

                    <entry name="/queue/DLQ" />

             </queue>

           

           

             <queue name="CallbackQueue">

                    <entry name="/queue/CallbackQueue" />

             </queue>

           

           

             <queue name="ExpiryQueue">

                    <entry name="/queue/ExpiryQueue" />

             </queue>

          • 2. Re: HornetQ cluster message distribution
            Andy Taylor Master

            can you check in the console to see if the bridges are connected properly, check the bridge itself and also the consumer count on the queue.

             

            you are using 0-0-0-0 for the connector so this will be the initial host used when the bridge connects, which will always be iytself until it ahas received the topology from thge cluster. can yoy try setting this explicitly.

            • 3. Re: HornetQ cluster message distribution
              Simon Harvey Newbie

              Hi Andy, thanks for getting back to me.

               

              I am sorry I am not sure how to check if the bridges are connected properly. In the log file the only entry related to this are the ones I included in the original post for the ClusterConnectionBridge. Using the JMX console I can see that the consumer count on the queue is one for node 1 and zero for node 2.

               

              I changed the connection address from 0.0.0.0 to the IP address of the node and observed some slightly different behavior. If no consumers are started, messages are produced evenly across both nodes. However, once I start a consumer and it gets associated to one of the nodes, all messages are then sent to the queue on that node. So, if the consumer is associated with node 1, all messages will go via node 1 & node 2 will not receive any messages.

               

              If HornetQ is able to distribute messages on a queue across all nodes in the cluster, am I right in thinking a consumer should be able to consume from a queue off all nodes in that cluster?

              • 4. Re: HornetQ cluster message distribution
                Simon Harvey Newbie

                Hi,

                 

                To follow on from my previous post I have set the following in the configuration for the cluster-connection:

                 

                <forward-when-no-consumers>true</forward-when-no-consumers>

                 

                Now I can see messages being sent to both nodes even though only one is being consumed off. So...my question now is really - can I get my consumer to consume from the queue off both node 1 and node 2 in my cluster?