HornetQ cluster message distribution
siharvey81 Aug 6, 2013 6:05 AMHi,
I have a clustered environment with 2 live nodes (node1 and node2). Both node1 and node2 have a queue configured ("sampleQueue") and I am connecting my clients (producer and consumer) by instantiating the JMS connection factory directly and specifying a list of connectors
Currently I am using one producer and one consumer. I would like to be able to have the messages distributed between the nodes in a round-robin fashion. So, the producer should be able to place a message on the queue and HornetQ will then handle distribution between the nodes. The consumer has a message-driven-bean listening on the queue and should be able to receive messages regardless of which node the message was allocated to.
I am having trouble getting message distribution to work correctly. When the producer is started it gets a connection to one of the nodes and messages are only placed on the queue for that that node. Messages are not being distributed between node 1 and node 2. The consumer is also getting a connection to one node and only consuming off that same node's queue. So, if the producer connects to node 1 and the consumer connects to node 2, messages just sit on the queue on node 1 with nothing to pick them up.
In the start-up logs I see the following:
Node 1 (192.168.100-12)
-----------------------
Aug 06, 2013 5:49:25 PM org.hornetq.core.logging.impl.JULLogDelegate info
INFO: Bridge ClusterConnectionBridge@25699176 [name=sf.mycluster.b540fe13-fe6a-11e2-949b-ed0487da14bd, queue=QueueImpl[name=sf.mycluster.b540fe13-fe6a-11e2-949b-ed0487da14bd, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=9f3dc4f0-fe6a-11e2-9782-e99149a726a6]]@35cced2a targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@25699176 [name=sf.mycluster.b540fe13-fe6a-11e2-949b-ed0487da14bd, queue=QueueImpl[name=sf.mycluster.b540fe13-fe6a-11e2-949b-ed0487da14bd, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=9f3dc4f0-fe6a-11e2-9782-e99149a726a6]]@35cced2a targetConnector=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192.168.100-13], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@647970853[nodeUUID=9f3dc4f0-fe6a-11e2-9782-e99149a726a6, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192.168.100-12, address=jms, server=HornetQServerImpl::serverUUID=9f3dc4f0-fe6a-11e2-9782-e99149a726a6])) [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192.168.100-13], discoveryGroupConfiguration=null]] is connected
Node 2 (192.168.100-13)
-----------------------
Aug 06, 2013 5:49:25 PM org.hornetq.core.logging.impl.JULLogDelegate info
INFO: Bridge ClusterConnectionBridge@30578ed2 [name=sf.mycluster.9f3dc4f0-fe6a-11e2-9782-e99149a726a6, queue=QueueImpl[name=sf.mycluster.9f3dc4f0-fe6a-11e2-9782-e99149a726a6, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=b540fe13-fe6a-11e2-949b-ed0487da14bd]]@446195c9 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@30578ed2 [name=sf.mycluster.9f3dc4f0-fe6a-11e2-9782-e99149a726a6, queue=QueueImpl[name=sf.mycluster.9f3dc4f0-fe6a-11e2-9782-e99149a726a6, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=b540fe13-fe6a-11e2-949b-ed0487da14bd]]@446195c9 targetConnector=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192.168.100-12], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1628186119[nodeUUID=b540fe13-fe6a-11e2-949b-ed0487da14bd, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=0-0-0-0, address=jms, server=HornetQServerImpl::serverUUID=b540fe13-fe6a-11e2-949b-ed0487da14bd])) [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=192.168.100-12], discoveryGroupConfiguration=null]] is connected
However, on one node (I have seen this on both nodes but haven't tied down the scenario when this happens yet) I get the following:
Aug 06, 2013 5:49:47 PM org.hornetq.core.loggstartuping.impl.JULLogDelegate warn
WARNING: Connection failure has been detected: Did not receive data from /192.168.100.13:48703. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]
1. From my understanding of HornetQ I think my requirements should work. Could you confirm this is correct?
2. Are you able to make any suggestions as to why message distribution is not working? I have added some details form my configuration at the end of this message.
Many thanks,
Simon
Configuration:
hornetq-configuration.xml
------------------------------------
<clustered>true</clustered>
<connectors>
<connector name="netty">
<factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
<param key="host" value="0.0.0.0"/>
<param key="port" value="5445"/>
</connector>
<connector name="netty-throughput">
<factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
<param key="host" value="0.0.0.0"/>
<param key="port" value="5446"/>
<param key="batch-delay" value="50"/>
</connector>
</connectors>
<acceptors>
<acceptor name="netty">
<factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
<param key="host" value="0.0.0.0"/>
<param key="port" value="5445"/>
</acceptor>
<acceptor name="netty-throughput">
<factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
<param key="host" value="0.0.0.0"/>
<param key="port" value="5446"/>
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
</acceptor>
</acceptors>
<cluster-user>platform-cluster</cluster-user>
<cluster-password>platform-cluster</cluster-password>
<broadcast-groups>
<broadcast-group name="platform-bg">
<group-address>239.195.50.10</group-address>
<group-port>9876</group-port>
<broadcast-period>5000</broadcast-period>
<connector-ref>netty</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="platform-dg">
<group-address>239.195.50.10</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="mycluster">
<address>jms</address>
<connector-ref>netty</connector-ref>
<discovery-group-ref discovery-group-name="platform-dg"/>
</cluster-connection>
</cluster-connections>
<address-settings>
<!--default for catch all -->
<address-setting match="#">
<max-delivery-attempts>6</max-delivery-attempts>
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>10000</redelivery-delay>
<max-size-bytes>104857600</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<redistribution-delay>0</redistribution-delay>
</address-setting>
</address-settings>
hornetq-jms.xml
-----------------------
<connection-factory name="NettyConnectionFactory">
<xa>true</xa>
<connectors>
<connector-ref connector-name="netty"/>
</connectors>
<entries>
<entry name="/XAConnectionFactory"/>
</entries>
</connection-factory>
<connection-factory name="NettyConnectionFactory">
<xa>false</xa>
<connectors>
<connector-ref connector-name="netty"/>
</connectors>
<entries>
<entry name="/ConnectionFactory"/>
</entries>
</connection-factory>
<connection-factory name="NettyThroughputConnectionFactory">
<xa>true</xa>
<connectors>
<connector-ref connector-name="netty-throughput"/>
</connectors>
<entries>
<entry name="/XAThroughputConnectionFactory"/>
</entries>
</connection-factory>
<connection-factory name="NettyThroughputConnectionFactory">
<xa>false</xa>
<connectors>
<connector-ref connector-name="netty-throughput"/>
</connectors>
<entries>
<entry name="/ThroughputConnectionFactory"/>
</entries>
</connection-factory>
<queue name="DLQ">
<entry name="/queue/DLQ" />
</queue>
<queue name="CallbackQueue">
<entry name="/queue/CallbackQueue" />
</queue>
<queue name="ExpiryQueue">
<entry name="/queue/ExpiryQueue" />
</queue>