1 Reply Latest reply on Mar 5, 2018 4:21 PM by srgibbs99

    Issue with hornetq-server cluster-connections

    antonyjohnson
    antonyjohnson Feb 26, 2017 8:09 AM

      We have two node cluster setup.

       

      When JBoss is stopped in the node-2, the cluster connections from the node1 which points to the node2 are getting closed properly due to timeout.  Then later, the JBoss is started in the node-2, In this case, as per the below configuration all cluster-connections to be re-initiated from the node-1 to node-2, but in our setup, the my-cluster-topics connections alone look fine from the node1 to node2 whereas the other 2 cluster connections (my-cluster-queues & my-cluster-connection-xxxxx) are not initiated

       

       

       

      configuration snippet:

       

                                 

                        <cluster-connections>
                          <cluster-connection name="my-cluster-topics">
                              <address>jms.topic.clustered</address>
                              <reconnect-attempts>5</reconnect-attempts>
                              <connector-ref>netty</connector-ref>
                              <discovery-group-ref discovery-group-name="dg-group1"/>
                          </cluster-connection>
                          <cluster-connection name="my-cluster-queues">
                              <address>jms.queue.clustered</address>
                              <reconnect-attempts>5</reconnect-attempts>
                              <connector-ref>netty</connector-ref>
                              <discovery-group-ref discovery-group-name="dg-group1"/>
                          </cluster-connection>
                          <cluster-connection name="my-cluster-connection-xxxxx">
                              <address>jms.topic.connection-xxxxx</address>
                              <reconnect-attempts>5</reconnect-attempts>
                              <connector-ref>netty</connector-ref>
                              <discovery-group-ref discovery-group-name="dg-group1"/>
                          </cluster-connection>
                      </cluster-connections>

       

       

       

       

      horenq.log snippet

       

       

      2017-01-18 09:44:46,267 WARN  [org.hornetq.core.server] (Thread-25 (HornetQ-server-HornetQServerImpl::serverUUID=e6ea7faf-7f39-11e6-9a86-e55f9479763b-657640790)) HQ222101: Bridge sf.my-cluster-queues.27b44362-dd61-11e6-b054-093a43823dea achieved 6 maxattempts=5 it will stop retrying to reconnect

      2017-01-18 09:44:46,275 INFO  [org.hornetq.core.server] (Thread-23 (HornetQ-server-HornetQServerImpl::serverUUID=e6ea7faf-7f39-11e6-9a86-e55f9479763b-657640790)) HQ221029: stopped bridge sf.my-cluster-queues.27b44362-dd61-11e6-b054-093a43823dea

      2017-01-18 09:44:46,280 WARN  [org.hornetq.core.server] (Thread-0 (HornetQ-server-HornetQServerImpl::serverUUID=e6ea7faf-7f39-11e6-9a86-e55f9479763b-657640790)) HQ222101: Bridge sf.my-cluster-topics.27b44362-dd61-11e6-b054-093a43823dea achieved 6 maxattempts=5 it will stop retrying to reconnect

      2017-01-18 09:44:46,295 WARN  [org.hornetq.core.server] (Thread-19 (HornetQ-server-HornetQServerImpl::serverUUID=e6ea7faf-7f39-11e6-9a86-e55f9479763b-657640790)) HQ222101: Bridge sf.my-cluster-connection-xxxxx.27b44362-dd61-11e6-b054-093a43823dea achieved 6 maxattempts=5 it will stop retrying to reconnect

      2017-01-18 09:44:46,296 INFO  [org.hornetq.core.server] (Thread-19 (HornetQ-server-HornetQServerImpl::serverUUID=e6ea7faf-7f39-11e6-9a86-e55f9479763b-657640790)) HQ221029: stopped bridge sf.my-cluster-connection-xxxxx.27b44362-dd61-11e6-b054-093a43823dea

      2017-01-18 09:44:46,300 INFO  [org.hornetq.core.server] (Thread-12 (HornetQ-server-HornetQServerImpl::serverUUID=e6ea7faf-7f39-11e6-9a86-e55f9479763b-657640790)) HQ221029: stopped bridge sf.my-cluster-topics.27b44362-dd61-11e6-b054-093a43823dea

      2017-01-18 09:45:37,027 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from /172.30.157.80:40841. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

      2017-01-18 09:45:37,028 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session 2dcbb645-dd61-11e6-b054-093a43823dea

      2017-01-18 09:45:37,029 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session 2dcbb645-dd61-11e6-b054-093a43823dea

      2017-01-18 09:45:37,032 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from /172.30.157.80:40837. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

      2017-01-18 09:45:37,032 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session 2da4a63d-dd61-11e6-b054-093a43823dea

      2017-01-18 09:45:37,032 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session 2da4a63d-dd61-11e6-b054-093a43823dea

      2017-01-18 09:45:37,033 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from /172.30.157.80:40840. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

      2017-01-18 09:45:37,033 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session 2dbac651-dd61-11e6-b054-093a43823dea

      2017-01-18 09:45:37,033 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session 2dbac651-dd61-11e6-b054-093a43823dea

      2017-01-18 09:46:39,194 WARN  [org.hornetq.core.client] (hornetq-discovery-group-thread-dg-group1) HQ212034: There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=27b44362-dd61-11e6-b054-093a43823dea

      2017-01-18 09:46:39,191 WARN  [org.hornetq.core.client] (hornetq-discovery-group-thread-dg-group1) HQ212034: There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=27b44362-dd61-11e6-b054-093a43823dea

      2017-01-18 09:46:39,191 WARN  [org.hornetq.core.client] (hornetq-discovery-group-thread-dg-group1) HQ212034: There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=27b44362-dd61-11e6-b054-093a43823dea

      2017-01-18 09:46:43,952 INFO  [org.hornetq.core.server] (Thread-21 (HornetQ-server-HornetQServerImpl::serverUUID=e6ea7faf-7f39-11e6-9a86-e55f9479763b-657640790)) HQ221027: Bridge ClusterConnectionBridge@4e429e36 [name=sf.my-cluster-topics.27b44362-dd61-11e6-b054-093a43823dea, queue=QueueImpl[name=sf.my-cluster-topics.27b44362-dd61-11e6-b054-093a43823dea, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=e6ea7faf-7f39-11e6-9a86-e55f9479763b]]@26dc4f05 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@4e429e36 [name=sf.my-cluster-topics.27b44362-dd61-11e6-b054-093a43823dea, queue=QueueImpl[name=sf.my-cluster-topics.27b44362-dd61-11e6-b054-093a43823dea, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=e6ea7faf-7f39-11e6-9a86-e55f9479763b]]@26dc4f05 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=172-30-157-80], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@888614329[nodeUUID=e6ea7faf-7f39-11e6-9a86-e55f9479763b, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=172-30-157-88, address=jms.topic.clustered, server=HornetQServerImpl::serverUUID=e6ea7faf-7f39-11e6-9a86-e55f9479763b])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=172-30-157-80], discoveryGroupConfiguration=null]] is connected