HornetQ fails to connect server then node crash

[shailendra.soni]

Environmental  Information

--------------------------------------------

Jboss Application Server :- 7.2 Final

Operating System :- Solaris Sparc

JVM :- Sun Java 1.6_25 64 bit.

RAM :- 32 GB

@@

Recently, we have migrated from jboss 6.1 to jboss 7.2 Final in our two solaris servers.

We have 76.12 server and 76.13 servers in same network and jboss 7.2 on each server. Each jboss 7.2 is having different application and different queue. Also We are configured jboss 7.2 as domain mode with 5 instance on each server.

So like

Server :- 76.12 We have jboss 7.2 and application name is SRSEar and one Queue name is SRSSchedule. This application itself as client and MDB, client send message to queue and MDB consume it and process it. And We have 5 domain instaces.

Server :- 76.13 We have jboss 7.2 and application name is IMSEar and one Queue name is IMSSchedule. This application itself as client and MDB, client send message to queue and MDB consume it and process it. And We have 5 domain instaces.

Now problem statements.

1. We have configured hornetQ as cluster mode in both jboss servers. In this Jboss 7.2 create bridge between 76.12 and 76.13 and then after some time any of the node got crashed with given message…

10:54:15,416 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

10:54:15,417 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session c6a1c0d0-7ba0-11e3-8169-f7da8b19e073

10:54:15,512 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222158: Cleared up resources for session c6a1c0d0-7ba0-11e3-8169-f7da8b19e073

Once above error comes, then jboss 7.2 have different kind of error comes. Like,

11:26:05,063 WARN [org.hornetq.jms.server] (Thread-492 (HornetQ-client-global-threads-1881806294)) HQ122017: Notified of connection failure in xa recovery connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=dc132230-7b9d-11e3-b692-e5ebefde02b7, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryGroupConfiguration=null], connectorConfig=TransportConfiguration(name=dc132230-7b9d-11e3-b692-e5ebefde02b7, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0, backupConfig=null] will attempt reconnect on next pass: HornetQException[errorType=NOT_CONNECTED message=HQ119025: Channel disconnected]

12:01:55,637 WARN [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMERR: javax.transaction.xa.XAException: Error trying to connect to any providers for xa recovery

                at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:314) [hornetq-jms-server-2.3.0.CR1.jar:]

My first question :- How do I stop bridge? As I do not want to jboss 7.2 connect to another jboss 7.2 in same network.

1. On overcome above problem, I have also removed cluster mode in horentQ and then also I have remove “dg-group1” reference from cluster-connections ( I think if cluster false then HornetQ will not use cluster-connections, may be silly change) , after that I have removed data and tmp directory of each and every nodes. And after restart the server, I got the following exception, after few hours my two nodes are crashed.

10:54:15,416 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

10:54:15,417 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session c6a1c0d0-7ba0-11e3-8169-f7da8b19e073

10:54:15,512 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222158: Cleared up resources for session c6a1c0d0-7ba0-11e3-8169-f7da8b19e073

So I need to two solutions,

1. How do I stop bridge so it can work as cluster on same machine?
2. And how do I stop the exception comes in point 2?

Please let me know if you would require more detail.

[ataylor]

firstly the warnings you see are the invm connections timing out, this is usually because you have too much load on them and the pings dont get received in time, you should up the ttl etc like the warning said or tune your server to handle the load better.

How do I stop bridge so it can work as cluster on same machine?

i dont really know what you are asking here could you be clearer?

[shailendra.soni]

Here I have two jboss 7 on two different servers with different applications, different MDB and different Queue but in same network. So whathappen, both servers are trying to connect each other if cluster mode true in horentQ. But if I make it cluster mode false, then if whatever node client issue the messages ,same node MDB is invoke even to much messages are issued. Now if cluster mode is true , then it is distributed all the servers in same network.

Also FYI ;- my production servers has not much loaded with messages.

[ataylor]

sorry, i still don't understand what you are exactly asking

[shailendra.soni]

I have two jboss 7.2 application server on different machine. One is on 10.0.76.12 Solaris Server and deploy one EAR and I have configured SRSQueue in domain.xml. Another jboss 7.2 is on 10.0.76.13 Solaris server and deploy one application and configured Queue with different name.

Now, In my application , we have one EJB which send the message to the Queue and I have MDB in same application which is process this message.

Now, as I was configured hornetQ as cluster = true then both jboss 7.2 is trying to connect each other.  I hope below INFO would help you,

17:53:20,300 INFO  [org.hornetq.core.server] (Thread-7 (HornetQ-server-HornetQServerImpl::serverUUID=fc6b9ad8-753a-11e3-a815-658ecc3766e0-1759627892)) HQ221032: Bridge ClusterConnectionBridge@1bb5d53a [name=sf.my-cluster.0c479bed-6001-11e3-a97f-0149870633e4, queue=QueueImpl[name=sf.my-cluster.0c479bed-6001-11e3-a97f-0149870633e4, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=fc6b9ad8-753a-11e3-a815-658ecc3766e0]]@3ec1e10c targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb5d53a [name=sf.my-cluster.0c479bed-6001-11e3-a97f-0149870633e4, queue=QueueImpl[name=sf.my-cluster.0c479bed-6001-11e3-a97f-0149870633e4, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=fc6b9ad8-753a-11e3-a815-658ecc3766e0]]@3ec1e10c targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5548&host=10-0-76-12], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1190640146[nodeUUID=fc6b9ad8-753a-11e3-a815-658ecc3766e0, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5746&host=10-0-76-13, address=jms, server=HornetQServerImpl::serverUUID=fc6b9ad8-753a-11e3-a815-658ecc3766e0])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5548&host=10-0-76-12], discoveryGroupConfiguration=null]] is connected

17:53:29,574 INFO  [org.hornetq.core.server] (Thread-5 (HornetQ-server-HornetQServerImpl::serverUUID=fc6b9ad8-753a-11e3-a815-658ecc3766e0-1759627892)) HQ221032: Bridge ClusterConnectionBridge@1f51c9a0 [name=sf.my-cluster.028815d4-753b-11e3-887e-dd842799dda6, queue=QueueImpl[name=sf.my-cluster.028815d4-753b-11e3-887e-dd842799dda6, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=fc6b9ad8-753a-11e3-a815-658ecc3766e0]]@328c3770 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1f51c9a0 [name=sf.my-cluster.028815d4-753b-11e3-887e-dd842799dda6, queue=QueueImpl[name=sf.my-cluster.028815d4-753b-11e3-887e-dd842799dda6, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=fc6b9ad8-753a-11e3-a815-658ecc3766e0]]@328c3770 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5747&host=10-0-76-13], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1190640146[nodeUUID=fc6b9ad8-753a-11e3-a815-658ecc3766e0, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5746&host=10-0-76-13, address=jms, server=HornetQServerImpl::serverUUID=fc6b9ad8-753a-11e3-a815-658ecc3766e0])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5747&host=10-0-76-13], discoveryGroupConfiguration=null]] is connected

Please let me know if you would like to more detail.

[ataylor]

sorry, i still dont know what your issue actually is, if you have 2 servers and you want them to cluster then configure them as clustered, if you dont want them clustered then configure them as non clustered, i.e. remove the cluster configurations

[shailendra.soni]

Thanks I have figured out. I had disabled cluster is false.

however I am still facing below warning and at the last node is crashed.

Line 8866: 11:21:35,237 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

  Line 8867: 11:21:35,238 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session d4ee854c-7b9d-11e3-af0b-4fd85fbaee2e

  Line 8868: 11:21:35,241 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222158: Cleared up resources for session d4ee854c-7b9d-11e3-af0b-4fd85fbaee2e

I have already implemented connection-ttl  , please see attached domain.xml file. We have used "full-ha" as profile.

[ataylor]

its impossible to say without further info, what are you doing when this warning occurs, is it on startup or does it happen when under load. like i say this only usually happens if the server becomes unresponsive

[shailendra.soni]

My observation is that, once this error/warning comes in anyone of the node then after few hours or days all the other nodes are crashed with hornetQ warning. And then once again restart the domain after removing all the content into Data and Tmp folder of each nodes. But then same problem come.

[ataylor]

try setting the ttl to something really high.

Also investigate what is happening at the point the warning occurs, maybe something else on the server itself is using all of the processing time

[shailendra.soni]

From our server, we had not able to telnet localhost to any port. We had resolved this issue and additionally I have removed cluster and set value 30 min to connection-ttl , after this I haven't faced this issue.