Connection failure and lack of managed connections relation
antonz Dec 21, 2016 8:05 AMHello!
I use HornetQ 2.3.21.Final running on EAP 6.3 I begin to get warns within a couple of days (~35) which look like the following:
2016-12-14 05:33:02,593 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
2016-12-14 05:33:02,593 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session 6cddfb8f-c1b4-11e6-957d-9527f5eabbff
2016-12-14 05:33:02,593 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session 6cddfb8f-c1b4-11e6-957d-9527f5eabbff
2016-12-14 05:33:02,594 WARN [org.hornetq.jms.server] (Thread-17566 (HornetQ-client-global-threads-353003541)) HQ122014: Notified of connection failure in xa recovery connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=c2c83d93-bd68-11e6-957d-9527f5eabbff, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryGroupConfiguration=null], connectorConfig=TransportConfiguration(name=c2c83d93-bd68-11e6-957d-9527f5eabbff, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0, backupConfig=null] will attempt reconnect on next pass: HornetQException[errorType=NOT_CONNECTED message=HQ119006: Channel disconnected]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:425) [hornetq-core-client-2.3.21.Final-redhat-1.jar:2.3.21.Final-redhat-1]
at org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:214) [hornetq-server-2.3.21.Final-redhat-1.jar:2.3.21.Final-redhat-1]
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105) [hornetq-core-client-2.3.21.Final-redhat-1.jar:2.3.21.Final-redhat-1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_112]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_112]
Everything works, but In a moment the queue stops working and it starts to generate numerous errors (hundreds of them):
2016-12-14 08:44:31,343 ERROR [org.hornetq.ra] (http-/0.0.0.0:9002-297) HQ154002: Could not create session: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/JmsXA
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:414) [ironjacamar-core-impl-1.0.27.Final-redhat-1.jar:1.0.27.Final-redhat-1]
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) [ironjacamar-core-impl-1.0.27.Final-redhat-1.jar:1.0.27.Final-redhat-1]
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) [ironjacamar-core-impl-1.0.27.Final-redhat-1.jar:1.0.27.Final-redhat-1]
at org.hornetq.ra.HornetQRASessionFactoryImpl.allocateConnection(HornetQRASessionFactoryImpl.java:832)
at org.hornetq.ra.HornetQRASessionFactoryImpl.createSession(HornetQRASessionFactoryImpl.java:465)
...
Caused by: javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])
I suppose that happens because all connections in pool are used by someone and never released. The question is: may the first type of error be the cause of the second one? Or I get something wrong and these errors are not related?
App server works well after restart but I need to know what causes this to happen. App is sometimes migrated by VMWare VCenter. May that cause HornetQ stop working?