3 Replies Latest reply on May 16, 2013 11:29 AM by Justin Bertram

    JBoss instability when hornetq idle

    rhinmass Newbie

      I have a jboss application that consists primarily of queue listeners.  I am using jboss-as-7.1.1.Final.  I've found that when it sits idle for a long period of time (i.e. overnight) it becomes unstable and must be restarted.  Apparently a connection is being left open and timing out, but I don't know where to manage the opening/closing of the in-vm connection.

       

      Here is what I see in the server.log

       

      08:11:10,278 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

      08:11:10,278 WARN  [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-262 (HornetQ-client-global-threads-1293640513)) Notified of connection failure in xa recovery connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0], discoveryGroupConfiguration=null], connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0, backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2 message=Channel disconnected]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380) [hornetq-core-2.2.13.Final.jar:]

                at org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203) [hornetq-core-2.2.13.Final.jar:]

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.13.Final.jar:]

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_24]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_24]

                at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]

       

       

      08:13:18,462 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMERR: javax.transaction.xa.XAException: Error trying to connect to any providers for xa recovery

                at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:275) [hornetq-jms-2.2.13.Final.jar:]

                at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.recover(HornetQXAResourceWrapper.java:77) [hornetq-jms-2.2.13.Final.jar:]

                at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecovery(XARecoveryModule.java:503) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:471) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:385) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:166) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-4.16.2.Final.jar:]

      Caused by: java.lang.IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)

                at org.hornetq.core.client.impl.ServerLocatorImpl.assertOpen(ServerLocatorImpl.java:1823) [hornetq-core-2.2.13.Final.jar:]

                at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:699) [hornetq-core-2.2.13.Final.jar:]

                at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connect(HornetQXAResourceWrapper.java:321) [hornetq-jms-2.2.13.Final.jar:]

                at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:251) [hornetq-jms-2.2.13.Final.jar:]

                ... 7 more

       

       

      08:15:28,483 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMERR: javax.transaction.xa.XAException: Error trying to connect to any providers for xa recovery

                at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:275) [hornetq-jms-2.2.13.Final.jar:]

                at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.recover(HornetQXAResourceWrapper.java:77) [hornetq-jms-2.2.13.Final.jar:]

                at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecovery(XARecoveryModule.java:503) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:471) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:385) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:166) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-4.16.2.Final.jar:]

      Caused by: java.lang.IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)

                at org.hornetq.core.client.impl.ServerLocatorImpl.assertOpen(ServerLocatorImpl.java:1823) [hornetq-core-2.2.13.Final.jar:]

                at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:699) [hornetq-core-2.2.13.Final.jar:]

                at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connect(HornetQXAResourceWrapper.java:321) [hornetq-jms-2.2.13.Final.jar:]

                at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:251) [hornetq-jms-2.2.13.Final.jar:]

                ... 7 more

       

      My configuration is as follows:

       

                      <address-settings>

                          <address-setting match="#">

                              <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                              <redelivery-delay>0</redelivery-delay>

                              <max-size-bytes>1048576</max-size-bytes>

                              <page-size-bytes>1024</page-size-bytes>

                              <address-full-policy>PAGE</address-full-policy>

                              <message-counter-history-day-limit>10</message-counter-history-day-limit>

                          </address-setting>

                      </address-settings>

       

       

                      <jms-connection-factories>

                          <connection-factory name="InVmConnectionFactory">

                              <connectors>

                                  <connector-ref connector-name="in-vm"/>

                              </connectors>

                              <entries>

                                  <entry name="java:/ConnectionFactory"/>

                              </entries>

                          </connection-factory>

                          <connection-factory name="RemoteConnectionFactory">

                              <connectors>

                                  <connector-ref connector-name="netty"/>

                              </connectors>

                              <entries>

                                  <entry name="RemoteConnectionFactory"/>

                                  <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>

                              </entries>

                          </connection-factory>

                          <pooled-connection-factory name="hornetq-ra">

                              <transaction mode="xa"/>

                              <connectors>

                                  <connector-ref connector-name="in-vm"/>

                              </connectors>

                              <entries>

                                  <entry name="java:/JmsXA"/>

                              </entries>

                          </pooled-connection-factory>

                      </jms-connection-factories>