6 Replies Latest reply on Aug 19, 2013 4:01 AM by Konstantin Zakharov

    in-VM connection failure [code=3]

    Konstantin Zakharov Newbie

      Hi,

       

      Issue details:

      • HornetQ ~2.2.21
      • in-VM connections
      • default settings for connection-ttl and client-failure-check-period, i.e.
        • client-failure-check-period => 30000
        • connection-ttl => 60000
      • hornetq-failure-check-thread detects the connection failures after some long period of time, ~12 hours
        • [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]
        • As total, 1118 matches found in the log file for the "Please check user manual for more information. The connection will now be closed. [code=3]", for the period of 2 seconds, start from 13:12:46,268 to 13:12:48,746
          • cannot say for sure, but seems it is all of the connections/sessions fail
      • The application (server and client side for this case) is deployed as single EAR

       

      There is no solid reproducer for this - it has been faced for several times though; takes quite a long time of server being run.

       

      Looking through the forums, didn't get the clue what can be the root-cause (network failures are not applicable to my case).

       

      So, at the first step, I would appreciate a help to sort out the the log files and differentiate the issues by:

      • either probable root-cause of the connection failures
      • or consequence of the connection failures

       

      Below are few excerpts from the log files with the different exceptions, sorted by time:

       

      2012.10.25 13:10:55,533 INFO  [com.arjuna.ats.arjuna] (Transaction Expired Entry Monitor) ARJUNA012296: ExpiredEntryMonitor running at Thu, 25 Oct 2012 13:10:55

      2012.10.25 13:12:46,236 WARN  [org.jboss.as.ejb3] (EJB default - 5) JBAS014143: Timer 891bba1a-a783-4b2e-811d-5c6cb25c2d1b is still active, skipping overlapping scheduled execution at: Thu Oct 25 13:12:46 IST 2012

      2012.10.25 13:12:46,268 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

      2012.10.25 13:12:46,269 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session c2b877e1-1e12-11e2-9acd-ab33cdd9b89e

      2012.10.25 13:12:46,269 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session c2b877e1-1e12-11e2-9acd-ab33cdd9b89e

      2012.10.25 13:12:46,270 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

      2012.10.25 13:12:46,304 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session c2bc215d-1e12-11e2-9acd-ab33cdd9b89e

      2012.10.25 13:12:46,305 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session c2bc215d-1e12-11e2-9acd-ab33cdd9b89e

      2012.10.25 13:12:46,305 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

      2012.10.25 13:12:46,305 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session c2b3bd71-1e12-11e2-9acd-ab33cdd9b89e

      2012.10.25 13:12:46,305 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session c2b3bd71-1e12-11e2-9acd-ab33cdd9b89e

      2012.10.25 13:12:46,306 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

      2012.10.25 13:12:46,359 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session c2995786-1e12-11e2-9acd-ab33cdd9b89e

      2012.10.25 13:12:46,359 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session c2995786-1e12-11e2-9acd-ab33cdd9b89e

      2012.10.25 13:12:46,359 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

      2012.10.25 13:12:46,360 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session c2b766d6-1e12-11e2-9acd-ab33cdd9b89e

      2012.10.25 13:12:46,360 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session c2b766d6-1e12-11e2-9acd-ab33cdd9b89e

             

      2012.10.25 13:12:47,253 WARN  [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-2321 (HornetQ-client-global-threads-2155850)) Notified of connection failure in xa recovery connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0], discoveryGroupConfiguration=null], connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0, backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2 message=Channel disconnected]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:381) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:212) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [rt.jar:1.6.0_25]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.6.0_25]

                at java.lang.Thread.run(Unknown Source) [rt.jar:1.6.0_25]

       

      2012.10.25 13:12:48,039 WARN  [org.hornetq.ra.HornetQRAManagedConnection] (Thread-1693) Handling JMS exception failure: org.hornetq.ra.HornetQRAManagedConnection@15c6e95: javax.jms.JMSException: HornetQException[errorCode=2 message=Channel disconnected]

                at org.hornetq.jms.client.HornetQConnection$JMSFailureListener.connectionFailed(HornetQConnection.java:665) [hornetq-jms-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:906) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:691) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:557) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:80) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:387) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [rt.jar:1.6.0_25]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.6.0_25]

                at java.lang.Thread.run(Unknown Source) [rt.jar:1.6.0_25]

      Caused by: HornetQException[errorCode=2 message=Channel disconnected]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:381) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:212) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                ... 4 more

       

      2012.10.25 13:12:48,780 WARN  [org.hornetq.core.client.impl.ClientSessionImpl] (extremenetworksSchedulerManagerMDB_12656) Exception occurred during commit,xid=XidImpl (8576399 bq:0.0.0.0.0.0.0.0.0.0.-1.-1.-84.16.0.0.100.-80.76.62.80.-120.68.71.0.3.-126.31.0.0.0.1.0.0.0.0 formatID:131077 gtxid:0.0.0.0.0.0.0.0.0.0.-1.-1.-84.16.0.0.100.-80.76.62.80.-120.68.71.0.3.-126.23.49 base64:AAAAAAAAAAAAAP__rBAAAGSwTD5QiERHAAOCHwAAAAEAAAAAAAAAAAAAAAAAAP__rBAAAGSwTD5QiERHAAOCFzEHAgIA,clientXID=XidWrapperImpl@2d1107[formatId=131077 globalTransactionId=яя¬   d°L>P€DG  ‚ 1 branchQualifier=яя¬   d°L>P€DG  ‚ jndiName=java:/JmsXA] throwing XAException.XA_RETRY: HornetQException[errorCode=2 message=Connection is destroyed]

                at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:233)

                at org.hornetq.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:1324)

                at org.hornetq.ra.HornetQRAXAResource.commit(HornetQRAXAResource.java:151)

                at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.commit(XAResourceWrapperImpl.java:90) [ironjacamar-core-impl-1.0.11.Final-redhat-1.jar:1.0.11.Final-redhat-1]

                at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelCommit(XAResourceRecord.java:451)

                at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2755)

                at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2671)

                at com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Commit(BasicAction.java:1806)

                at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1497)

                at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:98)

                at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)

                at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)

                at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)

                at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)

                at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:91) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:231) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:303) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:189) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:32) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:176) [jboss-as-ee-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at com.extremenetworks.epicenter.server.scheduler.beans.interfaces.SchedulerManager$$$view18.handleRequest(Unknown Source) [scheduler.jar:]

                at com.extremenetworks.epicenter.server.common.beans.impl.AbstractMDB.handleRequest(AbstractMDB.java:159) [common-server.jar:]

                at com.extremenetworks.epicenter.server.common.beans.impl.AbstractMDB.doHandleMessage(AbstractMDB.java:104) [common-server.jar:]

                at com.extremenetworks.epicenter.server.common.beans.impl.AbstractMDB.handleMessage(AbstractMDB.java:74) [common-server.jar:]

                at com.extremenetworks.epicenter.server.scheduler.beans.impl.SchedulerManagerMDB.onMessage(SchedulerManagerMDB.java:68) [scheduler.jar:]

                at sun.reflect.GeneratedMethodAccessor196.invoke(Unknown Source) [:1.6.0_25]

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [rt.jar:1.6.0_25]

                at java.lang.reflect.Method.invoke(Unknown Source) [rt.jar:1.6.0_25]

                at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:210) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.as.ejb3.tx.CMTTxInterceptor.notSupported(CMTTxInterceptor.java:287) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:187) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:43) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ejb3.component.messagedriven.MessageDrivenComponentDescription$5$1.processInvocation(MessageDrivenComponentDescription.java:184) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:176) [jboss-as-ee-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-1.jar:1.1.1.Final-redhat-1]

                at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at javax.jms.MessageListener$$$view7.onMessage(Unknown Source) [jboss-jms-api_1.1_spec-1.0.1.Final-redhat-1.jar:1.0.1.Final-redhat-1]

                at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source) [:1.6.0_25]

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [rt.jar:1.6.0_25]

                at java.lang.reflect.Method.invoke(Unknown Source) [rt.jar:1.6.0_25]

                at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(MessageEndpointInvocationHandler.java:142) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:73) [jboss-as-ejb3-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]

                at $Proxy137.onMessage(Unknown Source)          at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:311)

                at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1033)

                at org.hornetq.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50)

                at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1178)

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [rt.jar:1.6.0_25]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.6.0_25]

                at java.lang.Thread.run(Unknown Source) [rt.jar:1.6.0_25]

       

      2012.10.25 13:12:49,365 ERROR [extremenetworksSchedulerManagerMDB_12659][SchedulerManagerMDB]: Session is closed: javax.jms.IllegalStateException: Session is closed

           at org.hornetq.jms.client.HornetQSession.checkClosed(HornetQSession.java:1008)

           at org.hornetq.jms.client.HornetQSession.createObjectMessage(HornetQSession.java:158)

           at com.extremenetworks.epicenter.server.common.jms.JmsQueueSender.createObjectMessage(JmsQueueSender.java:95)

           at com.extremenetworks.epicenter.server.common.beans.impl.MessagingManagerBean.sendResponse(MessagingManagerBean.java:255)

           at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)

           at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

           at java.lang.reflect.Method.invoke(Unknown Source)

           at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)

           at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

           at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:210)

           at org.jboss.as.ejb3.tx.CMTTxInterceptor.notSupported(CMTTxInterceptor.java:287)

           at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:187)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:32)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

           at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165)

           at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:176)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

           at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72)

           at com.extremenetworks.epicenter.server.common.beans.interfaces.MessagingManager$$$view53.sendResponse(Unknown Source)

           at com.extremenetworks.epicenter.server.common.beans.impl.RequestExecutor.sendResponse(RequestExecutor.java:520)

           at com.extremenetworks.epicenter.server.common.beans.impl.AbstractMDB.handleRequest(AbstractMDB.java:206)

           at com.extremenetworks.epicenter.server.common.beans.impl.AbstractMDB.doHandleMessage(AbstractMDB.java:104)

           at com.extremenetworks.epicenter.server.common.beans.impl.AbstractMDB.handleMessage(AbstractMDB.java:74)

           at com.extremenetworks.epicenter.server.scheduler.beans.impl.SchedulerManagerMDB.onMessage(SchedulerManagerMDB.java:68)

           at sun.reflect.GeneratedMethodAccessor196.invoke(Unknown Source)

           at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

           at java.lang.reflect.Method.invoke(Unknown Source)

           at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)

           at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

           at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:210)

           at org.jboss.as.ejb3.tx.CMTTxInterceptor.notSupported(CMTTxInterceptor.java:287)

           at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:187)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:43)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ejb3.component.messagedriven.MessageDrivenComponentDescription$5$1.processInvocation(MessageDrivenComponentDescription.java:184)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

           at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165)

           at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:176)

           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

           at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

           at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72)

           at javax.jms.MessageListener$$$view7.onMessage(Unknown Source)

           at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)

           at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

           at java.lang.reflect.Method.invoke(Unknown Source)

           at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(MessageEndpointInvocationHandler.java:142)

           at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:73)

           at $Proxy137.onMessage(Unknown Source)

           at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:311)

           at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1033)

           at org.hornetq.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50)

           at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1178)

           at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

           at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

           at java.lang.Thread.run(Unknown Source)


                2012.10.25 13:12:57,606 WARN  [org.hornetq.core.client.impl.ClientConsumerImpl] (extremenetworksSchedulerManagerMDB_12651) Executor couldn't finish its operation before timeout : org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor@7ee01f

      2012.10.25 13:12:57,606 WARN  [org.hornetq.core.client.impl.ClientConsumerImpl] (extremenetworksSchedulerManagerMDB_12651) Timed out waiting for handler to complete processing: java.lang.Exception: trace

                at org.hornetq.core.client.impl.ClientConsumerImpl.waitForOnMessageToComplete(ClientConsumerImpl.java:953) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:1121) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.core.client.impl.ClientConsumerImpl.cleanUp(ClientConsumerImpl.java:486) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.core.client.impl.ClientSessionImpl.cleanUpChildren(ClientSessionImpl.java:1976) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.core.client.impl.ClientSessionImpl.cleanUp(ClientSessionImpl.java:941) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:709) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:557) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:80) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:387) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-clebert.jar:2.2.21.SNAPSHOT (HQ_2_2_21_clebertFridayNightDev, 122)]

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [rt.jar:1.6.0_25]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.6.0_25]

                at java.lang.Thread.run(Unknown Source) [rt.jar:1.6.0_25]

       

      2012.10.25 13:17:42,791 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3) - Session is closed

      2012.10.25 13:17:42,792 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3) javax.jms.IllegalStateException: Session is closed

      2012.10.25 13:17:42,792 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.hornetq.jms.client.HornetQSession.checkClosed(HornetQSession.java:1008)

      2012.10.25 13:17:42,792 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.hornetq.jms.client.HornetQSession.createObjectMessage(HornetQSession.java:158)

      2012.10.25 13:17:42,792 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at com.extremenetworks.epicenter.server.common.jms.JmsTopicPublisher$PublisherTask.createObjectMessage(JmsTopicPublisher.java:174)

      2012.10.25 13:17:42,792 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at com.extremenetworks.epicenter.server.common.beans.impl.MessagingManagerBean.postEventNonTx(MessagingManagerBean.java:165)

      2012.10.25 13:17:42,793 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at com.extremenetworks.epicenter.server.common.beans.impl.MessagingManagerBean.postEvent(MessagingManagerBean.java:339)

      2012.10.25 13:17:42,793 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at sun.reflect.GeneratedMethodAccessor530.invoke(Unknown Source)

      2012.10.25 13:17:42,793 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

      2012.10.25 13:17:42,793 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at java.lang.reflect.Method.invoke(Unknown Source)

      2012.10.25 13:17:42,793 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72)

      2012.10.25 13:17:42,793 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,793 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)

      2012.10.25 13:17:42,793 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36)

      2012.10.25 13:17:42,793 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,794 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)

      2012.10.25 13:17:42,794 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,794 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)

      2012.10.25 13:17:42,794 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,794 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)

      2012.10.25 13:17:42,794 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,794 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

      2012.10.25 13:17:42,794 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53)

      2012.10.25 13:17:42,794 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,794 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)

      2012.10.25 13:17:42,795 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,795 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:210)

      2012.10.25 13:17:42,795 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.tx.CMTTxInterceptor.notSupported(CMTTxInterceptor.java:273)

      2012.10.25 13:17:42,795 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:187)

      2012.10.25 13:17:42,795 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,795 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)

      2012.10.25 13:17:42,795 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,795 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42)

      2012.10.25 13:17:42,795 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,795 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)

      2012.10.25 13:17:42,795 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,796 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)

      2012.10.25 13:17:42,796 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,796 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:32)

      2012.10.25 13:17:42,796 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,796 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)

      2012.10.25 13:17:42,796 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,796 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

      2012.10.25 13:17:42,796 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165)

      2012.10.25 13:17:42,796 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:176)

      2012.10.25 13:17:42,796 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,796 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

      2012.10.25 13:17:42,796 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72)

      2012.10.25 13:17:42,797 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at com.extremenetworks.epicenter.server.common.beans.interfaces.MessagingManager$$$view53.postEvent(Unknown Source)

      2012.10.25 13:17:42,797 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at com.extremenetworks.epicenter.server.common.beans.impl.EventNotifier.postEvent(EventNotifier.java:52)

      2012.10.25 13:17:42,797 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at com.extremenetworks.epicenter.server.scheduler.beans.impl.SchedulerDAOBean.updateSchedulerTask(SchedulerDAOBean.java:167)

      2012.10.25 13:17:42,797 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at sun.reflect.GeneratedMethodAccessor529.invoke(Unknown Source)

      2012.10.25 13:17:42,797 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

      2012.10.25 13:17:42,797 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at java.lang.reflect.Method.invoke(Unknown Source)

      2012.10.25 13:17:42,797 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72)

      2012.10.25 13:17:42,797 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,797 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)

      2012.10.25 13:17:42,797 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36)

      2012.10.25 13:17:42,797 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,798 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)

      2012.10.25 13:17:42,798 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,798 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)

      2012.10.25 13:17:42,798 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,798 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)

      2012.10.25 13:17:42,798 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,798 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

      2012.10.25 13:17:42,798 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53)

      2012.10.25 13:17:42,798 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,798 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)

      2012.10.25 13:17:42,798 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,798 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:227)

      2012.10.25 13:17:42,799 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:303)

      2012.10.25 13:17:42,799 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:189)

      2012.10.25 13:17:42,799 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,799 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)

      2012.10.25 13:17:42,799 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,799 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42)

      2012.10.25 13:17:42,799 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,799 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)

      2012.10.25 13:17:42,799 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,799 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)

      2012.10.25 13:17:42,799 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,799 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:32)

      2012.10.25 13:17:42,800 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,800 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)

      2012.10.25 13:17:42,800 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,800 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

      2012.10.25 13:17:42,800 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165)

      2012.10.25 13:17:42,800 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:176)

      2012.10.25 13:17:42,800 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

      2012.10.25 13:17:42,800 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

      2012.10.25 13:17:42,800 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72)

      2012.10.25 13:17:42,800 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at com.extremenetworks.epicenter.server.scheduler.beans.interfaces.SchedulerDAO$$$view41.updateSchedulerTask(Unknown Source)

      2012.10.25 13:17:42,800 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at com.extremenetworks.epicenter.server.scheduler.listeners.SchedulerUpdate.updateSchedulerTaskRunInfo(SchedulerUpdate.java:100)

      2012.10.25 13:17:42,801 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at com.extremenetworks.epicenter.server.scheduler.listeners.SchedulerTriggerListener.triggerFired(SchedulerTriggerListener.java:77)

      2012.10.25 13:17:42,801 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.quartz.core.QuartzScheduler.notifyTriggerListenersFired(QuartzScheduler.java:1855)

      2012.10.25 13:17:42,801 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.quartz.core.JobRunShell.notifyListenersBeginning(JobRunShell.java:329)

      2012.10.25 13:17:42,801 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.quartz.core.JobRunShell.run(JobRunShell.java:190)

      2012.10.25 13:17:42,801 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3)           at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)

       

      2012.10.25 13:17:42,829 INFO  [stdout] (ExtremeNetworksScheduler_Worker-3) - Session is closed

      2012.10.25 13:17:45,645 ERROR [org.hornetq.ra.inflow.HornetQMessageHandler] (extremenetworksInventoryManagerMDB_12658) Failed to deliver message: HornetQException[errorCode=102 message=Session is closed]

                at org.hornetq.core.client.impl.ClientSessionImpl.checkClosed(ClientSessionImpl.java:1915)

                at org.hornetq.core.client.impl.ClientSessionImpl.acknowledge(ClientSessionImpl.java:771)

                at org.hornetq.core.client.impl.ClientConsumerImpl.doAck(ClientConsumerImpl.java:1166)

                at org.hornetq.core.client.impl.ClientConsumerImpl.acknowledge(ClientConsumerImpl.java:768)

                at org.hornetq.core.client.impl.ClientMessageImpl.acknowledge(ClientMessageImpl.java:98)

                at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:315)

                at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1033)

                at org.hornetq.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50)

                at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1178)

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [rt.jar:1.6.0_25]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.6.0_25]

                at java.lang.Thread.run(Unknown Source) [rt.jar:1.6.0_25]

       

      2012.10.25 13:17:45,647 WARN  [org.hornetq.ra.inflow.HornetQMessageHandler] (extremenetworksInventoryManagerMDB_12658) Unable to roll local transaction back

      2012.10.25 13:17:45,717 ERROR [org.hornetq.ra.inflow.HornetQMessageHandler] (extremenetworksInventoryManagerMDB_12652) Failed to deliver message: HornetQException[errorCode=102 message=Session is closed]

                at org.hornetq.core.client.impl.ClientSessionImpl.checkClosed(ClientSessionImpl.java:1915)

                at org.hornetq.core.client.impl.ClientSessionImpl.acknowledge(ClientSessionImpl.java:771)

                at org.hornetq.core.client.impl.ClientConsumerImpl.doAck(ClientConsumerImpl.java:1166)

                at org.hornetq.core.client.impl.ClientConsumerImpl.acknowledge(ClientConsumerImpl.java:768)

                at org.hornetq.core.client.impl.ClientMessageImpl.acknowledge(ClientMessageImpl.java:98)

                at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:315)

                at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1033)

                at org.hornetq.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50)

                at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1178)

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [rt.jar:1.6.0_25]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.6.0_25]

                at java.lang.Thread.run(Unknown Source) [rt.jar:1.6.0_25]

       

       

      Thanks,

      Konstantin