3 Replies Latest reply on Dec 18, 2013 10:26 AM by anelsonwilson

    Server Node shutdowns unexpectedly in JBoss 7.2

    anelsonwilson

      Hi, I have 3 nodes running in JBoss 7.2 server. I found that one of the node shutdowns unexpectedly. Can any body explain me for what reason the server nodes shutdowns in this unexpected manner?

        • 1. Re: Server Node shutdowns unexpectedly in JBoss 7.2
          wdfink

          What you are running? "I have 3 nodes running in JBoss 7.2 server" sounds like  you run a domain mode, right?

          Do you see any output in the logfile?

          • 2. Re: Server Node shutdowns unexpectedly in JBoss 7.2
            rhusar
            Can any body explain me for what reason the server nodes shutdowns in this unexpected manner?

            Is this really shutdown or crash? You should find out in the logs. But it can be crashing due to OS bugs, JVM bugs, application calling System.exit(...), exhausting resources, etc...

            • 3. Re: Server Node shutdowns unexpectedly in JBoss 7.2
              anelsonwilson

              I think its because of HornetQ exception:- Log is as follows

               

              12:00:35,760 ERROR [org.jboss.remoting.remote.connection] (Remoting "master:srs1:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer

              12:00:35,795 WARN  [org.quartz.impl.jdbcjobstore.JobStoreTX] (QuartzScheduler_SRSClusteredScheduler-PLPCCA1491386942227049_ClusterManager) This scheduler instance (PLPCCA1491386942227049) is still active but was recovered by another instance in the cluster.  This may cause inconsistent behavior.

              12:00:35,760 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

              12:00:35,795 WARN  [org.hornetq.core.server] (Thread-7636 (HornetQ-client-global-threads-913802640)) HQ222141: Connection failed with failedOver=false: HornetQException[errorType=NOT_CONNECTED message=HQ119025: Channel disconnected]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:418) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:754) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106) [hornetq-core-client-2.3.0.CR1.jar:]

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_11]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_11]

                at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]

               

               

              12:00:35,795 WARN  [org.hornetq.core.server] (Thread-7659 (HornetQ-client-global-threads-913802640)) HQ222141: Connection failed with failedOver=false: HornetQException[errorType=NOT_CONNECTED message=HQ119025: Channel disconnected]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:418) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:754) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106) [hornetq-core-client-2.3.0.CR1.jar:]

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_11]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_11]

                at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]

               

               

              12:00:35,999 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session a4b3297a-63fc-11e3-a9ac-215d32b8500b

              12:00:35,795 WARN  [org.hornetq.core.server] (Thread-7660 (HornetQ-client-global-threads-913802640)) HQ222141: Connection failed with failedOver=false: HornetQException[errorType=NOT_CONNECTED message=HQ119025: Channel disconnected]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:418) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:754) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106) [hornetq-core-client-2.3.0.CR1.jar:]

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_11]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_11]

                at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]

               

               

              12:00:36,003 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222158: Cleared up resources for session a4b3297a-63fc-11e3-a9ac-215d32b8500b

              12:00:36,032 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

              12:00:36,200 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session a4c10c2d-63fc-11e3-a9ac-215d32b8500b

              12:00:36,214 INFO  [com.eks.srs.web.login.action.LogoutAction] (ajp-/10.0.76.12:8110-2) Loggout from application, User login ID  4791.admin.ee

              12:00:36,244 INFO  [com.eks.srs.service.webservice.session.EKSWebServiceManagerImpl] (ajp-/10.0.76.12:8110-4) insertOrUpdateQty : hhtUserId : 4780-fb-0000-user1 articleCode :- 000000000108100294 ean :- 2924622 qty :- 50.56,isInsert :- true  scheduleRefId :- 96221hhtIp :10.201.46.100 hhtMachineId :001346E604EB010816FF-0AFF01FFEBFF

              12:00:36,244 INFO  [com.eks.srs.service.webservice.session.EKSWebServiceManagerImpl] (ajp-/10.0.76.12:8110-22) insertOrUpdateQty : hhtUserId : 4950-fb-0000-user1 articleCode :- 000000000108100346 ean :- 2924675 qty :- 1.2,isInsert :- true  scheduleRefId :- 96226hhtIp :10.201.49.99 hhtMachineId :001346E604EB010812FF-0AFF01FFC5FF

              12:00:36,246 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222158: Cleared up resources for session a4c10c2d-63fc-11e3-a9ac-215d32b8500b

              12:00:36,421 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

              12:00:36,487 INFO  [com.eks.common.SessionListener] (ajp-/10.0.76.12:8110-2) SessionListener sessionDestroyed

              12:00:36,612 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session a46ecd5c-63fc-11e3-a9ac-215d32b8500b

              12:00:36,612 INFO  [com.eks.srs.service.webservice.service.CycleCountWebServiceServiceImpl] (ajp-/10.0.76.12:8110-4)  insertOrUpdateQty :-  hhtUserId :- 4780-fb-0000-user1 articleCode :- 000000000108100294 ean :- 2924622 qty :- 50.56 isInsert :- true scheduleRefId :- 96221

              12:00:36,612 INFO  [com.eks.srs.service.webservice.service.CycleCountWebServiceServiceImpl] (ajp-/10.0.76.12:8110-22)  insertOrUpdateQty :-  hhtUserId :- 4950-fb-0000-user1 articleCode :- 000000000108100346 ean :- 2924675 qty :- 1.2 isInsert :- true scheduleRefId :- 96226

              12:00:37,032 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222158: Cleared up resources for session a46ecd5c-63fc-11e3-a9ac-215d32b8500b

              12:00:37,039 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

              12:00:37,055 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session a49b82bb-63fc-11e3-a9ac-215d32b8500b

              12:00:37,058 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222158: Cleared up resources for session a49b82bb-63fc-11e3-a9ac-215d32b8500b

              12:00:37,065 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

              12:00:37,081 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session a4931e48-63fc-11e3-a9ac-215d32b8500b

              12:00:37,082 INFO  [org.hornetq.core.server] (Thread-27 (HornetQ-server-HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c-199033866)) HQ221032: Bridge ClusterConnectionBridge@6f28bb44 [name=sf.my-cluster.7c3f3e7d-6001-11e3-984d-679851f4e056, queue=QueueImpl[name=sf.my-cluster.7c3f3e7d-6001-11e3-984d-679851f4e056, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c]]@393cd5b5 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@6f28bb44 [name=sf.my-cluster.7c3f3e7d-6001-11e3-984d-679851f4e056, queue=QueueImpl[name=sf.my-cluster.7c3f3e7d-6001-11e3-984d-679851f4e056, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c]]@393cd5b5 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5547&host=10-0-76-12], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@988929507[nodeUUID=02540094-6001-11e3-b958-2133d8b2a50c, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5546&host=10-0-76-12, address=jms, server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5547&host=10-0-76-12], discoveryGroupConfiguration=null]] is connected

              12:00:37,084 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222158: Cleared up resources for session a4931e48-63fc-11e3-a9ac-215d32b8500b

              12:00:37,109 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

              12:00:37,124 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session 8f4f91e0-66e4-11e3-a9ac-215d32b8500b

              12:00:37,125 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222158: Cleared up resources for session 8f4f91e0-66e4-11e3-a9ac-215d32b8500b

              12:00:37,125 INFO  [org.hornetq.core.server] (Thread-10 (HornetQ-server-HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c-199033866)) HQ221032: Bridge ClusterConnectionBridge@6f3e54b [name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, queue=QueueImpl[name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c]]@7652a27d targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@6f3e54b [name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, queue=QueueImpl[name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c]]@7652a27d targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5646&host=10-0-76-12], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@988929507[nodeUUID=02540094-6001-11e3-b958-2133d8b2a50c, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5546&host=10-0-76-12, address=jms, server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5646&host=10-0-76-12], discoveryGroupConfiguration=null]] is connected

              12:00:37,126 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

              12:00:37,128 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session a4a0b2de-63fc-11e3-a9ac-215d32b8500b

              12:00:37,126 WARN  [org.hornetq.jms.server] (Thread-7661 (HornetQ-client-global-threads-913802640)) HQ122017: Notified of connection failure in xa recovery connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=8cbc8475-63fc-11e3-a9ac-215d32b8500b, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryGroupConfiguration=null], connectorConfig=TransportConfiguration(name=8cbc8475-63fc-11e3-a9ac-215d32b8500b, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0, backupConfig=null] will attempt reconnect on next pass: HornetQException[errorType=NOT_CONNECTED message=HQ119025: Channel disconnected]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:418) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:214) [hornetq-server-2.3.0.CR1.jar:]

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106) [hornetq-core-client-2.3.0.CR1.jar:]

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_11]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_11]

                at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]

              .

              .

              .

              12:33:13,548 INFO  [com.eks.common.SessionListener] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) SessionListener sessionDestroyed

              12:33:13,102 WARN  [org.hornetq.core.client] (Thread-7765 (HornetQ-client-global-threads-913802640)) HQ212107: Connection failure has been detected: HQ119030: Did not receive data from server for InVMConnection [serverID=0, id=4393b974-66e7-11e3-a9ac-215d32b8500b] [code=CONNECTION_TIMEDOUT]

              12:37:37,860 WARN  [org.hornetq.core.client] (Thread-7773 (HornetQ-client-global-threads-913802640)) HQ212107: Connection failure has been detected: HQ119030: Did not receive data from server for InVMConnection [serverID=0, id=f947a4df-66e6-11e3-a9ac-215d32b8500b] [code=CONNECTION_TIMEDOUT]

              12:37:37,816 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session a4931e48-63fc-11e3-a9ac-215d32b8500b

              12:37:37,862 WARN  [org.hornetq.core.client] (Thread-7774 (HornetQ-client-global-threads-913802640)) HQ212107: Connection failure has been detected: HQ119030: Did not receive data from server for InVMConnection [serverID=0, id=f95dec09-66e6-11e3-a9ac-215d32b8500b] [code=CONNECTION_TIMEDOUT]

              12:37:37,816 WARN  [org.hornetq.core.client] (Thread-7771 (HornetQ-client-global-threads-913802640)) HQ212107: Connection failure has been detected: HQ119030: Did not receive data from server for InVMConnection [serverID=0, id=f961205b-66e6-11e3-a9ac-215d32b8500b] [code=CONNECTION_TIMEDOUT]

              12:40:43,265 WARN  [org.quartz.impl.jdbcjobstore.JobStoreTX] (QuartzScheduler_SRSClusteredScheduler-PLPCCA1491386942227049_ClusterManager) This scheduler instance (PLPCCA1491386942227049) is still active but was recovered by another instance in the cluster.  This may cause inconsistent behavior.

              12:29:50,964 WARN  [org.hornetq.core.client] (Finalizer) HQ212024: I am closing a core ClientSessionFactory you left open. Please make sure you close all ClientSessionFactories explicitly before letting them go out of scope! 1,712,232,240: java.lang.Exception

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.<init>(ClientSessionFactoryImpl.java:197)

                at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:671)

                at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:648)

                at org.hornetq.core.server.cluster.impl.ClusterConnectionBridge.createSessionFactory(ClusterConnectionBridge.java:149)

                at org.hornetq.core.server.cluster.impl.BridgeImpl.connect(BridgeImpl.java:795)

                at org.hornetq.core.server.cluster.impl.BridgeImpl$ConnectRunnable.run(BridgeImpl.java:1101)

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106)

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_11]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_11]

                at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]

               

              12:42:02,587 WARN  [org.hornetq.core.client] (Thread-7787 (HornetQ-client-global-threads-913802640)) HQ212107: Connection failure has been detected: HQ119030: Did not receive data from server for InVMConnection [serverID=0, id=f928f943-66e6-11e3-a9ac-215d32b8500b] [code=CONNECTION_TIMEDOUT]

              12:40:43,265 ERROR [org.hornetq.core.client] (Thread-7773 (HornetQ-client-global-threads-913802640)) HQ214005: Failed to handle failover: HornetQException[errorType=CONNECTION_TIMEDOUT message=HQ119033: Timed out waiting for response when sending packet 32]

                at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:369) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:1031) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:1043) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:710) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:580) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$100(ClientSessionFactoryImpl.java:85) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1674) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:570) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:341) [hornetq-core-client-2.3.0.CR1.jar:]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl$PingRunnable$1.run(ClientSessionFactoryImpl.java:1741) [hornetq-core-client-2.3.0.CR1.jar:]

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_11]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_11]

                at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]

               

              12:46:27,947 INFO  [org.jboss.as.messaging] (ServerService Thread Pool -- 141) JBAS011605: Unbound messaging object to jndi name java:/queue/CCScheduleQueue

              12:46:28,213 WARN  [org.quartz.impl.jdbcjobstore.JobStoreTX] (QuartzScheduler_SRSClusteredScheduler-PLPCCA1491386942227049_ClusterManager) This scheduler instance (PLPCCA1491386942227049) is still active but was recovered by another instance in the cluster.  This may cause inconsistent behavior.

              12:46:28,221 INFO  [com.eks.srs.common.CommonCache] (ServerService Thread Pool -- 168) removeCommonCache() started

              12:46:28,229 INFO  [com.eks.srs.common.CommonCache] (ServerService Thread Pool -- 168) removeCommonCache() Completed

              12:46:28,245 INFO  [com.eks.srs.common.AppSettingsCache] (ServerService Thread Pool -- 177) removeAppSettingCache() started

              12:46:28,293 INFO  [com.eks.srs.common.AppSettingsCache] (ServerService Thread Pool -- 177) removeAppSettingCache() completed

              12:46:28,572 INFO  [org.hornetq.core.server] (Thread-21 (HornetQ-server-HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c-199033866)) HQ221032: Bridge ClusterConnectionBridge@6f3e54b [name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, queue=QueueImpl[name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c]]@7652a27d targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@6f3e54b [name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, queue=QueueImpl[name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c]]@7652a27d targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5646&host=10-0-76-12], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@988929507[nodeUUID=02540094-6001-11e3-b958-2133d8b2a50c, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5546&host=10-0-76-12, address=jms, server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5646&host=10-0-76-12], discoveryGroupConfiguration=null]] is connected

              12:46:28,560 ERROR [org.jboss.as.txn] (ajp-/10.0.76.12:8110-6) JBAS010151: Unable to get transaction state: java.lang.IllegalStateException

                at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)

                at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.checkTransactionStatus(TransactionRollbackSetupAction.java:112)

                at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.teardown(TransactionRollbackSetupAction.java:66)

                at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:61)

                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:184)

                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)

                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)

                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336)

                at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488)

                at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420)

                at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920)

                at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]

              .

              .

              .

              14:54:48,669 INFO  [org.hornetq.core.server] (Thread-16 (HornetQ-server-HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c-199033866)) HQ221034: stopped bridge sf.my-cluster.0c479bed-6001-11e3-a97f-0149870633e4

              14:54:48,669 INFO  [org.hornetq.core.server] (Thread-20 (HornetQ-server-HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c-199033866)) HQ221034: stopped bridge sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b

              14:54:48,669 INFO  [org.hornetq.core.server] (Thread-26 (HornetQ-server-HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c-199033866)) HQ221034: stopped bridge sf.my-cluster.7c3f3e7d-6001-11e3-984d-679851f4e056

              14:54:48,667 ERROR [org.jboss.as.txn] (ajp-/10.0.76.12:8110-8) JBAS010151: Unable to get transaction state: java.lang.IllegalStateException

                at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)

                at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.checkTransactionStatus(TransactionRollbackSetupAction.java:112)

                at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.teardown(TransactionRollbackSetupAction.java:66)

                at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:61) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:184) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]

               

               

              14:54:48,714 ERROR [org.apache.catalina.connector] (ajp-/10.0.76.12:8110-8) JBWEB001018: An exception or error occurred in the container during the request processing: java.lang.RuntimeException: java.lang.IllegalStateException

                at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:67) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:184) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]

              Caused by: java.lang.IllegalStateException

                at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)

                at org.jboss.as.connector.deployers.ra.processors.CachedConnectionManagerSetupProcessor$CachedConnectionManagerSetupAction.teardown(CachedConnectionManagerSetupProcessor.java:105)

                at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:61) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

                ... 8 more

               

               

              14:54:48,772 WARN  [org.hornetq.core.client] (Thread-7943 (HornetQ-client-global-threads-913802640)) HQ212107: Connection failure has been detected: HQ119035: The connection was disconnected because of server shutdown [code=DISCONNECTED]

              14:54:49,432 INFO  [org.hornetq.core.server] (MSC service thread 1-122) HQ221004: HornetQ Server version 2.3.0.CR1 (buzzzzz!, 122) [02540094-6001-11e3-b958-2133d8b2a50c] stopped

              14:54:49,468 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-38) JBAS015877: Stopped deployment null (runtime-name: SRSEJB.jar) in 7966697ms

              14:54:49,605 INFO  [org.apache.coyote.ajp] (MSC service thread 1-84) JBWEB003048: Pausing Coyote AJP/1.3 on ajp-/10.0.76.12:8110

              14:54:49,607 INFO  [org.apache.coyote.ajp] (MSC service thread 1-84) JBWEB003051: Stopping Coyote AJP/1.3 on ajp-/10.0.76.12:8110

              14:54:49,828 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-19) JBAS015877: Stopped deployment null (runtime-name: SRSWeb.war) in 7967057ms

              14:54:49,841 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-5) JBAS015877: Stopped deployment SRSEar.ear (runtime-name: SRSEar.ear) in 8046355ms

              14:54:51,683 INFO  [org.jboss.as] (MSC service thread 1-99) JBAS015950: JBoss AS 7.2.0.Final "Janus" stopped in 7968087ms