-
1. Re: Server Node shutdowns unexpectedly in JBoss 7.2
wdfink Dec 18, 2013 2:06 AM (in response to anelsonwilson)What you are running? "I have 3 nodes running in JBoss 7.2 server" sounds like you run a domain mode, right?
Do you see any output in the logfile?
-
2. Re: Server Node shutdowns unexpectedly in JBoss 7.2
rhusar Dec 18, 2013 6:31 AM (in response to anelsonwilson)Can any body explain me for what reason the server nodes shutdowns in this unexpected manner?
Is this really shutdown or crash? You should find out in the logs. But it can be crashing due to OS bugs, JVM bugs, application calling System.exit(...), exhausting resources, etc...
-
3. Re: Server Node shutdowns unexpectedly in JBoss 7.2
anelsonwilson Dec 18, 2013 10:26 AM (in response to wdfink)I think its because of HornetQ exception:- Log is as follows
12:00:35,760 ERROR [org.jboss.remoting.remote.connection] (Remoting "master:srs1:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer
12:00:35,795 WARN [org.quartz.impl.jdbcjobstore.JobStoreTX] (QuartzScheduler_SRSClusteredScheduler-PLPCCA1491386942227049_ClusterManager) This scheduler instance (PLPCCA1491386942227049) is still active but was recovered by another instance in the cluster. This may cause inconsistent behavior.
12:00:35,760 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
12:00:35,795 WARN [org.hornetq.core.server] (Thread-7636 (HornetQ-client-global-threads-913802640)) HQ222141: Connection failed with failedOver=false: HornetQException[errorType=NOT_CONNECTED message=HQ119025: Channel disconnected]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:418) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:754) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106) [hornetq-core-client-2.3.0.CR1.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_11]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_11]
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]
12:00:35,795 WARN [org.hornetq.core.server] (Thread-7659 (HornetQ-client-global-threads-913802640)) HQ222141: Connection failed with failedOver=false: HornetQException[errorType=NOT_CONNECTED message=HQ119025: Channel disconnected]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:418) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:754) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106) [hornetq-core-client-2.3.0.CR1.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_11]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_11]
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]
12:00:35,999 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session a4b3297a-63fc-11e3-a9ac-215d32b8500b
12:00:35,795 WARN [org.hornetq.core.server] (Thread-7660 (HornetQ-client-global-threads-913802640)) HQ222141: Connection failed with failedOver=false: HornetQException[errorType=NOT_CONNECTED message=HQ119025: Channel disconnected]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:418) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:754) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106) [hornetq-core-client-2.3.0.CR1.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_11]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_11]
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]
12:00:36,003 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222158: Cleared up resources for session a4b3297a-63fc-11e3-a9ac-215d32b8500b
12:00:36,032 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
12:00:36,200 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session a4c10c2d-63fc-11e3-a9ac-215d32b8500b
12:00:36,214 INFO [com.eks.srs.web.login.action.LogoutAction] (ajp-/10.0.76.12:8110-2) Loggout from application, User login ID 4791.admin.ee
12:00:36,244 INFO [com.eks.srs.service.webservice.session.EKSWebServiceManagerImpl] (ajp-/10.0.76.12:8110-4) insertOrUpdateQty : hhtUserId : 4780-fb-0000-user1 articleCode :- 000000000108100294 ean :- 2924622 qty :- 50.56,isInsert :- true scheduleRefId :- 96221hhtIp :10.201.46.100 hhtMachineId :001346E604EB010816FF-0AFF01FFEBFF
12:00:36,244 INFO [com.eks.srs.service.webservice.session.EKSWebServiceManagerImpl] (ajp-/10.0.76.12:8110-22) insertOrUpdateQty : hhtUserId : 4950-fb-0000-user1 articleCode :- 000000000108100346 ean :- 2924675 qty :- 1.2,isInsert :- true scheduleRefId :- 96226hhtIp :10.201.49.99 hhtMachineId :001346E604EB010812FF-0AFF01FFC5FF
12:00:36,246 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222158: Cleared up resources for session a4c10c2d-63fc-11e3-a9ac-215d32b8500b
12:00:36,421 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
12:00:36,487 INFO [com.eks.common.SessionListener] (ajp-/10.0.76.12:8110-2) SessionListener sessionDestroyed
12:00:36,612 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session a46ecd5c-63fc-11e3-a9ac-215d32b8500b
12:00:36,612 INFO [com.eks.srs.service.webservice.service.CycleCountWebServiceServiceImpl] (ajp-/10.0.76.12:8110-4) insertOrUpdateQty :- hhtUserId :- 4780-fb-0000-user1 articleCode :- 000000000108100294 ean :- 2924622 qty :- 50.56 isInsert :- true scheduleRefId :- 96221
12:00:36,612 INFO [com.eks.srs.service.webservice.service.CycleCountWebServiceServiceImpl] (ajp-/10.0.76.12:8110-22) insertOrUpdateQty :- hhtUserId :- 4950-fb-0000-user1 articleCode :- 000000000108100346 ean :- 2924675 qty :- 1.2 isInsert :- true scheduleRefId :- 96226
12:00:37,032 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222158: Cleared up resources for session a46ecd5c-63fc-11e3-a9ac-215d32b8500b
12:00:37,039 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
12:00:37,055 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session a49b82bb-63fc-11e3-a9ac-215d32b8500b
12:00:37,058 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222158: Cleared up resources for session a49b82bb-63fc-11e3-a9ac-215d32b8500b
12:00:37,065 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
12:00:37,081 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session a4931e48-63fc-11e3-a9ac-215d32b8500b
12:00:37,082 INFO [org.hornetq.core.server] (Thread-27 (HornetQ-server-HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c-199033866)) HQ221032: Bridge ClusterConnectionBridge@6f28bb44 [name=sf.my-cluster.7c3f3e7d-6001-11e3-984d-679851f4e056, queue=QueueImpl[name=sf.my-cluster.7c3f3e7d-6001-11e3-984d-679851f4e056, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c]]@393cd5b5 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@6f28bb44 [name=sf.my-cluster.7c3f3e7d-6001-11e3-984d-679851f4e056, queue=QueueImpl[name=sf.my-cluster.7c3f3e7d-6001-11e3-984d-679851f4e056, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c]]@393cd5b5 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5547&host=10-0-76-12], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@988929507[nodeUUID=02540094-6001-11e3-b958-2133d8b2a50c, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5546&host=10-0-76-12, address=jms, server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5547&host=10-0-76-12], discoveryGroupConfiguration=null]] is connected
12:00:37,084 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222158: Cleared up resources for session a4931e48-63fc-11e3-a9ac-215d32b8500b
12:00:37,109 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
12:00:37,124 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session 8f4f91e0-66e4-11e3-a9ac-215d32b8500b
12:00:37,125 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222158: Cleared up resources for session 8f4f91e0-66e4-11e3-a9ac-215d32b8500b
12:00:37,125 INFO [org.hornetq.core.server] (Thread-10 (HornetQ-server-HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c-199033866)) HQ221032: Bridge ClusterConnectionBridge@6f3e54b [name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, queue=QueueImpl[name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c]]@7652a27d targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@6f3e54b [name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, queue=QueueImpl[name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c]]@7652a27d targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5646&host=10-0-76-12], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@988929507[nodeUUID=02540094-6001-11e3-b958-2133d8b2a50c, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5546&host=10-0-76-12, address=jms, server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5646&host=10-0-76-12], discoveryGroupConfiguration=null]] is connected
12:00:37,126 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212107: Connection failure has been detected: HQ119034: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
12:00:37,128 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session a4a0b2de-63fc-11e3-a9ac-215d32b8500b
12:00:37,126 WARN [org.hornetq.jms.server] (Thread-7661 (HornetQ-client-global-threads-913802640)) HQ122017: Notified of connection failure in xa recovery connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=8cbc8475-63fc-11e3-a9ac-215d32b8500b, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryGroupConfiguration=null], connectorConfig=TransportConfiguration(name=8cbc8475-63fc-11e3-a9ac-215d32b8500b, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0, backupConfig=null] will attempt reconnect on next pass: HornetQException[errorType=NOT_CONNECTED message=HQ119025: Channel disconnected]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:418) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:214) [hornetq-server-2.3.0.CR1.jar:]
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106) [hornetq-core-client-2.3.0.CR1.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_11]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_11]
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]
.
.
.
12:33:13,548 INFO [com.eks.common.SessionListener] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) SessionListener sessionDestroyed
12:33:13,102 WARN [org.hornetq.core.client] (Thread-7765 (HornetQ-client-global-threads-913802640)) HQ212107: Connection failure has been detected: HQ119030: Did not receive data from server for InVMConnection [serverID=0, id=4393b974-66e7-11e3-a9ac-215d32b8500b] [code=CONNECTION_TIMEDOUT]
12:37:37,860 WARN [org.hornetq.core.client] (Thread-7773 (HornetQ-client-global-threads-913802640)) HQ212107: Connection failure has been detected: HQ119030: Did not receive data from server for InVMConnection [serverID=0, id=f947a4df-66e6-11e3-a9ac-215d32b8500b] [code=CONNECTION_TIMEDOUT]
12:37:37,816 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222100: Client connection failed, clearing up resources for session a4931e48-63fc-11e3-a9ac-215d32b8500b
12:37:37,862 WARN [org.hornetq.core.client] (Thread-7774 (HornetQ-client-global-threads-913802640)) HQ212107: Connection failure has been detected: HQ119030: Did not receive data from server for InVMConnection [serverID=0, id=f95dec09-66e6-11e3-a9ac-215d32b8500b] [code=CONNECTION_TIMEDOUT]
12:37:37,816 WARN [org.hornetq.core.client] (Thread-7771 (HornetQ-client-global-threads-913802640)) HQ212107: Connection failure has been detected: HQ119030: Did not receive data from server for InVMConnection [serverID=0, id=f961205b-66e6-11e3-a9ac-215d32b8500b] [code=CONNECTION_TIMEDOUT]
12:40:43,265 WARN [org.quartz.impl.jdbcjobstore.JobStoreTX] (QuartzScheduler_SRSClusteredScheduler-PLPCCA1491386942227049_ClusterManager) This scheduler instance (PLPCCA1491386942227049) is still active but was recovered by another instance in the cluster. This may cause inconsistent behavior.
12:29:50,964 WARN [org.hornetq.core.client] (Finalizer) HQ212024: I am closing a core ClientSessionFactory you left open. Please make sure you close all ClientSessionFactories explicitly before letting them go out of scope! 1,712,232,240: java.lang.Exception
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.<init>(ClientSessionFactoryImpl.java:197)
at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:671)
at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:648)
at org.hornetq.core.server.cluster.impl.ClusterConnectionBridge.createSessionFactory(ClusterConnectionBridge.java:149)
at org.hornetq.core.server.cluster.impl.BridgeImpl.connect(BridgeImpl.java:795)
at org.hornetq.core.server.cluster.impl.BridgeImpl$ConnectRunnable.run(BridgeImpl.java:1101)
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_11]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_11]
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]
12:42:02,587 WARN [org.hornetq.core.client] (Thread-7787 (HornetQ-client-global-threads-913802640)) HQ212107: Connection failure has been detected: HQ119030: Did not receive data from server for InVMConnection [serverID=0, id=f928f943-66e6-11e3-a9ac-215d32b8500b] [code=CONNECTION_TIMEDOUT]
12:40:43,265 ERROR [org.hornetq.core.client] (Thread-7773 (HornetQ-client-global-threads-913802640)) HQ214005: Failed to handle failover: HornetQException[errorType=CONNECTION_TIMEDOUT message=HQ119033: Timed out waiting for response when sending packet 32]
at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:369) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:1031) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:1043) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:710) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:580) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$100(ClientSessionFactoryImpl.java:85) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1674) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:570) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:341) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl$PingRunnable$1.run(ClientSessionFactoryImpl.java:1741) [hornetq-core-client-2.3.0.CR1.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_11]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_11]
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]
12:46:27,947 INFO [org.jboss.as.messaging] (ServerService Thread Pool -- 141) JBAS011605: Unbound messaging object to jndi name java:/queue/CCScheduleQueue
12:46:28,213 WARN [org.quartz.impl.jdbcjobstore.JobStoreTX] (QuartzScheduler_SRSClusteredScheduler-PLPCCA1491386942227049_ClusterManager) This scheduler instance (PLPCCA1491386942227049) is still active but was recovered by another instance in the cluster. This may cause inconsistent behavior.
12:46:28,221 INFO [com.eks.srs.common.CommonCache] (ServerService Thread Pool -- 168) removeCommonCache() started
12:46:28,229 INFO [com.eks.srs.common.CommonCache] (ServerService Thread Pool -- 168) removeCommonCache() Completed
12:46:28,245 INFO [com.eks.srs.common.AppSettingsCache] (ServerService Thread Pool -- 177) removeAppSettingCache() started
12:46:28,293 INFO [com.eks.srs.common.AppSettingsCache] (ServerService Thread Pool -- 177) removeAppSettingCache() completed
12:46:28,572 INFO [org.hornetq.core.server] (Thread-21 (HornetQ-server-HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c-199033866)) HQ221032: Bridge ClusterConnectionBridge@6f3e54b [name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, queue=QueueImpl[name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c]]@7652a27d targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@6f3e54b [name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, queue=QueueImpl[name=sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c]]@7652a27d targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5646&host=10-0-76-12], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@988929507[nodeUUID=02540094-6001-11e3-b958-2133d8b2a50c, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5546&host=10-0-76-12, address=jms, server=HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5646&host=10-0-76-12], discoveryGroupConfiguration=null]] is connected
12:46:28,560 ERROR [org.jboss.as.txn] (ajp-/10.0.76.12:8110-6) JBAS010151: Unable to get transaction state: java.lang.IllegalStateException
at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.checkTransactionStatus(TransactionRollbackSetupAction.java:112)
at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.teardown(TransactionRollbackSetupAction.java:66)
at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:61)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:184)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488)
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920)
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]
.
.
.
14:54:48,669 INFO [org.hornetq.core.server] (Thread-16 (HornetQ-server-HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c-199033866)) HQ221034: stopped bridge sf.my-cluster.0c479bed-6001-11e3-a97f-0149870633e4
14:54:48,669 INFO [org.hornetq.core.server] (Thread-20 (HornetQ-server-HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c-199033866)) HQ221034: stopped bridge sf.my-cluster.11603c1f-6001-11e3-ba44-53fa20978c8b
14:54:48,669 INFO [org.hornetq.core.server] (Thread-26 (HornetQ-server-HornetQServerImpl::serverUUID=02540094-6001-11e3-b958-2133d8b2a50c-199033866)) HQ221034: stopped bridge sf.my-cluster.7c3f3e7d-6001-11e3-984d-679851f4e056
14:54:48,667 ERROR [org.jboss.as.txn] (ajp-/10.0.76.12:8110-8) JBAS010151: Unable to get transaction state: java.lang.IllegalStateException
at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.checkTransactionStatus(TransactionRollbackSetupAction.java:112)
at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.teardown(TransactionRollbackSetupAction.java:66)
at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:61) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:184) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]
14:54:48,714 ERROR [org.apache.catalina.connector] (ajp-/10.0.76.12:8110-8) JBWEB001018: An exception or error occurred in the container during the request processing: java.lang.RuntimeException: java.lang.IllegalStateException
at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:67) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:184) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_11]
Caused by: java.lang.IllegalStateException
at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
at org.jboss.as.connector.deployers.ra.processors.CachedConnectionManagerSetupProcessor$CachedConnectionManagerSetupAction.teardown(CachedConnectionManagerSetupProcessor.java:105)
at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:61) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]
... 8 more
14:54:48,772 WARN [org.hornetq.core.client] (Thread-7943 (HornetQ-client-global-threads-913802640)) HQ212107: Connection failure has been detected: HQ119035: The connection was disconnected because of server shutdown [code=DISCONNECTED]
14:54:49,432 INFO [org.hornetq.core.server] (MSC service thread 1-122) HQ221004: HornetQ Server version 2.3.0.CR1 (buzzzzz!, 122) [02540094-6001-11e3-b958-2133d8b2a50c] stopped
14:54:49,468 INFO [org.jboss.as.server.deployment] (MSC service thread 1-38) JBAS015877: Stopped deployment null (runtime-name: SRSEJB.jar) in 7966697ms
14:54:49,605 INFO [org.apache.coyote.ajp] (MSC service thread 1-84) JBWEB003048: Pausing Coyote AJP/1.3 on ajp-/10.0.76.12:8110
14:54:49,607 INFO [org.apache.coyote.ajp] (MSC service thread 1-84) JBWEB003051: Stopping Coyote AJP/1.3 on ajp-/10.0.76.12:8110
14:54:49,828 INFO [org.jboss.as.server.deployment] (MSC service thread 1-19) JBAS015877: Stopped deployment null (runtime-name: SRSWeb.war) in 7967057ms
14:54:49,841 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) JBAS015877: Stopped deployment SRSEar.ear (runtime-name: SRSEar.ear) in 8046355ms
14:54:51,683 INFO [org.jboss.as] (MSC service thread 1-99) JBAS015950: JBoss AS 7.2.0.Final "Janus" stopped in 7968087ms