Failure on control connection Thread?
Hi,
* JBoss 5.0.1.GA
* JBoss messaging 1.4.6.GA
* JBoss remoting 2.5.2
* Java jdk1.5.0_19
We have a cluster with two nodes using clustered queue, and recently we are getting the following errors with JBoss messaging:
1. The following appears in Node 1:
2009-11-30 16:21:47,525 [Timer-6] WARN remoting.transport.bisocket.BisocketServerInvoker : org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@10d107: detected failure on control connection Thread[control: Socket[addr=myhost2/172.32.29.21,port=32062,localport=1881],5,] (4sl3l3l-7k73nm-g2mqo1ic-1-g2mqxt8r-jn: requesting new control connection
2. After a few minutes, on node 2[172.32.29.21]:
2009-11-30 16:24:45,243 [Timer-3] WARN remoting.transport.bisocket.BisocketServerInvoker : org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@17663eb: detected failure on control connection Thread[control: Socket[addr=myhost2/172.32.29.20,port=18861,localport=1686],5,] (4sl3l3k-n6bz0c-g2mqho1y-1-g2mqqj8h-j6: requesting new control connection 2009-11-30 16:24:49,886 [Timer-7] WARN remoting.transport.bisocket.BisocketClientInvoker : Unable to send ping: shutting down PingTimerTask java.net.SocketException: Software caused connection abort: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:115) at org.jboss.remoting.transport.bisocket.BisocketClientInvoker$PingTimerTask.run(BisocketClientInvoker.java:723) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) 2009-11-30 16:25:45,864 [WorkerThread#0[172.21.129.129:1995]] ERROR jboss.messaging.util.ExceptionUtil : SessionEndpoint[l31-8c8rsm2g-1-y1ohqm2g-c0zb6n-k3l3ls4] createConsumerDelegate [m31-oc8rsm2g-1-y1ohqm2g-c0zb6n-k3l3ls4] javax.jms.InvalidDestinationException: No such destination: JBossTemporaryQueue[wj-xktxqm2g-1-ci1oqm2g-mn37k7-l3l3ls4] has it been deployed? at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegateInternal(ServerSessionEndpoint.java:1891) at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:262) at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:94) at org.jboss.jms.server.endpoint.advised.SessionAdvised$createConsumerDelegate_8721389917985689973.invokeTarget(SessionAdvised$createConsumerDelegate_8721389917985689973.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:123) at sun.reflect.GeneratedMethodAccessor3253.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.server.endpoint.advised.SessionAdvised.createConsumerDelegate(SessionAdvised.java) at org.jboss.jms.wireformat.SessionCreateConsumerDelegateRequest.serverInvoke(SessionCreateConsumerDelegateRequest.java:100) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:897) at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:768) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:721) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:575) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234) 2009-11-30 16:30:55,300 [Timer-3] WARN remoting.transport.bisocket.BisocketServerInvoker : org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@17663eb: detected failure on control connection Thread[control: Socket[addr=myhost1/172.32.29.20,port=18861,localport=1843],5,] (4sl3l3k-n6bz0c-g2mqho1y-1-g2mqqj8h-j6: requesting new control connection
3. Then on node 1:
2009-11-30 16:25:45,706 [Thread-75] ERROR jboss.jms.client.FailoverCommandCenter : Failover failed javax.jms.InvalidDestinationException: No such destination: JBossTemporaryQueue[wj-xktxqm2g-1-ci1oqm2g-mn37k7-l3l3ls4] has it been deployed? at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegateInternal(ServerSessionEndpoint.java:1891) at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:262) at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:94) at org.jboss.jms.server.endpoint.advised.SessionAdvised$createConsumerDelegate_8721389917985689973.invokeTarget(SessionAdvised$createConsumerDelegate_8721389917985689973.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:123) at sun.reflect.GeneratedMethodAccessor3253.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.server.endpoint.advised.SessionAdvised.createConsumerDelegate(SessionAdvised.java) at org.jboss.jms.wireformat.SessionCreateConsumerDelegateRequest.serverInvoke(SessionCreateConsumerDelegateRequest.java:100) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:897) at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:768) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:721) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:575) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234) at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:213) at org.jboss.remoting.Client.invoke(Client.java:1917) at org.jboss.remoting.Client.invoke(Client.java:768) at org.jboss.remoting.Client.invoke(Client.java:756) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160) at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$createConsumerDelegate$aop(ClientSessionDelegate.java:267) at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeTarget(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) at org.jboss.jms.client.container.StateCreationAspect.handleCreateConsumerDelegate(StateCreationAspect.java:136) at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateConsumerDelegate_13797435.invoke(StateCreationAspect_z_handleCreateConsumerDelegate_13797435.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.container.ConsumerAspect.handleCreateConsumerDelegate(ConsumerAspect.java:76) at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect_z_handleCreateConsumerDelegate_13797435.invoke(ConsumerAspect_z_handleCreateConsumerDelegate_13797435.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.delegate.ClientSessionDelegate.createConsumerDelegate(ClientSessionDelegate.java) at org.jboss.jms.client.state.SessionState.synchronizeWith(SessionState.java:249) at org.jboss.jms.client.delegate.ClientSessionDelegate.synchronizeWith(ClientSessionDelegate.java:128) at org.jboss.jms.client.state.ConnectionState.synchronizeWith(ConnectionState.java:177) at org.jboss.jms.client.delegate.ClientConnectionDelegate.synchronizeWith(ClientConnectionDelegate.java:129) at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:145) at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:62) at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:88) at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:498) org.jboss.jms.exception.MessagingJMSException: Failed to invoke at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:271) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205) at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165) at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:495) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeTarget(ClientSessionDelegate$send_6145266547759487588.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661) at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleSend_13797435.invoke(SessionAspect_z_handleSend_13797435.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java) at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:276) at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect_z_handleSend_13797435.invoke(ProducerAspect_z_handleSend_13797435.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:165) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:208) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:146) ............................... at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) at java.lang.Thread.run(Thread.java:595) Caused by: java.lang.Exception: Can not make remoting client invocation due to not being connected to server. at org.jboss.remoting.Client.invoke(Client.java:1922) at org.jboss.remoting.Client.invoke(Client.java:768) at org.jboss.remoting.Client.invokeOneway(Client.java:817) at org.jboss.remoting.Client.invokeOneway(Client.java:1003) at org.jboss.remoting.Client.invokeOneway(Client.java:994) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180) ... 102 more 2009-11-30 16:25:45,815 [Thread-75] ERROR jms.client.container.ConnectionFailureListener : Caught exception in handling failure javax.jms.InvalidDestinationException: No such destination: JBossTemporaryQueue[wj-xktxqm2g-1-ci1oqm2g-mn37k7-l3l3ls4] has it been deployed? at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegateInternal(ServerSessionEndpoint.java:1891) at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:262) at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:94) at org.jboss.jms.server.endpoint.advised.SessionAdvised$createConsumerDelegate_8721389917985689973.invokeTarget(SessionAdvised$createConsumerDelegate_8721389917985689973.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:123) at sun.reflect.GeneratedMethodAccessor3253.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.server.endpoint.advised.SessionAdvised.createConsumerDelegate(SessionAdvised.java) at org.jboss.jms.wireformat.SessionCreateConsumerDelegateRequest.serverInvoke(SessionCreateConsumerDelegateRequest.java:100) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:897) at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:768) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:721) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:575) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234) at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:213) at org.jboss.remoting.Client.invoke(Client.java:1917) at org.jboss.remoting.Client.invoke(Client.java:768) at org.jboss.remoting.Client.invoke(Client.java:756) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160) at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$createConsumerDelegate$aop(ClientSessionDelegate.java:267) at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeTarget(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) at org.jboss.jms.client.container.StateCreationAspect.handleCreateConsumerDelegate(StateCreationAspect.java:136) at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateConsumerDelegate_13797435.invoke(StateCreationAspect_z_handleCreateConsumerDelegate_13797435.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.container.ConsumerAspect.handleCreateConsumerDelegate(ConsumerAspect.java:76) at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect_z_handleCreateConsumerDelegate_13797435.invoke(ConsumerAspect_z_handleCreateConsumerDelegate_13797435.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.delegate.ClientSessionDelegate.createConsumerDelegate(ClientSessionDelegate.java) at org.jboss.jms.client.state.SessionState.synchronizeWith(SessionState.java:249) at org.jboss.jms.client.delegate.ClientSessionDelegate.synchronizeWith(ClientSessionDelegate.java:128) at org.jboss.jms.client.state.ConnectionState.synchronizeWith(ConnectionState.java:177) at org.jboss.jms.client.delegate.ClientConnectionDelegate.synchronizeWith(ClientConnectionDelegate.java:129) at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:145) at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:62) at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:88) at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:498)
We saw the similar issue with JBoss messaing 1.4.2.SP1, but still got it with JBoss messaging 1.4.6, as a result, the two nodes lost connection with each other, can you please advise what could be the cause? Thanks in advance.
