1 Reply Latest reply on Dec 3, 2009 1:30 AM by Yong Hao Gao

    Failure on control connection Thread?

    Yong Zhang Newbie

      Hi,

      * JBoss 5.0.1.GA
      * JBoss messaging 1.4.6.GA
      * JBoss remoting 2.5.2
      * Java jdk1.5.0_19

      We have a cluster with two nodes using clustered queue, and recently we are getting the following errors with JBoss messaging:

      1. The following appears in Node 1:

      2009-11-30 16:21:47,525 [Timer-6] WARN remoting.transport.bisocket.BisocketServerInvoker : org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@10d107: detected failure on control connection Thread[control: Socket[addr=myhost2/172.32.29.21,port=32062,localport=1881],5,] (4sl3l3l-7k73nm-g2mqo1ic-1-g2mqxt8r-jn: requesting new control connection
      


      2. After a few minutes, on node 2[172.32.29.21]:

      2009-11-30 16:24:45,243 [Timer-3] WARN remoting.transport.bisocket.BisocketServerInvoker : org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@17663eb: detected failure on control connection Thread[control: Socket[addr=myhost2/172.32.29.20,port=18861,localport=1686],5,] (4sl3l3k-n6bz0c-g2mqho1y-1-g2mqqj8h-j6: requesting new control connection
      2009-11-30 16:24:49,886 [Timer-7] WARN remoting.transport.bisocket.BisocketClientInvoker : Unable to send ping: shutting down PingTimerTask
      java.net.SocketException: Software caused connection abort: socket write error
       at java.net.SocketOutputStream.socketWrite0(Native Method)
       at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
       at java.net.SocketOutputStream.write(SocketOutputStream.java:115)
       at org.jboss.remoting.transport.bisocket.BisocketClientInvoker$PingTimerTask.run(BisocketClientInvoker.java:723)
       at java.util.TimerThread.mainLoop(Timer.java:512)
       at java.util.TimerThread.run(Timer.java:462)
      2009-11-30 16:25:45,864 [WorkerThread#0[172.21.129.129:1995]] ERROR jboss.messaging.util.ExceptionUtil : SessionEndpoint[l31-8c8rsm2g-1-y1ohqm2g-c0zb6n-k3l3ls4] createConsumerDelegate [m31-oc8rsm2g-1-y1ohqm2g-c0zb6n-k3l3ls4]
      javax.jms.InvalidDestinationException: No such destination: JBossTemporaryQueue[wj-xktxqm2g-1-ci1oqm2g-mn37k7-l3l3ls4] has it been deployed?
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegateInternal(ServerSessionEndpoint.java:1891)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:262)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:94)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$createConsumerDelegate_8721389917985689973.invokeTarget(SessionAdvised$createConsumerDelegate_8721389917985689973.java)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
       at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:123)
       at sun.reflect.GeneratedMethodAccessor3253.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.createConsumerDelegate(SessionAdvised.java)
       at org.jboss.jms.wireformat.SessionCreateConsumerDelegateRequest.serverInvoke(SessionCreateConsumerDelegateRequest.java:100)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:897)
       at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:768)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:721)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:575)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
      2009-11-30 16:30:55,300 [Timer-3] WARN remoting.transport.bisocket.BisocketServerInvoker : org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@17663eb: detected failure on control connection Thread[control: Socket[addr=myhost1/172.32.29.20,port=18861,localport=1843],5,] (4sl3l3k-n6bz0c-g2mqho1y-1-g2mqqj8h-j6: requesting new control connection
      


      3. Then on node 1:
      2009-11-30 16:25:45,706 [Thread-75] ERROR jboss.jms.client.FailoverCommandCenter : Failover failed
      javax.jms.InvalidDestinationException: No such destination: JBossTemporaryQueue[wj-xktxqm2g-1-ci1oqm2g-mn37k7-l3l3ls4] has it been deployed?
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegateInternal(ServerSessionEndpoint.java:1891)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:262)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:94)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$createConsumerDelegate_8721389917985689973.invokeTarget(SessionAdvised$createConsumerDelegate_8721389917985689973.java)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
       at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:123)
       at sun.reflect.GeneratedMethodAccessor3253.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.createConsumerDelegate(SessionAdvised.java)
       at org.jboss.jms.wireformat.SessionCreateConsumerDelegateRequest.serverInvoke(SessionCreateConsumerDelegateRequest.java:100)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:897)
       at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:768)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:721)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:575)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
       at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:213)
       at org.jboss.remoting.Client.invoke(Client.java:1917)
       at org.jboss.remoting.Client.invoke(Client.java:768)
       at org.jboss.remoting.Client.invoke(Client.java:756)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$createConsumerDelegate$aop(ClientSessionDelegate.java:267)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeTarget(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
       at org.jboss.jms.client.container.StateCreationAspect.handleCreateConsumerDelegate(StateCreationAspect.java:136)
       at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateConsumerDelegate_13797435.invoke(StateCreationAspect_z_handleCreateConsumerDelegate_13797435.java)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.container.ConsumerAspect.handleCreateConsumerDelegate(ConsumerAspect.java:76)
       at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect_z_handleCreateConsumerDelegate_13797435.invoke(ConsumerAspect_z_handleCreateConsumerDelegate_13797435.java)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.createConsumerDelegate(ClientSessionDelegate.java)
       at org.jboss.jms.client.state.SessionState.synchronizeWith(SessionState.java:249)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.synchronizeWith(ClientSessionDelegate.java:128)
       at org.jboss.jms.client.state.ConnectionState.synchronizeWith(ConnectionState.java:177)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate.synchronizeWith(ClientConnectionDelegate.java:129)
       at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:145)
       at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:62)
       at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:88)
       at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:498)
      
      org.jboss.jms.exception.MessagingJMSException: Failed to invoke
       at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:271)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:495)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeTarget(ClientSessionDelegate$send_6145266547759487588.java)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
       at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661)
       at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleSend_13797435.invoke(SessionAspect_z_handleSend_13797435.java)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
       at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:276)
       at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect_z_handleSend_13797435.invoke(ProducerAspect_z_handleSend_13797435.java)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:165)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:208)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:146)
       ...............................
      
       at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
       at java.lang.Thread.run(Thread.java:595)
      Caused by: java.lang.Exception: Can not make remoting client invocation due to not being connected to server.
       at org.jboss.remoting.Client.invoke(Client.java:1922)
       at org.jboss.remoting.Client.invoke(Client.java:768)
       at org.jboss.remoting.Client.invokeOneway(Client.java:817)
       at org.jboss.remoting.Client.invokeOneway(Client.java:1003)
       at org.jboss.remoting.Client.invokeOneway(Client.java:994)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180)
       ... 102 more
      2009-11-30 16:25:45,815 [Thread-75] ERROR jms.client.container.ConnectionFailureListener : Caught exception in handling failure
      javax.jms.InvalidDestinationException: No such destination: JBossTemporaryQueue[wj-xktxqm2g-1-ci1oqm2g-mn37k7-l3l3ls4] has it been deployed?
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegateInternal(ServerSessionEndpoint.java:1891)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:262)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:94)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$createConsumerDelegate_8721389917985689973.invokeTarget(SessionAdvised$createConsumerDelegate_8721389917985689973.java)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
       at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:123)
       at sun.reflect.GeneratedMethodAccessor3253.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.createConsumerDelegate(SessionAdvised.java)
       at org.jboss.jms.wireformat.SessionCreateConsumerDelegateRequest.serverInvoke(SessionCreateConsumerDelegateRequest.java:100)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:897)
       at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:768)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:721)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:575)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
       at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:213)
       at org.jboss.remoting.Client.invoke(Client.java:1917)
       at org.jboss.remoting.Client.invoke(Client.java:768)
       at org.jboss.remoting.Client.invoke(Client.java:756)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$createConsumerDelegate$aop(ClientSessionDelegate.java:267)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeTarget(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
       at org.jboss.jms.client.container.StateCreationAspect.handleCreateConsumerDelegate(StateCreationAspect.java:136)
       at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateConsumerDelegate_13797435.invoke(StateCreationAspect_z_handleCreateConsumerDelegate_13797435.java)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.container.ConsumerAspect.handleCreateConsumerDelegate(ConsumerAspect.java:76)
       at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect_z_handleCreateConsumerDelegate_13797435.invoke(ConsumerAspect_z_handleCreateConsumerDelegate_13797435.java)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.createConsumerDelegate(ClientSessionDelegate.java)
       at org.jboss.jms.client.state.SessionState.synchronizeWith(SessionState.java:249)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.synchronizeWith(ClientSessionDelegate.java:128)
       at org.jboss.jms.client.state.ConnectionState.synchronizeWith(ConnectionState.java:177)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate.synchronizeWith(ClientConnectionDelegate.java:129)
       at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:145)
       at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:62)
       at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:88)
       at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:498)
      


      We saw the similar issue with JBoss messaing 1.4.2.SP1, but still got it with JBoss messaging 1.4.6, as a result, the two nodes lost connection with each other, can you please advise what could be the cause? Thanks in advance.