remote client cannot connect: detected failure on control co

[bartvh]

I have a remote client connecting to a clustered jboss-messaging server, using jboss 4.2.0. The JBM version is 1.3.0.GA. Because of some experimentation I frequently had restarted my client application. After a number of restarts, I get into a situation where the client is unable to reconnect to the JMS server.

6:02:28,612 WARN [BisocketServerInvoker] org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@1272332: detected failure on control connection Thread[control: Socket[addr=/127.1.0.1,port=3652,localport=34297],5,jboss]: requesting new control connection

The client is retrying to connect to the server in case of failures. At everty retry this message is printed out. Restarting the client does not help to fix the problem, but restarting the server does.

Note: this might be a pure remoting issue, but I'm not sure, so this is posted in the jbm forum.

[bartvh]

I want to add that when is problem occurs, shutting down the server hangs. I did a stack trace dump and it shows:

"JBoss Shutdown Hook" daemon prio=1 tid=0x08abc388 nid=0x2d94 waiting for monitor entry [0x85adb000..0x85adf030]
 at org.jboss.remoting.ServerInvoker.removeCallbackHandler(ServerInvoker.java:1671)
 - waiting to lock <0x911bbb28> (a java.util.HashMap)
 at org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1341)
 at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:552)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:721)
 at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
 at org.jboss.remoting.Client.invoke(Client.java:1550)
 at org.jboss.remoting.Client.invoke(Client.java:530)
 at org.jboss.remoting.Client.removeListener(Client.java:1043)
 at org.jboss.jms.client.remoting.JMSRemotingConnection.stop(JMSRemotingConnection.java:297)
 at org.jboss.jms.client.container.ConnectionAspect.handleClose(ConnectionAspect.java:181)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
 at org.jboss.jms.client.delegate.ClientConnectionDelegate$close_N4742752445160157748.invokeNext(ClientConnectionDelegate$close_N4742752445160157748.java)
 at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:91)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
 at org.jboss.jms.client.delegate.ClientConnectionDelegate$close_N4742752445160157748.invokeNext(ClientConnectionDelegate$close_N4742752445160157748.java)
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
 at org.jboss.jms.client.delegate.ClientConnectionDelegate$close_N4742752445160157748.invokeNext(ClientConnectionDelegate$close_N4742752445160157748.java)
 at org.jboss.jms.client.delegate.ClientConnectionDelegate.close(ClientConnectionDelegate.java)
 at org.jboss.jms.client.JBossConnection.close(JBossConnection.java:132)
 at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerStopDelivery(JMSContainerInvoker.java:944)
 at org.jboss.ejb.plugins.jms.JMSContainerInvoker.stopService(JMSContainerInvoker.java:863)
 at org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:315)
 at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:247)
 at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)

and

"WorkerThread#2[127.1.0.1:48325]" prio=1 tid=0x080e0e10 nid=0xb97 in Object.wait() [0x82ada000..0x82adaeb0]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x94f79fa0> (a java.util.HashSet)
 at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.<init>(BisocketClientInvoker.java:206)
 - locked <0x94f79fa0> (a java.util.HashSet)
 at org.jboss.remoting.transport.bisocket.TransportClientFactory.createClientInvoker(TransportClientFactory.java:44)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.remoting.InvokerRegistry.loadClientInvoker(InvokerRegistry.java:419)
 at org.jboss.remoting.InvokerRegistry.createClientInvoker(InvokerRegistry.java:320)
 - locked <0x90b20338> (a java.lang.Object)
 at org.jboss.remoting.Client.connect(Client.java:441)
 at org.jboss.remoting.callback.ServerInvokerCallbackHandler.init(ServerInvokerCallbackHandler.java:167)
 at org.jboss.remoting.callback.ServerInvokerCallbackHandler.<init>(ServerInvokerCallbackHandler.java:139)
 at org.jboss.remoting.ServerInvoker.getCallbackHandler(ServerInvoker.java:1655)
 - locked <0x911bbb28> (a java.util.HashMap)
 at org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1336)
 at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:552)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:721)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:560)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:383)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)

Looks like a thread is holding some lock and waiting to get notified of something...