SocketClientInvoker - NEW ClientSocketWrapper Errors

[vbs_jboss_accnt]

We have a messaging server running with the following versions and have a few clients that connect to this messaging server (with the same version of jar files....)

jboss messaging 1.4.6

jboss remoting 2.5.2

Once in a few days , We get this error on the server debug logs and the whole messaging server freezes and we need to restart the server to make it work again -

The odd thing being the IP address (X) on the thread name is different from the bisocket's one...(Is this OK ? Is this because that worker thread was created for an ip X and then went back to the pool and is being reused for another ip Y? ).

Any inputs to fix this issue is appreciated..... What does this exception mean ???

DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker,] (WorkerThread#194[X:46255]) SocketClientInvoker[6ffe4080, bisocket://Y:950817640] got exception: NEW ClientSocketWrapper[Socket[addr=/Y,port=1640,localport=14396].e4e0c40]
java.io.IOException: Connection reset
                at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
                at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
                at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
                at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
                at java.io.DataOutputStream.flush(DataOutputStream.java:106)
                at org.jboss.jms.wireformat.ClientDelivery.write(ClientDelivery.java:93)
                at org.jboss.jms.wireformat.JMSWireFormat.write(JMSWireFormat.java:237)
                at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:1325)
                at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:861)
                at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:461)
                at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167)
                at org.jboss.remoting.Client.invoke(Client.java:1917)
                at org.jboss.remoting.Client.invoke(Client.java:768)
                at org.jboss.remoting.Client.invokeOneway(Client.java:817)
                at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:835)
                at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:708)
                at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1467)
                at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1379)
                at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:327)
                at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
                at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:583)
                at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
                at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:665)
                at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:506)
                at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:246)
                at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2345)
                at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:515)
                at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:777)
                at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399)
                at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
                at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
                at sun.reflect.GeneratedMethodAccessor234.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
                at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
                at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)
                at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:897)
                at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:768)
                at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:721)
                at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:575)
                at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)

----Continuation of  this stack trace  i found these  errors too....

SocketClientInvoker[6ffe4080, bisocket://:950817640] unable to write Bisocket.CREATE_ORDINARY_SOCKET
java.net.SocketException: Broken pipe
                at java.net.SocketOutputStream.socketWrite0(Native Method)
                at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
                at java.net.SocketOutputStream.write(SocketOutputStream.java:115)
                at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:495)
                at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1165)
                at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:816)
                at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:461)
                at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167)
                at org.jboss.remoting.Client.invoke(Client.java:1917)
                at org.jboss.remoting.Client.invoke(Client.java:768)
                at org.jboss.remoting.Client.invokeOneway(Client.java:817)
                at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:835)
                at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:708)
                at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1467)
                at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1379)
                at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:327)
                at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
                at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:583)
                at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
                at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:665)
                at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:506)
                at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:246)
                at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2345)
                at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:515)
                at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:777)
                at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399)
                at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
                at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
                at sun.reflect.GeneratedMethodAccessor234.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
                at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
                at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)
                at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:897)
                at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:768)
                at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:721)
                at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:575)
                at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)

[ron_sigal]

Hi Vidhya,

Re: "The odd thing being the IP address (X) on the thread name is different  from the bisocket's one...(Is this OK ? Is this because that worker  thread was created for an ip X and then went back to the pool and is  being reused for another ip Y? )"

When a ServerThread is retrieved from the pool its name is reset to reflect its new socket.  The address in ServerThread.toString() is generated by a call to Socket.getInetAddress().getHostAddress() and the address in ClientSocketWrapper is generated in Socket.toString() by a call to java.net.SocketImp.getInetAddress().  I would expect them to refer to the same host, but, possibly, look a little different.  Are they actually referring to distinct hosts?

Both of your stacktraces indicate that something has happened to a client.  In the first one, JBossMessaging is trying to send some messages to a consumer and the connection has been closed on the client side.  The second stacktrace suggests that, in response to the broken connection, Remoting is trying to create a new connection by writing a CREATE_ORDINARY_SOCKET message on the bisocket control connection (for the details see Section  5.4.17 "Bisocket transport" of the Remoting Guide at http://docs.jboss.org/jbossremoting/2.5.3.SP1/html/).  So, I don't know what the problem is, but you should check out what's going on at the client side.

-Ron