Threads Hanging in MicroSocketClientInvoker.readVersion()

[snacker]

Note: this issue was originally reported here:

https://community.jboss.org/message/548382#548382

On the web server side we see occasionally see threads with this stack:

75: [Thread[jrpp-1390,5,jrpp]]/id=49440/hc=13398717/shc=*/state=RUNNABLE/name=*
        java.net.SocketInputStream.socketRead0(Native Method)
        java.net.SocketInputStream.read(SocketInputStream.java:129)
        java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
        java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2429)
        java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2499)
        java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2571)
        java.io.ObjectInputStream.read(ObjectInputStream.java:820)
        org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1263)
        org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:838)
        org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:165)
        org.jboss.remoting.Client.invoke(Client.java:1724)
        org.jboss.remoting.Client.invoke(Client.java:629)
        org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
        org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
        org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
        org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
        org.jboss.ejb3.security.client.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
        org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
        org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:74)
        org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
        org.jboss.aspects.remoting.PojiProxy.invoke(PojiProxy.java:62)
        $Proxy5.invoke(Unknown Source)
        org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:207)
        org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:164)
        $Proxy1643.completeMyOrder(Unknown Source)

        ... rest of our business process thread stack...

On the jboss server for each thread above we usually see at least 2 jboss threads with the following stack:

816: [WorkerThread#10[10.1.4.245:46454]]/id=6923/hc=1806701791/shc=*/state=RUNNABLE/name=*
         java.net.SocketInputStream.socketRead0(Native Method)
         java.net.SocketInputStream.read(SocketInputStream.java:129)
         java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
         java.io.BufferedInputStream.read(BufferedInputStream.java:237)
         java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
         java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2429)
         java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2499)
         java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2571)
         java.io.ObjectInputStream.read(ObjectInputStream.java:820)
         org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:1038)
         org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:673)
         org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551)
         org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)

I am not sure what is going on, but it looks like there is something that occasionally gets messed up with these "readVersion(...)" methods.

These threads will hang for hours and days.

The only way we can get these to stop hanging is to send a Thread.interrupt() to the threads on the web server side.

Sending the interrupt to the thread on the jboss side only moves the problem to a different thread and doesn't stop the web server threads from hanging.

We are not sure what is causing this, but it appears to be a bug in jboss.

We are using 5.1.0 GA.

It looks like the class allows for timeout and retry settings, but I am not sure how to set these on the client side.

http://anonsvn.jboss.org/repos/jbossremoting/remoting2/tags/2.5.1/src/main/org/jboss/remoting/transport/socket/MicroSocketClientInvoker.java

The MANIFEST.MF of our jboss-remoting.jar shows version "2.5.1"

Is this related to https://jira.jboss.org/browse/JBREM-1188 ?

If so, I think the default of 30 seconds would be too long for us.

Is there a way to change that via a config file on the client side?

[ron_sigal]

Hi Nathan,

Note that I moved this discussion to the JBossRemoting users forum.

There are two different phenomena in your stacktraces.  The first one shows a client hanging, waiting for a response to an invocation, in this case the response to an invocation on an EJB3.  The fact that it's hanging means that, for some reason, the server isn't supplying a result.  Do you see anything going on in server.log?

The second stacktrace shows a worker thread sitting in a read() call, waiting for some work to do.  It's perfectly natural.

A timeout value for both the client side read() and the server side read() can be configured on the server.  Where you do the configuring depends on which subsystem you want to configure.  For EJB3, for example, go to $JBOSS_HOME/server/$CONFIG/deploy/ejb3-connectors-jboss-beans.xml and add "timeout" to the "InvokerLocator" property.  For example,

<property name="invokerLocator">

      <value-factory bean="ServiceBindingManager" method="getStringBinding">
        <parameter>
          jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3
        </parameter>
        <parameter>
          <null />
        </parameter>
        <parameter>socket://${jboss.bind.address}:${port}/?timeout=60000</parameter>
        <parameter>
          <null />
        </parameter>
        <parameter>3873</parameter>
      </value-factory>

would set the timeout to one minute on both the client and server side.

-Ron

[snacker]

When we see the clients hanging there are no threads running in any of our code (as viewe by jstack and by the thread dump in the jmx-console).

We don't see anything useful in the jboss nor server logs on the client nor server side (i.e errors, network interface messages, etc.)

What we've done is updated jboss-remoting.jar to version 2.5.2.SP3 on all of the clients and on the jboss servers.

We have not seen any issues since then.

I will update this if we start seeing them again.

Thanks for the info about setting the timeouts.

That will be useful if we do start seeing this again.

(btw, is that documented somewhere?)

[ron_sigal]

Nathan Ciliberto wrote:

.

(btw, is that documented somewhere?)

Chapter 5 of the Remoting Guide (http://docs.jboss.org/jbossremoting/2.5.3.SP1/html/) is all about configuring Remoting.

[snacker]

I will take a look.

Thanks again!