5 Replies Latest reply on Feb 15, 2007 7:04 PM by ron_sigal

    Client timeouts while waiting for response from server

    tomerc

      Hi,

      I am running JBoss 4.0.4GA over Redhat AS 4 with jboss remoting version 1.4.3GA and i am expriencing the following problem:

      Under heavy load on the client machines the following exception is shown:

      Socket timed out. Waited 1800000 milliseconds for response while calling on InvokerLocator


      this error appears on all of the client machines for afew minutes and afterwards the following error is shown:

      Can not get connection to server. Problem establishing socket connection.


      Upgrading to version 1.4.5GA as the following link suggests , did not solve the problem
      http://www.jboss.com/index.html?module=bb&op=viewtopic&t=80814


      When the problem occurs all of the connections to port 8084 on the server machine are either in SYN_RECV or in CLOSE_WAITING state, only a restart to the jboss process solves the problem for afew hours.

      Any help would be highly appreciated

      Thanks alot.
      Tomer

        • 1. Re: Client timeouts while waiting for response from server

          Is this with use of ejb3 or JBoss Remoting directly? How many clients are being used? Do they make frequent or infrequent invocations requests (meaning client continually making call or just every few minutes)? What is the configuration for org.jboss.remoting.transport.Connector?

          • 2. Re: Client timeouts while waiting for response from server
            tomerc

            Hi,

            First of all thank you for the quick reply.

            We are working directly with jboss remoting (without any ejb), we currently have 4 clients machines talking to one server. the client machines are under a heavy load and are constantly quering the server.

            The configuration for the connector is :

            <mbean code="org.jboss.remoting.transport.Connector" xmbean-dd="org/jboss/remoting/transport/Connector.xml"
             name="jboss.remoting:service=Connector,transport=Socket"
             display-name="Socket Transport Connector">
             <depends>jboss.remoting:service=MulticastDetector,transport=multicast</depends>
             <depends>SIM:service=Storage</depends>
             <!--attribute name="InvokerLocator"><![CDATA[socket://storage:8084]]></attribute-->
             <!--attribute name="InvokerLocator"><![CDATA[socket://${bind.addrres}:8084]]></attribute-->
             <attribute name="InvokerLocator"><![CDATA[socket://${bind.addrres}:8084/?simStorageMode=${storage.mode}]]></attribute>
             <attribute name="Configuration">
             <config>
             <handlers>
             <handler subsystem="get">com.flash.storage.handlers.server.GetServerInvocationHandler</handler>
             <handler subsystem="remove">com.flash.storage.handlers.server.RemoveServerInvocationHandler</handler>
             <handler subsystem="set">com.flash.storage.handlers.server.SetServerInvocationHandler</handler>
             <handler subsystem="exists">com.flash.storage.handlers.server.ExistsServerInvocationHandler</handler>
             </handlers>
             </config>
             </attribute>
             </mbean>


            Thank you.
            Tomer

            • 3. Re: Client timeouts while waiting for response from server
              tomerc

              After doing some further investigating i found out the problem,
              turns out there is a deadlock in BlockedLinkedQueue (from Doug Lee concurrent pacakge) for more information you can check out :
              http://altair.cs.oswego.edu/pipermail/concurrency-interest/2004-September/001037.html


              Following a stack trace of two of the jboss remoting threads :

              Thread: SocketServerInvokerThread-10.184.76.155-172 : priority:5, demon:false, threadId:954, threadState:BLOCKED, threadLockName:org.jboss.util.threadpool.MinPooledExecutor@9e264b
               EDU.oswego.cs.dl.util.concurrent.PooledExecutor.execute(PooledExecutor.java:875)
               org.jboss.util.threadpool.BasicThreadPool.execute(BasicThreadPool.java:413)
               org.jboss.util.threadpool.BasicThreadPool.runTaskWrapper(BasicThreadPool.java:192)
              
              org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:212)
              
              org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:206)
              
              org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:910)
              
              org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:788)
              
              org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:681)
              
              org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:358)
               org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:412)
              
              org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:239)
              
              
               Thread: SocketServerInvokerThread-10.184.76.155-6 : priority:5, demon:false, threadId:197, threadState:BLOCKED, threadLockName:java.lang.Object@1749fb5
               EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.offer(BoundedLinkedQueue.java:328)
              
              EDU.oswego.cs.dl.util.concurrent.PooledExecutor.execute(PooledExecutor.java:885)
              
              org.jboss.util.threadpool.BasicThreadPool.execute(BasicThreadPool.java:413)
              
              org.jboss.util.threadpool.BasicThreadPool.runTaskWrapper(BasicThreadPool.java:192)
              
              org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:212)
              
              org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:206)
              
              org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:910)
              
              org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:788)
              
              org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:681)
              
              org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:358)
              
              org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:412)
              
              org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:239)
              
              


              Is there anyway to tell jboss remoting to use a different queue ?


              Thanks.

              • 4. Re: Client timeouts while waiting for response from server

                Thanks Tomer. I have created a jira issue for this (http://jira.jboss.com/jira/browse/JBREM-658).

                It is possible to configure the thread pool implementation to use for oneway invocations. Can see http://labs.jboss.com/portal/jbossremoting/docs/guide/ch05.html#d0e970 for more information.

                • 5. Re: Client timeouts while waiting for response from server
                  ron_sigal

                  Hi Tomer,

                  Thanks for alerting us to this problem. We have changed the blocking policy from WAIT to RUN in org.jboss.remoting.Client and org.jboss.remoting.ServerInvoker.

                  Ideally, it would be nice to be able to configure the queue used in Remoting's default thread pool, org.jboss.util.threadpool.BasicThreadPool, so that the BoundedLinkQueue could be replaced with a BoundedBuffer. To that end a JIRA issue has been created in the JBoss Common project: http://jira.jboss.com/jira/browse/JBCOMMON-15.