5 Replies Latest reply on Nov 6, 2006 2:05 PM by Tom Elrod

    Socket read stuck on a client invoke call

    saravanan c Newbie

      Hi all,

      I am using JBoss Remoting 1.4.0 version. Sometimes I get into an issue where the client.invoke() gets stuck at the socketRead0 method. The request eventually times out after 30 minutes.

      I have two systems that communicate with each other. The client connector object is created and stored in a hashtable. When a method has to be called, the client connector is retrieved from this hash and then the invoke is called.

      I also have the connection listener to notify me every 10 seconds when the connection is lost. The listener implementation removes the connector object from the hash when the connection lost event is received. However, the connection listener does not get invoked until the first invoke method is completed/released.

      I reboot one of the systems. The invoke is called from the active system before the listener is called. This is the use case where I get stuck on the call.

      Is there any suggestions as to how I can work around this issue? Would upgrading to 2.0.0 help?

      Thanks
      Saravanan

        • 1. Re: Socket read stuck on a client invoke call
          Ron Sigal Master

          Hi Saravanan,

          I don't quite follow. When you say


          However, the connection listener does not get invoked until the first invoke method is completed/released.


          are you saying that the call to Client.addConnectionListener() is not made until after Client.invoke() is called for the first time? But in that case the connection listener hasn't yet been installed at the time the client side enters socketRead0(). I must be missing something.

          Is it the client or the server that you're restarting? Could you elaborate a little?





          • 2. Re: Socket read stuck on a client invoke call
            saravanan c Newbie

            Hi,

            The scenario is the following: I have two systems that communicates with each other using remoting. When each system is started, I create a client connector to send requests to peer and a server connector to receive requests from peer.

            I start both systems and the two way comms is up and running. When one node wants to invoke a API on the peer node, it calls the invoke method of the client connector. This connector has a listener for callback incase the peer is lost. The ping time for this listener is set at 10 seconds.

            In the above scenario, I reboot one machine. In about 10 seconds, I get a connect lost event on the listener. I remove the client connector object.

            The problem that I encountered was in this 10 second time frame, if I call client.invoke() API, the request gets stuck. I specified the timeout (socketTimeout) in the URI to be 120seconds. However, the request was stuck and did not timeout until 30 mins (this happens to be the default timeout).

            All other requests are blocked at this time. Even the listener call back for the connection lost is also blocked. After 30 minutes, I see a large number of queued connection lost events in the queue in the listener. Every event in the queue creates another thread and calls the listener method.

            Hope I am clear. Let me know if you need any more information

            Saravanan

            • 3. Re: Socket read stuck on a client invoke call
              saravanan c Newbie

              Hi all,

              The stack trace when this happenned is given below. I have also attached the code base where I create the client connector. Initially, I had tried with timeout specified in the URI only (No config map when I create the connector).

              I encountered the same issue. Now, I am specifying both. Would that cause any issue?

              Thanks
              Saravanan


              int timeout=120000;
              String ipAddr = System.getProperty(SharedConstants.VIRTUAL_IP_ADDRESS, "localhost");
              String locatorURI = SharedConstants.CONNECTION_PROTO_SOCK + "://" + ipAddr + ":" + SharedConstants.CONNECTION_PROTO_SOCK_PORT;
              locatorURI += "/?timeout=120000";
              // Client connector
              HashMap config = new HashMap();
              config.put("timeout", new Integer(timeout));
              config.put("socketTimeout", new Integer(timeout));
              InvokerLocator locator = new InvokerLocator(locatorURI);
              Client c = new Client(locator, config);




              Socket timed out. Waited 1800000 milliseconds for response while calling on Invok
              erLocator [socket://172.16.52.1:6450/?timeout=120000]; nested exception is:
              java.net.SocketTimeoutException: Read timed out
              java.net.SocketTimeoutException: Read timed out
              at java.net.SocketInputStream.socketRead0(Native Method)
              at java.net.SocketInputStream.read(SocketInputStream.java:129)
              at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
              at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
              at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java
              :2133)


              Caused by: java.rmi.MarshalException: Socket timed out. Waited 1800000 millisec
              onds for response while calling on InvokerLocator [socket://172.16.52.1:6450/?ti
              meout=120000]; nested exception is:
              java.net.SocketTimeoutException: Read timed out
              at org.jboss.remoting.transport.socket.SocketClientInvoker.transport(Soc
              ketClientInvoker.java:288)
              at org.jboss.remoting.RemoteClientInvoker.invoke(RemoteClientInvoker.jav
              a:136)
              at org.jboss.remoting.Client.invoke(Client.java:444)
              at org.jboss.remoting.Client.invoke(Client.java:407)
              at org.jboss.remoting.Client.invoke(Client.java:392)
              at com.zforce.objrep.DbProxy.invoke(DbProxy.java:80)
              at com.zforce.objrep.DbProxy.proxyInvoke(DbProxy.java:109)
              at com.zforce.objrep.ObjectRepository.baseGetByNameFromDb(ObjectReposito
              ry.java:840)
              at com.zforce.objrep.ObjectRepository.baseGetByName(ObjectRepository.jav
              a:560)
              at com.zforce.rain.server.base.BaseMgr.readObjectByName(BaseMgr.java:536
              )
              ... 3 more
              Caused by: java.net.SocketTimeoutException: Read timed out
              at java.net.SocketInputStream.socketRead0(Native Method)
              at java.net.SocketInputStream.read(SocketInputStream.java:129)
              at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
              at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
              at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.jav

              • 4. Re: Socket read stuck on a client invoke call
                Ron Sigal Master

                Hi Saravanan,

                In the message


                Caused by: java.rmi.MarshalException: Socket timed out. Waited 1800000 millisec
                onds for response while calling on InvokerLocator [socket://172.16.52.1:6450/?ti
                meout=120000]; nested exception is:


                1800000 really is the currently configured timeout value. I don't have an old copy of Remoting 1.4.0 handy, but I suspect that that version doesn't recognize the "timeout" parameter. Why not upgrade?


                I encountered the same issue. Now, I am specifying both. Would that cause any issue?


                The value in the InvokerLocator will override the value in the configuration map.

                • 5. Re: Socket read stuck on a client invoke call
                  Tom Elrod Master

                  Ron is correct. In version 1.4.0 of remoting, the configuration property for socket timeout is 'socketTimeout'. I don't think it was until 2.0.0 release of remoting before all the transports were standardized to use the 'timeout' property. All the socket invoker configuration properties for 1.4.0 are covered under section 4.4.4 of the JBoss_Remoting_Guide.pdf included with the JBossRemoting 1.4.0 release download.