Socket read stuck on a client invoke call

[sarbu]

Hi all,

I am using JBoss Remoting 1.4.0 version. Sometimes I get into an issue where the client.invoke() gets stuck at the socketRead0 method. The request eventually times out after 30 minutes.

I have two systems that communicate with each other. The client connector object is created and stored in a hashtable. When a method has to be called, the client connector is retrieved from this hash and then the invoke is called.

I also have the connection listener to notify me every 10 seconds when the connection is lost. The listener implementation removes the connector object from the hash when the connection lost event is received. However, the connection listener does not get invoked until the first invoke method is completed/released.

I reboot one of the systems. The invoke is called from the active system before the listener is called. This is the use case where I get stuck on the call.

Is there any suggestions as to how I can work around this issue? Would upgrading to 2.0.0 help?

Thanks
Saravanan

[ron_sigal]

Hi Saravanan,

I don't quite follow. When you say

However, the connection listener does not get invoked until the first invoke method is completed/released.

are you saying that the call to Client.addConnectionListener() is not made until after Client.invoke() is called for the first time? But in that case the connection listener hasn't yet been installed at the time the client side enters socketRead0(). I must be missing something.

Is it the client or the server that you're restarting? Could you elaborate a little?

[sarbu]

Hi,

The scenario is the following: I have two systems that communicates with each other using remoting. When each system is started, I create a client connector to send requests to peer and a server connector to receive requests from peer.

I start both systems and the two way comms is up and running. When one node wants to invoke a API on the peer node, it calls the invoke method of the client connector. This connector has a listener for callback incase the peer is lost. The ping time for this listener is set at 10 seconds.

In the above scenario, I reboot one machine. In about 10 seconds, I get a connect lost event on the listener. I remove the client connector object.

The problem that I encountered was in this 10 second time frame, if I call client.invoke() API, the request gets stuck. I specified the timeout (socketTimeout) in the URI to be 120seconds. However, the request was stuck and did not timeout until 30 mins (this happens to be the default timeout).

All other requests are blocked at this time. Even the listener call back for the connection lost is also blocked. After 30 minutes, I see a large number of queued connection lost events in the queue in the listener. Every event in the queue creates another thread and calls the listener method.

Hope I am clear. Let me know if you need any more information

Saravanan

[sarbu]

Hi all,

The stack trace when this happenned is given below. I have also attached the code base where I create the client connector. Initially, I had tried with timeout specified in the URI only (No config map when I create the connector).

I encountered the same issue. Now, I am specifying both. Would that cause any issue?

Thanks
Saravanan


int timeout=120000;
String ipAddr = System.getProperty(SharedConstants.VIRTUAL_IP_ADDRESS, "localhost");
String locatorURI = SharedConstants.CONNECTION_PROTO_SOCK + "://" + ipAddr + ":" + SharedConstants.CONNECTION_PROTO_SOCK_PORT;
locatorURI += "/?timeout=120000";
// Client connector
HashMap config = new HashMap();
config.put("timeout", new Integer(timeout));
config.put("socketTimeout", new Integer(timeout));
InvokerLocator locator = new InvokerLocator(locatorURI);
Client c = new Client(locator, config);




Socket timed out. Waited 1800000 milliseconds for response while calling on Invok
erLocator [socket://172.16.52.1:6450/?timeout=120000]; nested exception is:
java.net.SocketTimeoutException: Read timed out
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java
:2133)


Caused by: java.rmi.MarshalException: Socket timed out. Waited 1800000 millisec
onds for response while calling on InvokerLocator [socket://172.16.52.1:6450/?ti
meout=120000]; nested exception is:
java.net.SocketTimeoutException: Read timed out
at org.jboss.remoting.transport.socket.SocketClientInvoker.transport(Soc
ketClientInvoker.java:288)
at org.jboss.remoting.RemoteClientInvoker.invoke(RemoteClientInvoker.jav
a:136)
at org.jboss.remoting.Client.invoke(Client.java:444)
at org.jboss.remoting.Client.invoke(Client.java:407)
at org.jboss.remoting.Client.invoke(Client.java:392)
at com.zforce.objrep.DbProxy.invoke(DbProxy.java:80)
at com.zforce.objrep.DbProxy.proxyInvoke(DbProxy.java:109)
at com.zforce.objrep.ObjectRepository.baseGetByNameFromDb(ObjectReposito
ry.java:840)
at com.zforce.objrep.ObjectRepository.baseGetByName(ObjectRepository.jav
a:560)
at com.zforce.rain.server.base.BaseMgr.readObjectByName(BaseMgr.java:536
)
... 3 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.jav

[ron_sigal]

Hi Saravanan,

In the message

Caused by: java.rmi.MarshalException: Socket timed out. Waited 1800000 millisec
onds for response while calling on InvokerLocator [socket://172.16.52.1:6450/?ti
meout=120000]; nested exception is:

1800000 really is the currently configured timeout value. I don't have an old copy of Remoting 1.4.0 handy, but I suspect that that version doesn't recognize the "timeout" parameter. Why not upgrade?

I encountered the same issue. Now, I am specifying both. Would that cause any issue?

The value in the InvokerLocator will override the value in the configuration map.

[tom.elrod]

Ron is correct. In version 1.4.0 of remoting, the configuration property for socket timeout is 'socketTimeout'. I don't think it was until 2.0.0 release of remoting before all the transports were standardized to use the 'timeout' property. All the socket invoker configuration properties for 1.4.0 are covered under section 4.4.4 of the JBoss_Remoting_Guide.pdf included with the JBossRemoting 1.4.0 release download.