Remoting issue

[ron_sigal]

A thread in the Remoting user forum ("JBREM-877: New Socket Connection is being Created for Every" - http://www.jboss.com/index.html?module=bb&op=viewtopic&t=126382) has brought up an interesting issue concerning the use of Remoting in EJB3.

In AS 4.0.4/4.0.5, when EJB3 was using Remoting 1.4.x, each call to org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke() created a new org.jboss.remoting.Client, and when it returned, it left the underlying client invoker intact in the org.jboss.remoting.InvokerRegistry. The next call found the existing client invoker and reused its socket pool. (Simplifying somewhat, but basically true.)

Now, in AS 4.2.x/5.0.0, each call to InvokeRemoteInterceptor.invoke() calls Client.disconnect() before returning, which, if the reference count of the underlying client invoker goes to zero, leads to the destruction of the client invoker, along with its socket pool. So, unless there are multiple proxies in the same JVM running simultaneously and keeping the client invoker's reference count above zero, each invocation will lead to the creation of a new socket. The guy that posted JBREM-877 says that performance is 25% of what it was in AS 4.0.4.

One solution would be to add a feature which allows a delay in the destruction of the client invoker. E.g., the invoker locator

socket://${jboss.bind.address}:3873/?invokerdestructiondelay=5000

would introduce a 5 second delay before the client invoker gets destroyed, increasing the probability that it would be reused. The default behavior would be to destroy the client invoker immediately.

Any thoughts from the EJB3 team?

-Ron

[ron_sigal]

I went ahead and added the delayed destruction feature to the Remoting 2.4 branch. In a pure Remoting in-jvm test, I got a speed up of about 2 by using delayed client invoker destruction.

For more information, see the thread http://www.jboss.com/index.html?module=bb&op=viewtopic&t=126382&postdays=0&postorder=asc&start=10.

[ron_sigal]

"carlos" wrote:

I think Bill would like a patch for EAP 4.2 as well.

Bill as in [e-mail], the original Bill? Or the other Patriot fan Bill?

In any case, since this is a Remoting feature, it might be good if someone from EJB3 created a JBPAPP issue asking for it.

[alrubinger]

Bill as in ... the original Bill? Or the other Patriot fan Bill?

The original Bill DeCoste. ;)

S,
ALR

[bdecoste]

Oh god. I'm good. I have the remoting version that I need.

[hamdurrahman]

I am still getting the exception even after adding invokerDestructionDelay.
The log has changed and we can see the IP as well as port number(This is changing) and also the WorkerThread#_ is changing

11:14:41,353 ERROR [ServerThread] WorkerThread#3[150.158.75.156:1406] exception occurred during first invocation
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1784)
at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:691)
at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(JavaSerializationManager.java:121)
at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(JavaSerializationManager.java:95)
at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
at org.jboss.remoting.transport.socket.ServerThread.versionedWrite(ServerThread.java:998)
at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:781)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:695)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:522)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:230)

[ron_sigal]

This issue has come up again recently, and I've ported the "invoker destruction delay" feature to the Remoting 2.2 branch so that it will appear in release 2.2.3.SP1. I've been informally suggesting 5 seconds as a reasonable default delay, but I'm wondering if anyone from the EJB3 team has any better intuition.

Thanks,
Ron

[jaikiran]

Hi Ron,

A few questions

1) Is the default destruction delay being maintained programatically within Remoting? I don't see this delay being added within the EJB3 remoting configurations

2) From what i understand in those threads, the issue was a socket being created on every method invocation on a remote EJB3 proxy. The destruction delay tries to fix this by delaying the destruction of the socket (and the subsequent creation of a new one). Did i understand this right?

I've been informally suggesting 5 seconds as a reasonable default delay, but I'm wondering if anyone from the EJB3 team has any better intuition.

Assuming, i understood the issue correctly, i guess the ideal timeout really depends on how the end users use the EJB3 proxies. Some users might have this:

someClientMethod()
{
// lookup remote bean
Bean bean = ctx.lookup(...);
// invoke method
bean.doSomething(); // creates a socket

// do something that takes a really long time
waitForUserInputOrSomeOtherTimeConsumingOperation();

// use the earlier proxy again
bean.doSomeOtherThing(); // creates a socket (again)
}

Although if the proxy is being used for multiple method invocations within that default timeout then the performance improvement is going to be noticeable.

3) What is the (performance) impact of specifying a very long destruction delay? Does it degrade the performance or introduce any other issues?