7 Replies Latest reply on Jul 28, 2009 5:42 AM by jaikiran pai

    Remoting issue

    Ron Sigal Master

      A thread in the Remoting user forum ("JBREM-877: New Socket Connection is being Created for Every" - http://www.jboss.com/index.html?module=bb&op=viewtopic&t=126382) has brought up an interesting issue concerning the use of Remoting in EJB3.

      In AS 4.0.4/4.0.5, when EJB3 was using Remoting 1.4.x, each call to org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke() created a new org.jboss.remoting.Client, and when it returned, it left the underlying client invoker intact in the org.jboss.remoting.InvokerRegistry. The next call found the existing client invoker and reused its socket pool. (Simplifying somewhat, but basically true.)

      Now, in AS 4.2.x/5.0.0, each call to InvokeRemoteInterceptor.invoke() calls Client.disconnect() before returning, which, if the reference count of the underlying client invoker goes to zero, leads to the destruction of the client invoker, along with its socket pool. So, unless there are multiple proxies in the same JVM running simultaneously and keeping the client invoker's reference count above zero, each invocation will lead to the creation of a new socket. The guy that posted JBREM-877 says that performance is 25% of what it was in AS 4.0.4.

      One solution would be to add a feature which allows a delay in the destruction of the client invoker. E.g., the invoker locator

      socket://${jboss.bind.address}:3873/?invokerdestructiondelay=5000
      


      would introduce a 5 second delay before the client invoker gets destroyed, increasing the probability that it would be reused. The default behavior would be to destroy the client invoker immediately.

      Any thoughts from the EJB3 team?

      -Ron

        • 1. Re: Remoting issue
          Ron Sigal Master

          I went ahead and added the delayed destruction feature to the Remoting 2.4 branch. In a pure Remoting in-jvm test, I got a speed up of about 2 by using delayed client invoker destruction.

          For more information, see the thread http://www.jboss.com/index.html?module=bb&op=viewtopic&t=126382&postdays=0&postorder=asc&start=10.

          • 2. Re: Remoting issue
            Ron Sigal Master

             

            "carlos" wrote:

            I think Bill would like a patch for EAP 4.2 as well.


            Bill as in bill@jboss.org, the original Bill? Or the other Patriot fan Bill?

            In any case, since this is a Remoting feature, it might be good if someone from EJB3 created a JBPAPP issue asking for it.

            • 3. Re: Remoting issue
              Andrew Rubinger Master

               

              "ron.sigal@jboss.com" wrote:
              Bill as in ... the original Bill? Or the other Patriot fan Bill?


              The original Bill DeCoste. ;)

              S,
              ALR

              • 4. Re: Remoting issue
                William DeCoste Apprentice

                Oh god. I'm good. I have the remoting version that I need.

                • 5. Re: Remoting issue
                  Hamdur Rahman Newbie

                  I am still getting the exception even after adding invokerDestructionDelay.
                  The log has changed and we can see the IP as well as port number(This is changing) and also the WorkerThread#_ is changing

                  11:14:41,353 ERROR [ServerThread] WorkerThread#3[150.158.75.156:1406] exception occurred during first invocation
                  java.net.SocketException: Connection reset by peer: socket write error
                  at java.net.SocketOutputStream.socketWrite0(Native Method)
                  at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
                  at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
                  at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
                  at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
                  at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1784)
                  at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:691)
                  at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(JavaSerializationManager.java:121)
                  at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(JavaSerializationManager.java:95)
                  at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
                  at org.jboss.remoting.transport.socket.ServerThread.versionedWrite(ServerThread.java:998)
                  at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:781)
                  at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:695)
                  at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:522)
                  at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:230)

                  • 6. Re: Remoting issue
                    Ron Sigal Master

                    This issue has come up again recently, and I've ported the "invoker destruction delay" feature to the Remoting 2.2 branch so that it will appear in release 2.2.3.SP1. I've been informally suggesting 5 seconds as a reasonable default delay, but I'm wondering if anyone from the EJB3 team has any better intuition.

                    Thanks,
                    Ron

                    • 7. Re: Remoting issue
                      jaikiran pai Master

                      Hi Ron,

                      A few questions

                      1) Is the default destruction delay being maintained programatically within Remoting? I don't see this delay being added within the EJB3 remoting configurations

                      2) From what i understand in those threads, the issue was a socket being created on every method invocation on a remote EJB3 proxy. The destruction delay tries to fix this by delaying the destruction of the socket (and the subsequent creation of a new one). Did i understand this right?

                      I've been informally suggesting 5 seconds as a reasonable default delay, but I'm wondering if anyone from the EJB3 team has any better intuition.


                      Assuming, i understood the issue correctly, i guess the ideal timeout really depends on how the end users use the EJB3 proxies. Some users might have this:

                      someClientMethod()
                      {
                      // lookup remote bean
                      Bean bean = ctx.lookup(...);
                      // invoke method
                      bean.doSomething(); // creates a socket
                      
                      // do something that takes a really long time
                      waitForUserInputOrSomeOtherTimeConsumingOperation();
                      
                      // use the earlier proxy again
                      bean.doSomeOtherThing(); // creates a socket (again)
                      }
                      
                      
                      


                      Although if the proxy is being used for multiple method invocations within that default timeout then the performance improvement is going to be noticeable.

                      3) What is the (performance) impact of specifying a very long destruction delay? Does it degrade the performance or introduce any other issues?