Remote call performance - EJB vs. RMI

[mentiro]

My application uses a remoting proxy for remoting all server-side business objects. Thus, it is very easy for us to swap out which remoting technology we use. In the past we have been EJB-based (all remote invocations were done via a Stateless Session EJB), but have recently switched to RMI for performance reasons. The difference in performance between RMI and EJB invocations is astounding, and I feel sure that there is something we could have done to make the EJB calls more performant.

I have been using a profiler to run numerous tests and observe how long remote invocations take. On a particular remote method call:

RMI
Using RMI, each remote method call took roughly 1.5-2ms. If I invoked the method 200 times, it would take roughly 300-350ms.

EJB
Our framework cached the EJB Home interface, but for each method invocation, it 1) invoked create on the EJB Home interface 2) invoked the method on the EJB object 3) called remove on the EJB. Is this correct or should we also cache the EJB object? If we cache the EJB object, is there any effect on instance pooling, clustering, etc? (we are not currently using clustering, but may in the future)

Calling the above 3 methods looks like it results in 3 separate remote calls to the application server. Obviously if we cached the EJB object, it would improve performance tremendously.

For the three remote invocations:
create() - The create method took an average of 10 ms per invocation (6ms with PooledInvoker). The majority of this time was consistently spent in ClientContainer.readExternal (An average of about 6ms per create was spent in readExternal) Why is this?

method invocation - The method invocation itself took an average of 4 ms (3ms with PooledInvoker). This is still almost 3 times the amount of time a plain RMI call took, but may be normal considering the added complexity of an EJB invocation.

remove() - The remove method took an average of around 3 ms per call (1.5ms with PooledInvoker). This is still double what the entire invocation took using RMI.

Summary
Since we primarily only used EJB as a remoting infrastructure, we have now switched to plain RMI. However, I would like to understand if we were doing anything wrong with EJBs or if there is additional tuning we could have done to improve EJB performance.

Summary of my questions:

1. Should we call create and remove on each invocation, or should we cache the stateless session EJB object in our remoting infrastructure? What effect does this have on server-side instance pooling, clustering, etc.?
   
2. Why does the create() method take so long? Is it normal for readExternal() to take up so much time?
   
3. Is there additional tuning we can do to speed up EJB performance?
   
4. Another problem we saw was that every now and then a create() call would take 2+ seconds. From the profiler, it appeared that this was caused by the MarshalledInputStream.resolveProxyClass() method called when the MarshalledObject.get() method is called by the ClientContainer.readExternal() method. Has anyone seen this problem or does anyone have any ideas why this could happen?
   
   
   Environment
   JBoss 4.0.2
   JDK 1.5.0_04
   JProfiler 4
   All testing was done on a local LAN - obviously the performance would be different over a slower network.
   JRMPInvoker was used, except where it was noted that PooledInvoker.
   
   Thanks!
   Jeremy Haile

[mentiro]

Ok - well I think my post almost answered itself.

I think our basic problem was calling create() and remove() on every EJB method invocation. If we had cached the EJB object stub, each remote call would have taken about 4ms (3ms with pooled invoker) which is pretty reasonable.

So, my remaining questions are:

1. 
   
2. Is caching the EJB stub the correct way to handle things? Is this cross-app server compatible? Any effect on clustering?
   
3. Does anyone have any idea why the EJB create() method on the home interface would occassionally take 2+ seconds to execute.
   

[triathlon98]

Does anyone have any idea why the EJB create() method on the home interface would occassionally take 2+ seconds to execute.

This may have something to do with reverse dns resolution (in fact, this can cause much higher delays, and can be very dependent from one system to another, we have seen delays of up to 20s).
For this reason it is best to always make sure the destination system resolves to a hostname (even when accessing using the ip address).

Joachim