EJB3 / $Proxy110465 / jboss RemotingClassLoader

[snacker]

We are have a problem with memory when we convert our ejbs from EJB 2.1 to EJB 3.0 in jboss.

With EJB 2.1 everything is fine.

However with EJB 3.0 we eventually run out of memory.

The PermGen is set to 512 MB and the -Xmx is set to 2048 GB.

We are running stress tests against our site and are running into out of memory errors after a few hours.

When I use jmap to look at the heap I see may $Proxy#### which have been loaded by org.jboss.remoting.loading.RemotingClassLoader@####.

There are many of these RemotingClassLoader instances and (as can bee seen in the title) many 1000's of the $Proxy classes which are proxying our ejb's.

From the heap dump the number starts $Proxy108810 and ends $Proxy110465, so over 10000 of these are hanging around.

There are 6253 of the org.jboss.remoting.loading.RemotingClassLoader instances in the heap as well... why?

Is there any way to determine why so many of these (proxy's and RemotingClassLoaders) are being created?

What creates the RemotingClassLoaders and why would they all think they need to create new $Proxy### classes for ejb's that already have $Proxy### classes?

Do the RemotingClassLoaders "timeout" or something and another created in it's place?

If so... why don't they get garbage collected?

[snacker]

However, in addition to the above it says it is running out of memory, but the heap and permgen look fine to me (still 1000's of proxys though):

This is what jmap shows:

bash-2.05# jmap -heap 14912
Attaching to process ID 14912, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 14.0-b16

using thread-local object allocation.
Parallel GC with 2 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 1073741824 (1024.0MB)
   NewSize          = 2228224 (2.125MB)
   MaxNewSize       = 4294901760 (4095.9375MB)
   OldSize          = 4194304 (4.0MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 16777216 (16.0MB)
   MaxPermSize      = 268435456 (256.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 316014592 (301.375MB)
   used     = 7066480 (6.7391204833984375MB)
   free     = 308948112 (294.63587951660156MB)
   2.2361245900948776% used
From Space:
   capacity = 8323072 (7.9375MB)
   used     = 0 (0.0MB)
   free     = 8323072 (7.9375MB)
   0.0% used
To Space:
   capacity = 23920640 (22.8125MB)
   used     = 0 (0.0MB)
   free     = 23920640 (22.8125MB)
   0.0% used
PS Old Generation
   capacity = 239075328 (228.0MB)
   used     = 91060992 (86.842529296875MB)
   free     = 148014336 (141.157470703125MB)
   38.08882863898026% used
PS Perm Generation
   capacity = 109051904 (104.0MB)
   used     = 78839208 (75.18692779541016MB)
   free     = 30212696 (28.813072204589844MB)
   72.29512288020207% used

All of the OutOfMemoryErrors occur in java.lang.Class.getDeclaredFields0(Native Method) as follows:

java.lang.OutOfMemoryError
    at java.lang.Class.getDeclaredFields0(Native Method)
    at java.lang.Class.privateGetDeclaredFields(Class.java:2291)
    at java.lang.Class.getDeclaredField(Class.java:1880)
    at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1610)
    at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:52)
    at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:425)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
    at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
    at java.io.ObjectStreamClass.initProxy(ObjectStreamClass.java:507)
    at java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1539)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1493)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
    at org.jboss.aop.joinpoint.InvocationResponse.readExternal(InvocationResponse.java:119)
    at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1792)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1751)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
    at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.receiveObjectVersion2_2(JavaSerializationManager.java:238)
    at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.receiveObject(JavaSerializationManager.java:138)
    at org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.read(SerializableUnMarshaller.java:123)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedRead(MicroSocketClientInvoker.java:1299)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:907)
    at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167)
    at org.jboss.remoting.Client.invoke(Client.java:1927)
    at org.jboss.remoting.Client.invoke(Client.java:770)
    at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.ejb3.proxy.impl.remoting.IsLocalProxyFactoryInterceptor.invoke(IsLocalProxyFactoryInterceptor.java:72)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.aspects.remoting.PojiProxy.invoke(PojiProxy.java:62)
    at $Proxy2.createProxyBusiness(Unknown Source)
    at org.jboss.ejb3.proxy.impl.objectfactory.session.SessionProxyObjectFactory.createProxy(SessionProxyObjectFactory.java:129)
    at org.jboss.ejb3.proxy.impl.objectfactory.session.stateless.StatelessSessionProxyObjectFactory.getProxy(StatelessSessionProxyObjectFactory.java:79)
    at org.jboss.ejb3.proxy.impl.objectfactory.ProxyObjectFactory.getObjectInstance(ProxyObjectFactory.java:158)
    at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
    at org.jnp.interfaces.NamingContext.getObjectInstance(NamingContext.java:1479)
    at org.jnp.interfaces.NamingContext.getObjectInstanceWrapFailure(NamingContext.java:1496)
    at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:822)
    at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
    at javax.naming.InitialContext.lookup(InitialContext.java:392)

    ... our other business methods below ...

Is it possible the native code is getting a different error, but just interpreting it as an OutOfMemoryError?

[jaikiran]

Nathan Ciliberto wrote:

From the heap dump the number starts $Proxy108810 and ends $Proxy110465, so over 10000 of these are hanging around.

There are 6253 of the org.jboss.remoting.loading.RemotingClassLoader instances in the heap as well... why?

Is there any way to determine why so many of these (proxy's and RemotingClassLoaders) are being created?

How are you using those proxies in your application? Can you provide more details about your application? Are you caching those proxies? How do you do the lookup?

[snacker]

We are doing the ejb3 lookups as follows:

Hashtable       ht = new Hashtable();
InitialContext  ic;
Object          ejbObj;
MyRemoteEjbInterface ejb;

ht.put( "java.naming.provider.url"      , "jnp://serverA:1099,jnp://serverB:1099,jnp://serverC:1099" );
ht.put( "java.naming.factory.url.pkgs"  , "org.jboss.naming:org.jnp.interfaces"     );
ht.put( "java.naming.factory.initial"   , "org.jnp.interfaces.NamingContextFactory" );
ht.put( "jnp.disableDiscovery"          , "true" );
ht.put( "jnp.sotimeout"                 , "100"  );
ht.put( "jnp.timeout"                   , "100"  );

ic     = new InitialContext( ht );
ejbObj = ic.lookup( jndiName );
ejb    = (MyRemoteEjbInterface) ejbObj;

... call methods on ejb ...

For ejb2 references we call PortableRemoteObject.narrow(...) when appropriate, then call the create() method on the reference.

We are not caching the ejb object returned... is it safe to do so? What about if the jboss instance goes down or sharing these between web sessions or other threads?

-Nathan

[snacker]

We have an update on this issue.

We have had some free time and decided to take a look at this again.

We created a simple test class that just loops forever calling "ic.lookup( jndiName )" until an error occurs.

It usually errors after ~7000 calls on certain EJB's.

So far it appears to be due to a bug in jdk 1.6.0_20.

We have tried it with jdk 1.6.0_22 and have not been able to reproduce it.

We have not tested this with jdk 1.6.0_21.

Also, we are not sure what was specifically fixed since *_20 that may have fixed this.

[wdfink]

We are not caching the ejb object returned... is it safe to do so? What about if the jboss instance goes down or sharing these between web sessions or other threads?

The references (in your case MyRemoteEjbInterface ejb) can be 'cached' and reused if it is a reference to a SLSB!

In case of crash/shutdown of one node of a cluster (if you have one) the proxy is updated by next access with the actual clusterview.

In case of all nodes (or the only one) are down I suppose it depends to the configuration, for EJB2 there is a special client-side-interceptor whithin the chain to handle this 'reconnect' (AFAIK it is not the standard configuration)

For EJB3 I'm not sure about this I've it not tested now.