5 Replies Latest reply on Nov 13, 2010 6:06 AM by Wolf-Dieter Fink

    EJB3 / $Proxy110465 / jboss RemotingClassLoader

    Nathan Ciliberto Newbie

      We are have a problem with memory when we convert our ejbs from EJB 2.1 to EJB 3.0 in jboss.

       

      With EJB 2.1 everything is fine.

       

      However with EJB 3.0 we eventually run out of memory.

       

      The PermGen is set to 512 MB and the -Xmx is set to 2048 GB.

       

      We are running stress tests against our site and are running into out of memory errors after a few hours.

       

      When I use jmap to look at the heap I see may $Proxy#### which have been loaded by org.jboss.remoting.loading.RemotingClassLoader@####.

       

      There are many of these RemotingClassLoader instances and (as can bee seen in the title) many 1000's of the $Proxy classes which are proxying our ejb's.

      From the heap dump the number starts $Proxy108810 and ends $Proxy110465, so over 10000 of these are hanging around.

      There are 6253 of the org.jboss.remoting.loading.RemotingClassLoader instances in the heap as well... why?

       

      Is there any way to determine why so many of these (proxy's and RemotingClassLoaders) are being created?

       

      What creates the RemotingClassLoaders and why would they all think they need to create new $Proxy### classes for ejb's that already have $Proxy### classes?

       

      Do the RemotingClassLoaders "timeout" or something and another created in it's place?

      If so... why don't they get garbage collected?

        • 1. Re: EJB3 / $Proxy110465 / jboss RemotingClassLoader
          Nathan Ciliberto Newbie

          However, in addition to the above it says it is running out of memory, but the heap and permgen look fine to me (still 1000's of proxys though):

           

          This is what jmap shows:

           

          bash-2.05# jmap -heap 14912
          Attaching to process ID 14912, please wait...
          Debugger attached successfully.
          Server compiler detected.
          JVM version is 14.0-b16

           

          using thread-local object allocation.
          Parallel GC with 2 thread(s)

           

          Heap Configuration:
             MinHeapFreeRatio = 40
             MaxHeapFreeRatio = 70
             MaxHeapSize      = 1073741824 (1024.0MB)
             NewSize          = 2228224 (2.125MB)
             MaxNewSize       = 4294901760 (4095.9375MB)
             OldSize          = 4194304 (4.0MB)
             NewRatio         = 2
             SurvivorRatio    = 8
             PermSize         = 16777216 (16.0MB)
             MaxPermSize      = 268435456 (256.0MB)

           

          Heap Usage:
          PS Young Generation
          Eden Space:
             capacity = 316014592 (301.375MB)
             used     = 7066480 (6.7391204833984375MB)
             free     = 308948112 (294.63587951660156MB)
             2.2361245900948776% used
          From Space:
             capacity = 8323072 (7.9375MB)
             used     = 0 (0.0MB)
             free     = 8323072 (7.9375MB)
             0.0% used
          To Space:
             capacity = 23920640 (22.8125MB)
             used     = 0 (0.0MB)
             free     = 23920640 (22.8125MB)
             0.0% used
          PS Old Generation
             capacity = 239075328 (228.0MB)
             used     = 91060992 (86.842529296875MB)
             free     = 148014336 (141.157470703125MB)
             38.08882863898026% used
          PS Perm Generation
             capacity = 109051904 (104.0MB)
             used     = 78839208 (75.18692779541016MB)
             free     = 30212696 (28.813072204589844MB)
             72.29512288020207% used

           

           

          All of the OutOfMemoryErrors occur in java.lang.Class.getDeclaredFields0(Native Method) as follows:

           

          java.lang.OutOfMemoryError
              at java.lang.Class.getDeclaredFields0(Native Method)
              at java.lang.Class.privateGetDeclaredFields(Class.java:2291)
              at java.lang.Class.getDeclaredField(Class.java:1880)
              at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1610)
              at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:52)
              at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:425)
              at java.security.AccessController.doPrivileged(Native Method)
              at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
              at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
              at java.io.ObjectStreamClass.initProxy(ObjectStreamClass.java:507)
              at java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1539)
              at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1493)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
              at org.jboss.aop.joinpoint.InvocationResponse.readExternal(InvocationResponse.java:119)
              at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1792)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1751)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
              at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
              at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
              at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.receiveObjectVersion2_2(JavaSerializationManager.java:238)
              at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.receiveObject(JavaSerializationManager.java:138)
              at org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.read(SerializableUnMarshaller.java:123)
              at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedRead(MicroSocketClientInvoker.java:1299)
              at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:907)
              at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167)
              at org.jboss.remoting.Client.invoke(Client.java:1927)
              at org.jboss.remoting.Client.invoke(Client.java:770)
              at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
              at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
              at org.jboss.ejb3.proxy.impl.remoting.IsLocalProxyFactoryInterceptor.invoke(IsLocalProxyFactoryInterceptor.java:72)
              at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
              at org.jboss.aspects.remoting.PojiProxy.invoke(PojiProxy.java:62)
              at $Proxy2.createProxyBusiness(Unknown Source)
              at org.jboss.ejb3.proxy.impl.objectfactory.session.SessionProxyObjectFactory.createProxy(SessionProxyObjectFactory.java:129)
              at org.jboss.ejb3.proxy.impl.objectfactory.session.stateless.StatelessSessionProxyObjectFactory.getProxy(StatelessSessionProxyObjectFactory.java:79)
              at org.jboss.ejb3.proxy.impl.objectfactory.ProxyObjectFactory.getObjectInstance(ProxyObjectFactory.java:158)
              at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
              at org.jnp.interfaces.NamingContext.getObjectInstance(NamingContext.java:1479)
              at org.jnp.interfaces.NamingContext.getObjectInstanceWrapFailure(NamingContext.java:1496)
              at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:822)
              at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
              at javax.naming.InitialContext.lookup(InitialContext.java:392)

              ... our other business methods below ...

           

          Is it possible the native code is getting a different error, but just interpreting it as an OutOfMemoryError?

          • 2. Re: EJB3 / $Proxy110465 / jboss RemotingClassLoader
            jaikiran pai Master

            Nathan Ciliberto wrote:

             


            From the heap dump the number starts $Proxy108810 and ends $Proxy110465, so over 10000 of these are hanging around.

            There are 6253 of the org.jboss.remoting.loading.RemotingClassLoader instances in the heap as well... why?

             

            Is there any way to determine why so many of these (proxy's and RemotingClassLoaders) are being created?

             


            How are you using those proxies in your application? Can you provide more details about your application? Are you caching those proxies? How do you do the lookup?

            • 3. Re: EJB3 / $Proxy110465 / jboss RemotingClassLoader
              Nathan Ciliberto Newbie

              We are doing the ejb3 lookups as follows:

               

              Hashtable       ht = new Hashtable();
              InitialContext  ic;
              Object          ejbObj;
              MyRemoteEjbInterface ejb;

               

              ht.put( "java.naming.provider.url"      , "jnp://serverA:1099,jnp://serverB:1099,jnp://serverC:1099" );
              ht.put( "java.naming.factory.url.pkgs"  , "org.jboss.naming:org.jnp.interfaces"     );
              ht.put( "java.naming.factory.initial"   , "org.jnp.interfaces.NamingContextFactory" );
              ht.put( "jnp.disableDiscovery"          , "true" );
              ht.put( "jnp.sotimeout"                 , "100"  );
              ht.put( "jnp.timeout"                   , "100"  );

               

              ic     = new InitialContext( ht );
              ejbObj = ic.lookup( jndiName );
              ejb    = (MyRemoteEjbInterface) ejbObj;

               

              ... call methods on ejb ...

               

              For ejb2 references we call PortableRemoteObject.narrow(...) when appropriate, then call the create() method on the reference.

               

              We are not caching the ejb object returned... is it safe to do so? What about if the jboss instance goes down or sharing these between web sessions or other threads?

               

              -Nathan

              1 of 1 people found this helpful
              • 4. Re: EJB3 / $Proxy110465 / jboss RemotingClassLoader
                Nathan Ciliberto Newbie

                We have an update on this issue.

                 

                We have had some free time and decided to take a look at this again.

                 

                We created a simple test class that just loops forever calling "ic.lookup( jndiName )" until an error occurs.

                It usually errors after ~7000 calls on certain EJB's.

                 

                So far it appears to be due to a bug in jdk 1.6.0_20.

                 

                We have tried it with jdk 1.6.0_22 and have not been able to reproduce it.

                 

                We have not tested this with jdk 1.6.0_21.

                Also, we are not sure what was specifically fixed since *_20 that may have fixed this.

                1 of 1 people found this helpful
                • 5. Re: EJB3 / $Proxy110465 / jboss RemotingClassLoader
                  Wolf-Dieter Fink Master
                  We are not caching the ejb object returned... is it safe to do so? What about if the jboss instance goes down or sharing these between web sessions or other threads?

                  The references (in your case MyRemoteEjbInterface ejb) can be 'cached' and reused if it is a reference to a SLSB!

                  In case of crash/shutdown of one node of a cluster (if you have one) the proxy is updated by next access with the actual clusterview.

                  In case of all nodes (or the only one) are down I suppose it depends to the configuration, for EJB2 there is a special client-side-interceptor whithin the chain to handle this 'reconnect' (AFAIK it is not the standard configuration)

                  For EJB3 I'm not sure about this I've it not tested now.