2 Replies Latest reply on Jun 7, 2004 5:54 PM by genman

    JBoss 3.2.3 memory leak in application, no idea what to do

      Hello community,

      we've developed a J2EE application using MySQL 4.0.13 InnoDB, JBoss 3.2.3 and having only stateless session beans, entity beans. Our server is running under linux and with sun's j2sdk build 1.4.2_04-b05.

      Our problem is that the application eats more and more memory until no more memory is available. We do _not_ redeploy at a regular interval (that seemed to be a known bug). After about 5-6 hours the server is down (2GB RAM) and throws OutOfMemoryErrors.

      What did I do?
      1. Search JBoss forums, found "MaxPermSize advice".
      That did not solve our problem. I analyzed the virtual machine with "vmstat" and the size of the perm generation was at his maximum 24MB (default MaxPermSize is 64MB). All I could notice was that the old generation gets bigger and bigger...
      2. Use a profiler
      I've downloaded "jprobe memorydebugger" and installed it on a test computer. I did a stress test on our application and these are the results (in short):

       Package Class Count Count Change Cumulative Count Memory Memory Change Cumulative Memory
       ------- ----- ----- ------------ ---------------- ------ ------------- -----------------
       Total 434.976 +123.597 27.089.141 40.039.352 +14.055.336 1.822.799.136
       char[ ] 54.709 (12,6%) +9.863 6.307.745 (0,9%) 7.854.400 (19,6%) +1.165.520 745.269.448 (1,1%)
      java.lang String 57.047 (13,1%) +8.710 4.686.103 (1,2%) 1.369.128 (3,4%) +209.040 112.466.472 (1,2%)
      java.lang Object 9.159 (2,1%) +8.685 41.827 (21,9%) 73.272 (0,2%) +69.480 334.616 (21,9%)
       byte[ ] 14.183 (3,3%) +8.658 2.272.752 (0,6%) 10.903.816 (27,2%) +9.158.672 520.754.784 (2,1%)
       int[ ] 16.695 (3,8%) +6.800 549.896 (3,0%) 8.897.200 (22,2%) +784.800 27.699.176 (32,1%)
      java.util TreeMap$Entry 6.336 (1,5%) +6.182 25.117 (25,2%) 202.752 (0,5%) +197.824 803.744 (25,2%)
      java.lang ThreadLocal$ThreadLocalMap$Entry 4.835 (1,1%) +4.770 5.008 (96,5%) 154.720 (0,4%) +152.640 160.256 (96,5%)
      java.util HashMap$Entry 28.465 (6,5%) +4.770 1.035.635 (2,7%) 683.160 (1,7%) +114.480 24.855.240 (2,7%)
      org.jboss.ejb EntityEnterpriseContext 3.641 (0,8%) +3.504 19.570 (18,6%) 262.152 (0,7%) +252.288 1.409.040 (18,6%)
      org.jboss.ejb.plugins.cmp.bridge EntityBridgeInvocationHandler 3.641 (0,8%) +3.504 19.626 (18,6%) 116.512 (0,3%) +112.128 628.032 (18,6%)
      org.jboss.ejb.plugins.lock NonReentrantLock 3.641 (0,8%) +3.504 19.570 (18,6%) 87.384 (0,2%) +84.096 469.680 (18,6%)
      org.jboss.ejb EntityEnterpriseContext$EntityContextImpl 3.641 (0,8%) +3.504 19.570 (18,6%) 87.384 (0,2%) +84.096 469.680 (18,6%)
      java.util HashMap 10.777 (2,5%) +3.237 1.214.225 (0,9%) 431.080 (1,1%) +129.480 48.569.000 (0,9%)
      org.jboss.ejb.plugins.cmp.jdbc.bridge JDBCCMP2xFieldBridge$FieldState 3.601 (0,8%) +2.467 185.671 (1,9%) 144.040 (0,4%) +98.680 7.426.840 (1,9%)
      java.lang.reflect Constructor 3.164 (0,7%) +2.118 8.649 (36,6%) 151.872 (0,4%) +101.664 415.152 (36,6%)
      java.util Vector 4.170 (1,0%) +2.037 11.769 (35,4%) 100.080 (0,2%) +48.888 282.456 (35,4%)
      java.lang.ref SoftReference 4.202 (1,0%) +1.732 50.888 (8,3%) 134.464 (0,3%) +55.424 1.628.416 (8,3%)
      java.util ArrayList 3.800 (0,9%) +1.597 658.362 (0,6%) 91.200 (0,2%) +38.328 15.800.688 (0,6%)
       short[ ] 7.139 (1,6%) +1.469 389.097 (1,8%) 398.920 (1,0%) +40.392 30.854.616 (1,3%)
      com.sun.net.ssl.internal.ssl SunJSSE_aq 1.410 (0,3%) +1.404 1.716 (82,2%) 45.120 (0,1%) +44.928 54.912 (82,2%)
      java.lang Class 5.677 (1,3%) +1.387 6.091 (93,2%) 317.912 (0,8%) +77.672 341.096 (93,2%)
      
      java.util HashSet 3.366 (0,8%) +1.382 215.656 (1,6%) 53.856 (0,1%) +22.112 3.450.496 (1,6%)
      java.lang.reflect Method 19.199 (4,4%) +1.335 100.823 (19,0%) 1.075.144 (2,7%) +74.760 5.646.088 (19,0%)
      
      


      My question is:
      could there be a memory leak in the jboss classes?
      Why are after my test case +3504 EntityEnterpriseContext objects in memory?
      Who can help us out with this problem? What additional info do you need?

        • 1. Re: JBoss 3.2.3 memory leak in application, no idea what to

          Hi community,

          as you can see in my first post there are very much byte and char arrays.
          I went into the detail and determined that most of the objects were instantiated in JSSE classes.
          Indeed we use encrypted communication between our web start swing clients and our jboss server:

          using ssl:

          <invoker-proxy-bindings>
           <!-- Aufruf der Stateless SessionBeans per SSL -->
           <invoker-proxy-binding>
           <name>stateless-rmi-invoker</name>
           <invoker-mbean>jboss:service=invoker,type=jrmp,socketType=SSL</invoker-mbean>
           <proxy-factory>org.jboss.proxy.ejb.ProxyFactory</proxy-factory>
           <proxy-factory-config>
           <client-interceptors>
           <home>
           <interceptor>org.jboss.proxy.ejb.HomeInterceptor</interceptor>
           <interceptor>org.jboss.proxy.SecurityInterceptor</interceptor>
           <interceptor>org.jboss.proxy.TransactionInterceptor</interceptor>
           <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
           </home>
           <bean>
           <interceptor>org.jboss.proxy.ejb.StatelessSessionInterceptor</interceptor>
           <interceptor>org.jboss.proxy.SecurityInterceptor</interceptor>
           <interceptor>org.jboss.proxy.TransactionInterceptor</interceptor>
           <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
           </bean>
           </client-interceptors>
           </proxy-factory-config>
           </invoker-proxy-binding>
           <!-- Aufruf der Stateful SessionBeans per SSL -->
           <invoker-proxy-binding>
           <name>stateful-rmi-invoker</name>
           <invoker-mbean>jboss:service=invoker,type=jrmp,socketType=SSL</invoker-mbean>
           <proxy-factory>org.jboss.proxy.ejb.ProxyFactory</proxy-factory>
           <proxy-factory-config>
           <client-interceptors>
           <home>
           <interceptor>org.jboss.proxy.ejb.HomeInterceptor</interceptor>
           <interceptor>org.jboss.proxy.SecurityInterceptor</interceptor>
           <interceptor>org.jboss.proxy.TransactionInterceptor</interceptor>
           <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
           </home>
           <bean>
           <interceptor>org.jboss.proxy.ejb.StatefulSessionInterceptor</interceptor>
           <interceptor>org.jboss.proxy.SecurityInterceptor</interceptor>
           <interceptor>org.jboss.proxy.TransactionInterceptor</interceptor>
           <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
           </bean>
           </client-interceptors>
           </proxy-factory-config>
           </invoker-proxy-binding>
           </invoker-proxy-bindings>
          


          Allocation points of objects that weren't garbage collected (app was stress tested about 1/2 hour):
          (number of allocations) (allocation place)
          5367 No trace available null
          2340 SunJSSE_au.<init>(byte[])
          1404 SunJSSE_at.<init>(SunJSSE_ar)
          1018 Unsafe.defineClass(String, byte[], int, int, ClassLoader, ProtectionDomain) Unsafe.java
           936 Object.clone() Object.java
           539 JDBCEntityBridge$EntityState.<init>(JDBCEntityBridge) JDBCEntityBridge.java: 1104
           468 SunJSSE_ax.b(byte[])
           270 ClassLoader.defineClass0(String, byte[], int, int, ProtectionDomain) ClassLoader.java
           238 SocketOutputStream.<init>(PlainSocketImpl) SocketOutputStream.java: 31
           235 AppInputStream.<init>(SSLSocketImpl)
           235 InputRecord.<init>()
           235 AppOutputStream.<init>(SSLSocketImpl)
           235 ByteArrayOutputStream.<init>(int) ByteArrayOutputStream.java: 59
           234 MD5.init() MD5.java: 206
           234 MD5.init() MD5.java: 207
           28 SunJSSE_a2.<init>(SecureRandom)
          ...
          


          After deactivating the encryption the big memory leak seems to be away: the jboss runs and runs and runs.

          Does anyone know if this is a JSSE leak in the classes of sun? Or is it a jboss leak?

          We cannot run our application without encryption so the problem isn't really solved yet.



          • 2. Re: JBoss 3.2.3 memory leak in application, no idea what to
            genman

            You should be able to get a stack trace of when the SSL objects are being constructed. Could you post those?

            I don't see that MBean ObjectName in the JBoss 3.2.3 distribution, so I assume it's customer.

            There is something close (in ./server/default/conf/jboss-service.xml)

             <mbean code="org.jboss.invocation.jrmp.server.JRMPInvoker"
             name="jboss:service=invoker,type=jrmp">
             <attribute name="RMIObjectPort">4444</attribute>
             <attribute name="ServerAddress">${jboss.bind.address}</attribute>
             <!--
             <attribute name="RMIClientSocketFactory">custom</attribute>
             <attribute name="RMIServerSocketFactory">custom</attribute>
             <attribute name="SecurityDomain">ssl-domain-name</attribute>
             -->
            
             <depends>jboss:service=TransactionManager</depends>
             </mbean>


            What were the values used for your install?

            If you can construct a test case, somebody at JBoss would likely fix the problem.