4 Replies Latest reply on Feb 6, 2002 4:58 AM by adrian.brock

    Memory leak?

    neilmcc

      Hi,

      I've been wrestling with an apparent memory leak problem for the past few days. I've got jBoss 2.4.4 running on RH6.2 with 512Mb (physical) RAM on a AMD 300Mhz box. The application I'm deploying has about 15 beans - a mix of BMP entities, stateful session beans and a single MDB listening to JMS (JbossMQ). The beans are accessed by a set of 25 standalone Tomcat 3.2.3 JSP's binding to the beans using the Orion Server ejb tags.

      It seems that repeated client/JSP access causes memory usage to steadily grow and grow to the point where ultimately MQ reports EOF stream exceptions and jBoss and Tomcat throw an OutOfMemoryException.

      I've seen posts elsewhere suggesting that the JVM may be the main problem - I've tried Sun JDK1.3.0_02, 1.3.1_02 (with/without Hostpot in classic/server mode) plus IBM's 1.3 JVM. Regardless of JVM, the mem leak still appears (though the IBM JVM seems to use even more memory than the Sun variant). With the Sun JVM, JBoss seems to create a new thread for each bean accessed. Switching to green threads hides this issue (at least when looking at ps output) but mem issues still seem to be there.

      I've been through the JSP/servlets ensuring there aren't any large objects hanging around in HttpSessions etc. Have also reviewed EJB code to ensure all database connections are being correctly returned to the pool etc. I've also adapted the cache sizes (downwards) in standardjboss.xml in the hope that the beans get persisted quicker - but no luck there.

      It's possible the ejb tags are holding onto Jboss beans longer than they need to - this is the main tag code :

      protected EJBHome getEjbHome() throws JspException {
      try {
      InitialContext context = getContext(ejbHost);
      // Get the home interface class using the proper (the page's) ClassLoader.
      Class homeClass = Class.forName(remoteHome, true, pageContext.getPage().getClass().getClassLoader());
      // Lookup the home object
      Object homeObject = context.lookup(beanName);
      // Narrow it
      EJBHome home = (EJBHome)javax.rmi.PortableRemoteObject.narrow(homeObject, homeClass);
      return home;
      } catch(NamingException e) {
      throw new JspException("Error looking up home at " + this.beanName + " for EJB host " + ejbHost + " - " + e.getMessage());
      }
      catch(ClassNotFoundException e) {
      throw new JspException("Home class not found: " + this.remoteHome);
      }
      }

      public int doStartTag() throws JspException {
      // Put the variable into the PageContext
      EJBHome home = getEjbHome();
      pageContext.setAttribute(this.getId(), home);
      return SKIP_BODY;
      }

      Has anyone any thoughts how I might go about debugging/resolving this? It's a big issue for us. Has anyone else manage to successfully deploy JBoss on RH6.2 with any of the JVM's above?

      Any suggestions really welcome!

      Thanks.

        • 1. Re: Memory leak?

          You can use hprof to debug this.
          Add -Xrunhprof:heap=sites to the java command in run_with_catalina.sh

          If there's a problem it should show up as a large
          number in live objs at the bottom of the report.

          The report java.prof.txt is printed when the VM exits.
          Don't wait for the OutOfMemoryException otherwise it
          might not work. Just let it run for a while then
          stop JBoss.

          I posted a 3.0 patch for Garbage Monitor service
          on sourceforge. You could adapt that
          to 2.4 and add it to jboss.jcml (or just put the code in your app).
          This will help you test that the home objects added to
          the page context eventually get garbage collected.

          Regards,
          Adrian

          • 2. Re: Memory leak?
            neilmcc

            Thanks Adrian - how do I go about
            I posted a 3.0 patch for Garbage Monitor service
            on sourceforge. You could adapt that
            to 2.4 and add it to jboss.jcml (or just put the code in your app).
            This will help you test that the home objects added to
            the page context eventually get garbage collected.


            ?

            I'm a bit of a CVS virgin...

            • 3. Re: Memory leak?
              neilmcc

              OK - hprof stats for jboss (tomcat is running standalone in the background) - no idea what this lot means! Looks like quite a lot of live objs to me (but then I'm no expert...). Dubious or not?

              (Trace shows 5 mins run)

              SITES BEGIN (ordered by live bytes) Tue Feb 5 22:40:01 2002
              percent live alloc'ed stack class
              rank self accum bytes objs bytes objs trace name
              1 9.14% 9.14% 524292 1 524292 1 10976 [B
              2 3.59% 12.73% 205800 1470 205800 1470 2029 java/lang/Class
              3 2.88% 15.61% 165248 1336 9058888 63390 1464 [C
              4 2.12% 17.73% 121424 1110 522472 4190 3872 [C
              5 1.94% 19.66% 111028 2555 111128 2556 2075 [C
              6 1.70% 21.36% 97720 32 4396624 1470 1641 [B
              7 1.58% 22.95% 90708 337 404416 1446 3873 [C
              8 1.18% 24.12% 67440 856 67440 856 2067 [C
              9 1.15% 25.27% 65808 1828 190008 5278 15455 [C
              10 1.14% 26.41% 65540 1 65540 1 4555 [B
              11 1.00% 27.42% 57636 25 105708 49 1475 [B
              12 0.98% 28.40% 56436 813 166444 2303 9987 [C
              13 0.98% 29.38% 56200 2810 56280 2814 1530 java/util/HashMap$Entry
              14 0.94% 30.32% 54144 344 2505192 16088 1399 [C
              15 0.91% 31.23% 52140 2607 105560 5278 15454 java/lang/String
              16 0.89% 32.12% 51100 2555 51120 2556 2074 java/lang/String
              17 0.86% 32.98% 49168 414 255232 1664 3564 [C
              18 0.86% 33.84% 49164 3 49164 3 20492 [C
              19 0.81% 34.65% 46732 1669 213416 7622 1698 [C
              20 0.80% 35.45% 45924 89 735816 1426 1490 [B
              21 0.71% 36.17% 41000 10 176300 43 15535 [B
              22 0.70% 36.87% 40056 194 1656832 8294 1408 [C
              23 0.66% 37.52% 37664 856 37664 856 2069 java/util/HashMap
              24 0.58% 38.10% 33380 1669 152440 7622 1699 java/lang/String
              25 0.57% 38.68% 32776 2 32776 2 18010 [B
              26 0.57% 39.25% 32776 2 32776 2 20818 [C
              27 0.57% 39.82% 32776 2 65552 4 3509 [C
              28 0.57% 40.39% 32772 1 32772 1 8841 [C
              29 0.57% 40.96% 32460 1327 205216 7744 2422 [C
              30 0.56% 41.51% 31996 1 31996 1 15163 [C
              31 0.53% 42.05% 30672 2556 30684 2557 2076 java/util/jar/Attributes$Name
              32 0.53% 42.58% 30304 824 30416 828 1536 [L;
              33 0.52% 43.09% 29696 30 29696 30 14588 [B
              34 0.48% 43.58% 27656 208 1972868 11223 1407 [C
              35 0.48% 44.05% 27396 761 1374912 38192 1463 [C
              36 0.48% 44.53% 27384 978 54768 1956 16582 [L;
              37 0.43% 44.96% 24852 327 24852 327 3233 java/util/SimpleTimeZone

              • 4. Re: Memory leak?

                That looks pretty normal, but you'll need to run it
                for longer than 5 mins (unless you get OutOfMemory after 5 mins???).

                What you are looking for is a large number in the
                fourth column.
                e.g. look at java/util/HashMap$Entry this reports says
                you have 2810 "live" objects out of 2814 allocated.
                That means 4 have been garbage collected.

                If you have a problem you will start seeing numbers
                like 50000+ live objects.

                Unfortunatley, the report is ordered by number of bytes
                instead of number of objects. You might need to look
                down as far as rank 100. Sometimes the object is small,
                but there are lots of them.

                Regards,
                Adrian