7 Replies Latest reply on Dec 19, 2013 7:20 AM by wdfink

    JBoss EAP 6.2 Memory Leak, Memory Profiler (how to attach)

    paata

      hello team,

       

      We're trying to migrate on jboss eap 6.2

      We run some services on it and our jboss leak a memory.

      We have 1 EJB application on it (JMS, Stateless Session Beans, Hibernate JPA), database is Oracle 12c and oracle timesten database.

      Our platform is CentOS release 6.4 (Final).

      JDK : jdk1.7.0_21 (also we tried oracle jrockit-jdk1.6.0_45-R28.2.7-4.1.0) result is the same.

       

      our jboss process leak a memory and when he approach 99 % of host memory. operating system kills it.

       

      After this problem we decide to attach memory profiler (f.e JProfiler or visualvm) . but we can't.

       

      our startup script :

       

      #!/bin/bash                                                                                                                                                                         
      $(dirname $0)/standalone.sh -c standalone-full.xml -b 192.9.200.9 -Djboss.bind.address.management=192.9.200.9$@>
      

       

      from the visualvm / jconsole :

      connection / remote process : service:jmx:remoting-jmx://192.9.200.9:9999

      username : admin

      password: XXXXXXXX

       

      but connection failed.

       

      We've read most of forum posts bu no success.

       

      how we can detect memory leak ? or how we can attach  memory profiler ???

       

      Thant you in advance.

       

      Regards.

      Paata Lominadze.

        • 1. Re: JBoss EAP 6.2 Memory Leak, Memory Profiler (how to attach)
          wdfink

          You can get a heap dump and analyze it with tools.

           

          Also if you look to the bin/standalone.conf file there are some commented settings at the end which might help.

          • 2. Re: JBoss EAP 6.2 Memory Leak, Memory Profiler (how to attach)
            paata

            hello again,

            thank you for your post.

             

            I made heap dump but it doesn't contains any useful information.

             

            I don't know how to stop memory leak.

            • 3. Re: JBoss EAP 6.2 Memory Leak, Memory Profiler (how to attach)
              wdfink

              Paata Lominadze wrote:

              our jboss process leak a memory and when he approach 99 % of host memory. operating system kills it.

              What you mean by this? Does the JVM grab all the system memory? If yes you might have a wrong JVM configuration as you can set -Xmx -Xms to set the used heap size, consider that you need to have a bit memory for the JVM and other stuff as well.

              How your environment look like? How does the JVM settings look like?

              • 4. Re: Re: JBoss EAP 6.2 Memory Leak, Memory Profiler (how to attach)
                paata

                Wolf-Dieter Fink wrote:

                 

                Paata Lominadze wrote:

                our jboss process leak a memory and when he approach 99 % of host memory. operating system kills it.

                What you mean by this? Does the JVM grab all the system memory? If yes you might have a wrong JVM configuration as you can set -Xmx -Xms to set the used heap size, consider that you need to have a bit memory for the JVM and other stuff as well.

                How your environment look like? How does the JVM settings look like?

                 

                hello

                 

                Yes JVM grab all system memory.of course I have jvm parameters, my standalone.conf :

                 

                JAVA_OPTS="-Xms3g -Xmx3g -d64 -server -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true"                                                                                     
                JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true"
                

                 

                java version : jdk1.7.0_21 or jrockit-jdk1.6.0_45-R28.2.7-4.1.0

                Platform : CentOS release 6.4 (Final) or Solaris 10 1/06 s10s_u1wos_19a SPARC (SunOS pps1 5.10 Generic_118833-24 sun4u sparc SUNW,Netra-T12 Solaris)

                CentOS has 16GB mem and sun solaris 96GB,

                 

                 

                Thank you in advance.

                Regards,

                Paata Lominadze.

                • 5. Re: JBoss EAP 6.2 Memory Leak, Memory Profiler (how to attach)
                  wdfink

                  I'm not sure that I understand it correct.

                   

                  Did you analyze the JVM heap size? I suppose it will have about 3.5G as the limit is. I did not heard about that a JVM take more.

                  JBoss did not have control of memory access, so this should not an issue here.

                  Is it really a single JVM provess?

                  As you use two different JVM versions I suppose this is no JVM bug. On the other hand you use two different OS which mean the problem is not OS releated.

                  So ATM I'm confused

                   

                  Where do you see that the JVM grab all the memory? One possibility is that there are a huge number of threads and the JVM grep memory for the thread stack (which is not limited AFAIR)

                  • 6. Re: Re: JBoss EAP 6.2 Memory Leak, Memory Profiler (how to attach)
                    paata

                    you're right jvm has 3.5gb limit.

                     

                    I can show you something strange :

                     

                    [jboss@jboss-new bin]$ jrcmd 14451 print_memusage
                    14451:
                    Total mapped                  7519524KB           (reserved=2901796KB)
                    -              Java heap      3145728KB           (reserved=0KB)
                    -              GC tables       105232KB          
                    -          Thread stacks        46412KB           (#threads=138)
                    -          Compiled code      1048576KB           (used=15178KB)
                    -               Internal         1416KB          
                    -                     OS       162196KB          
                    -                  Other      2715820KB          
                    -            Classblocks        10496KB           (malloced=9603KB #28437)
                    -        Java class data       282624KB           (malloced=192131KB #133932 in 28437 classes)
                    - Native memory tracking         1024KB           (malloced=285KB #10)
                    
                    
                    [jboss@jboss-new bin]$ 
                    

                     

                    Java Heap is 3145728KB   and not increasing.  but Other is 2715820KB and increasing every time I execute this command.

                    current setup is : Centos  & JRockit.

                     

                    what is "Other" ???

                     

                    Also I want to say. I have 4 JBoss EAP 4.3 on other platforms (Sun Solaris. Linux) run on same jrockit and jdk but I don't have any problem.

                     

                    Problems is on JBoss EAP 6.1. and 6.2.

                     

                     

                    Thanks.

                    Regards,

                    Paata Lominadze.

                     

                     

                           

                    • 7. Re: JBoss EAP 6.2 Memory Leak, Memory Profiler (how to attach)
                      wdfink

                      From the output I don't see that this is a ThreadStack issue as this is listed separate.

                      I don't have an idea what "Other" can be.

                       

                      If you have a valid subscription you should open a support case and a JVM expert will look into this.