9 Replies Latest reply on Jan 24, 2012 2:51 PM by genman

    Rhq-agent 4.3.0 memory leak

    cesello1969

      Hi,

      I've compiled the last git release (or close to last one ,2b26d4412f3fbc28f4131efa7732f3f5e31e4955) and is running now on 4 clients (3 rhel5, 1 Debian 6.0).

      I've experienced a slow system memory leak (faster in rhel5 and slower in Debian 6.0 , 15MB per hour in rhel5 and 5Mb per hour in Debian 6.0).

      It's possible some unclosed handle using augeas or sigar ?

        • 1. Re: Rhq-agent 4.3.0 memory leak
          cesello1969

          I can add a sample from rhq server used as monitor for resourcesleak.JPG

          • 2. Re: Rhq-agent 4.3.0 memory leak
            ips

            What types of and how many server Resources are the Agents managing? (e.g. 2 JBoss EAP 4.3 servers and 2 Apache 2.2 servers per Agent)

             

            Also, hava you tried using a profiler such as JProfiler to pinpoint what is leaking?

            • 3. Re: Rhq-agent 4.3.0 memory leak
              juliobarreto

              Good afternoon

              How do I download page for version RHQ 4.3

              thank you

              Julio

              • 4. Re: Rhq-agent 4.3.0 memory leak
                ips

                4.3 hasn't been released yet. 4.2 is still the latest release. If you want to try out the very latest code, you will have to build from source.

                • 5. Re: Rhq-agent 4.3.0 memory leak
                  cesello1969

                  Hi Ian,

                  both are monitoring a single instance of jboss as 7 and typical OS stuffs (filesystem,sshd, cpu.. the standard) . The rhel5 is monitoring a postgres 8.4 instance.

                  The jpg attached is taken from rhq server interface but obviously I see the memory leaks directly from OS too. Tipically after 3-4 days I've a 2.4G swap file used and I have to restart rhq-agent.

                  I'll use Jprofile to pinpoint the problem a little better

                  • 6. Re: Rhq-agent 4.3.0 memory leak
                    genman

                    There is definitely a native memory leak.

                     

                    I updated my hosts from 4.0 to 4.1 and I see the process looks like this:

                     

                      PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEM


                    15   0 11.0g  10g  13m S  0.0 33.0  26:03.99 java                                                                                                            

                     

                    That's 11g of memory usage. jhat and the like show quite a lot less in the JVM itself. Here is a histogram:

                     

                    num #instances     #bytes  class name
                       1:     39295    5184840  <constMethodKlass>
                       2:     39295    4725784  <methodKlass>
                       3:      4091    4414496  <constantPoolKlass>
                       4:     43604    3832328  [C
                       5:     64747    3511200  <symbolKlass>
                       6:      4712    3101592  [I
                       7:      4091    2994248  <instanceKlassKlass>
                       8:      3706    2709792  <constantPoolCacheKlass>
                       9:     23291    2499224  [Ljava.lang.Object;
                      10:     12682    2042320  [Ljava.util.HashMap$Entry;

                     

                    $ /usr/java/jdk1.6.0_17/bin/jmap -heap 28675 
                    Attaching to process ID 28675, please wait...
                    Debugger attached successfully.
                    Server compiler detected.
                    JVM version is 14.3-b01
                    
                    using thread-local object allocation.
                    Parallel GC with 8 thread(s)
                    
                    Heap Configuration:
                       MinHeapFreeRatio = 40
                       MaxHeapFreeRatio = 70
                       MaxHeapSize      = 134217728 (128.0MB)
                       NewSize          = 2686976 (2.5625MB)
                       MaxNewSize       = 17592186044415 MB
                       OldSize          = 5439488 (5.1875MB)
                       NewRatio         = 2
                       SurvivorRatio    = 8
                       PermSize         = 21757952 (20.75MB)
                       MaxPermSize      = 88080384 (84.0MB)
                    
                    I'm guessing the plugin update/upgrade doesn't update the shared libraries as well?
                    
                    Or the new versions have leaks?
                    
                    Here is what I have (notice the dates):
                    
                    $ ls -l /usr/local/rhq-agent/lib/augeas/lib/
                    total 412
                    -rw-r--r-- 1 root root 260432 Jul 26 20:19 libaugeas.so
                    -rw-r--r-- 1 root root  72404 Jul 26 20:19 libfa.so
                    -rw-r--r-- 1 root root  72404 Jul 26 20:19 libfa.so.1
                    

                     

                    Added formatting

                    • 7. Re: Rhq-agent 4.3.0 memory leak
                      mazz

                      Did you write up a BZ? If not, can you do so and document all the relevent info (OS version, hardware, etc. etc.). Also indicate what resources you have in inventory and are monitoring. Don't think we've had any reports of their agent process growing to 11gig. Its either gonna be sigar or jna/augeas. My money would be on jna/augeas but that's just a WAG.

                      • 8. Re: Rhq-agent 4.3.0 memory leak
                        genman
                        Bug 766959 has been added to the database

                        I added information on Inventory. It's nothing special (no weird resources, I think.)
                        • 9. Re: Rhq-agent 4.3.0 memory leak
                          genman

                          I found out leaks in the Grub and OpenSSH plugins. If you disable them, then you shouldn't see any more leaks.