1 2 Previous Next 25 Replies Latest reply on Jun 10, 2011 3:32 PM by wkirke Go to original post
      • 15. Re: JBoss 5.1 crash
        peterj

        -XX:ParallelGCThreads=20

        ...

        Number of processors: 4

        Never set GC threads > number of CPUs. By setting 20 GC threads, GC will run slower due to processor contention among the threads. GC is a memory-bound operation - each thread will use up its quota before being kicked off of the CPU. Thus for 4 CPUs, 4 GC threads is optimal.

         

        Also, I think that your permsize setting is low - I would up it to 256M or higher.

         

        Committed virtual memory permanently grows and when no Free swap space is availible service is killed.

         

        This is not unusual due to how Windows manages memory and how the JVM manages the heap. Youy might consider gathering the GC statistics and graphing them.  For suggestions on how to gather GC data and analyze it, see these white papers:

        Java Garbage Collection Statistical Analysis 101

        Java Garbage Collection Performance Analysis 201

        at http://www.cmg.org/cgi-bin/search.cgi?q=java+peter+johnson&x=30&y=10

        • 16. Re: JBoss 5.1 crash in the end (Native Memory Leak)
          peterj
          Can +UseParallelOldGC cousing the Virtual Memory growth?

          No. The JVM allocates all of the GC when it starts. Those threads hang around in a wait state until GC kicks in, at which point they becoime active. Once GC is done, they are inactive again. So there is no allocation/deallocation of GC threads. (If you do a thread dump, youy will see the GC threads at the bottom of the dump output.)

          • 17. Re: JBoss 5.1 crash in the end (Native Memory Leak)
            sashka_nem

            Peter thank for useful response. I read the white papers and will do the statistical analysis of my GC.

            I will take in mind your advice regarding parallel GC threads.

             

            Just for the info at the moment I have upgraded to 1.6.0_23 Oracle (Sun) JDK (to give it a try), but my Windows Virtual Memory still permanently grows (mostely in daytime when the application has nearly 80-120 concurrent users).

             

            Also, I think that your permsize setting is low - I would up it to 256M or higher.

            I have attached my perm gen picture to the post, and used size always stays at the same value (Very very litttle variations). Why should I rise the size?

            JC5.gif

             

            This is not unusual due to how Windows manages memory and how the JVM manages the heap.

            So how can I limit Virtual Memory usage for my JVM (to avoid reaching maximum aqvailible 10GB in two days)?

            How GC statistics can help me with this problem?

            • 18. Re: JBoss 5.1 crash in the end (Native Memory Leak)
              sashka_nem

              upgrading to 1.6.0_23 had no effect, Virtual Memory keeps growing

              • 19. Re: JBoss 5.1 crash in the end (Native Memory Leak)
                peterj

                Have you tracked the number of threads being used? There are only two things I can think of that would cause Java process memory usage to grow:

                 

                • opening file because each file handle takes up memory outside of the heap (though I would be morfe suspicious of this if the permgen kept growing because that could imply JAR and similar files being opened but theh andles not released)
                • new threads being created. Each tread uses 512KB - 1024KB of additional memory. If I recall correctly, there have been posts about http threads increasing in number.
                • 20. Re: JBoss 5.1 crash in the end (Native Memory Leak)
                  sashka_nem

                  Thank you Peter for your time

                   

                  Memory growth I've investigated so far are defenitely related to UI interactions (facelets, richfaces, ....).

                  Maybe something is not closing open file handles, or some leak when rescanning xhtml files inside zips(jars).

                  At the moment I'm simulating multi-user load by using selenium server on several computers.

                  The good thing is that I'm able to reproduce memory growth on testing server so I will write what is cousing my problem eventually.

                  • 21. Re: JBoss 5.1 crash in the end (Native Memory Leak)
                    sph007

                    I've seen native memory leaks on Linux systems due to zip input streams that were dereferenced without being explicitly closed.  You may want to check your source code for that condition.

                    • 22. JBoss 5.1 crash in the end (Native Memory Leak)
                      sashka_nem

                      So far I have come with conclusion that

                       

                      <context-param>

                        <param-name>facelets.REFRESH_PERIOD</param-name>

                        <param-value>-1</param-value>

                      </context-param>

                       

                      Solved the problem.

                      • 23. JBoss 5.1 crash in the end (Native Memory Leak)
                        kellyfj1

                        Alexander,

                            In what file or files do I place that <context-param> - do you have any references for why that works?

                         

                        Thanks!

                         

                        -Frank

                        • 24. JBoss 5.1 crash in the end (Native Memory Leak)
                          sashka_nem

                          Hi frank, in case you use facelets in your project, you must put that params in web.xml for production.

                          It avoids rescanning ZIP Archives in tmp files of JBoss, to avoid native memory leak, Win, 64bit enviornment.

                          • 25. Re: JBoss 5.1 crash in the end (Native Memory Leak)
                            wkirke

                            If you're still having memory leak issues with JBoss 5.1, you might check out these posts:

                            http://www.zeroturnaround.com/forum/topic.php?id=1020

                            The last post by aheino says that his leak was caused by a bug in JBoss, that has a fix that he did not have installed.

                             

                            https://jira.jboss.org/browse/JBVFS-134

                            This is the bug aheino referred to.  Apparently, the default configuration for the VFS cache repetedly creates context objects once it times out and these objects need finalization, so our GC's finalize queue grew faster than the GC could finalze the objects, thus causing memory to be consumed over time.

                             

                            This bug was apparently fixed in JBoss in May 2010 and also incorporated in another patch in July 2010, but my 5.1 installation did not have these patches.  I used memory dumps and eclipse Memory Analysis to figure out what classes were consuming memory and how many instances were in the heap.  I then compared multiple dumps over time to see which large memory usage classes increased their count over time.  I saw tons of these classes were using most of our memory: org.jboss.virtual.plugins.context.zip.ZipEntryContext.

                             

                            Since I don't control the patches on the machine my project uses, I tried the workaround (changed IterableTimedVFSCache to TimedVFSCache), and it fixed our memory leak.  Here's the details of what I changed:

                             

                            File: conf/bootstrap/vfs.xml

                            Changed the bean class used for property name="realCache"

                            From "org.jboss.virtual.plugins.cache.IterableTimedVFSCache" to "org.jboss.virtual.plugins.cache.TimedVFSCache".

                             

                            -Bill

                             

                            1 2 Previous Next