6 Replies Latest reply on Apr 21, 2008 11:10 AM by peterj

    JBoss 4.05 don't release free memory - Out of Memory error

    bakry

      Dear All,

      I know this is not the first time someone asks about OOME... but I have searched all relevant topics with no results...

      My problem is with Jboss 4.0.5, I have one application deployed in this server with very minimal hits on the server. however the server keeps consuming memory by time... the strange thing is that there always free memory not returned to the OS... after 3-5 days the server total memory reaches the max allowed memory, then the server throws the OOME.

      following is my memory settings in the run.cfg

      JAVA_OPTS="-Xms128m -Xmx1500m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"


      Following is what the web console is displaying about the server state. you can see there is 981MB free memory not returned to the OS. my understanding is that Jboss start with the minimum memory configured to 128MB and keeps getting more as needed, but when the allocated memory is not used anymore (becomes Free memory), Jboss should return this memory to the OS and the total memory figure should go down to show the actual currently use memory + the buffered memory in Jboss. Please correct me if I am wrong...

      JVM Environment
      Free Memory: 981 MB
      Max Memory: 1333 MB
      Total Memory: 1058 MB
      #Threads: 78
      JVM Version: 1.5.0_07-b03 (Sun Microsystems Inc.)
      JVM Name: Java HotSpot(TM) Server VM


      I tried every thing from increasing the heap to memory tuning in my application but itis not working.

      Regards,
      Bakry

        • 1. Re: JBoss 4.05 don't release free memory - Out of Memory err
          peterj

           

          my understanding is that Jboss start with the minimum memory configured to 128MB and keeps getting more as needed, but when the allocated memory is not used anymore (becomes Free memory), Jboss should return this memory to the OS


          No, that is not correct. The JVM, once it asks the OS for memory, will never release that memory back to the OS. If you do not want the JVM to consume 1.5GB of memory, do not set -Xmx that high.

          • 2. Re: JBoss 4.05 don't release free memory - Out of Memory err
            peterj

            I was wrong. I just talked to my JVM expert and the JVM will give memory back to the OS under certain circumstances. It will give back memory only during a garbage collection. Taking a snapshot of memory usage is not sufficient to make any conclusion about memory. If the JVM is not doing anything it will not just suddenly wake up and say "oh, I have too much memory, I think I will give some back." You need to gather garbage collection data while running the JVM and that data will indicate when, or if, the JVM is giving memory back to the OS.

            Note that JBossAS has no say whatsoever in this matter.

            • 3. Re: JBoss 4.05 don't release free memory - Out of Memory err
              bakry

              hi peter,

              thanks for the reply..

              I agree with u, the JVM should return memory to OS after GC... but this is not happening in my case because I can see the total memory never decrease (while monitoring for 3-5 days)... so there is no memory returned to OS, which conflict with our theory....

              I have also monitored the jboss process memory-utilization on the unix machine it gave the same figure as the Jboss admin console... that is my jboss process keeps eating memory without returning it back to OS till it reaches the max assigned memory then it give the OOME....

              I am wondering is this a known issue in 4.0.5 (I have another server 4.2.1 running in the same machine, itis not doing that).... if not, then what to do to get red of this... i tried every thing with no result...

              Please help..

              Regards
              Bakry

              • 4. Re: JBoss 4.05 don't release free memory - Out of Memory err
                peterj

                You mention monitoring JVM memory usage. How are you doing that, using system tools? Unfortunately, that is not very helpful. You really need to monitory garbage collection data (e.g., -verbose:gc and similar JVM options) and examine that. That is the only thing that will tell you what is really going on.

                Be aware that there will often be unused heap memory - the JVM will rarely use all of the memory that it asks for. For example, with a 1GB heap and a 200MB you generation, you will typically have at least 200MB of unused heap memory (unless you have a memory leak). The unused memory is a side0effect of the algorithm the JVM uses to determine when to do a full garbage collection.

                See my presentation at http://www.cecmg.de/doc/tagung_2007/agenda07/24-mai/2b3-peter-johnson/index.html

                • 5. Re: JBoss 4.05 don't release free memory - Out of Memory err
                  bakry

                  I have monitored the memory using system tools, jboss web console and jprofiler... all gave the same result..

                  following is snapshot of jboss web-console main screen showing the resouce utilization

                  JVM Environment
                  Free Memory: 981 MB
                  Max Memory: 1333 MB
                  Total Memory: 1058 MB
                  #Threads: 78
                  JVM Version: 1.5.0_07-b03 (Sun Microsystems Inc.)
                  JVM Name: Java HotSpot(TM) Server VM

                  In the run.cfg I configured the startup memory to 128m and the max to 1.5g... jboss start with 128m and kept eating more memory (although that there were always free memory avaialble --- monitoring the free memory figure of the web-console)..... till it reached this state of consuming 1058mb... my understanding Jboss/JVM should not ask the OS for more memory if there is free memory available (cleaned by garbage collector)... with time the total memory reaches the max memory and the jboss starts throwing OOME.

                  so the question again, what can I do to make the JVM/JBoss uses the free memory instead of consuming new memory from the OS.

                  Regards,
                  Bakry

                  • 6. Re: JBoss 4.05 don't release free memory - Out of Memory err
                    peterj

                    I repeat : none of those tools really tell you what is going on with garbage collection. They only give snapshots of current memory usage. You need to use -verbose:gc or -XX:PrintHeapAtGC, or a similar option to find out what is really going on.

                    The JVM will eat up memory if it thinks it needs more - this has more to do with the amount of objects that survive the gc than with how much free space is available after the gc. If you want to limit its memory consumption, set the max heap to a reasonable amount. I always recommend setting min and max heap to the same size. But only after analyzing -verbose:gc data to determine the optimal heap size.

                    Is the OOME due to the heap or the perm gen? What is the exact error message?