7 Replies Latest reply on Jun 26, 2009 11:59 AM by Stuart Douglas

    Seam app on tomcat and growing swap...

    Gregory Nikle Apprentice

      Hi,
      I've created seam app, on Seam 2.0.1GA, richfaces,facelets, jsf 1.2, withoud ejb/jpa, just pure jdbc on tomcat 6.
      My server has 4 GB ram, core 2 duo, centos OS.


      Here is my tomcat conf.



        JAVA_OPTS="$JAVA_OPTS -server -Xms2048m -Xmx2048m -XX:PermSize=64m -XX:MaxPermSize=128m"
      
        JAVA_OPTS="$JAVA_OPTS -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
      
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:-TraceClassUnloading -XX:+ClassUnloading -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -XX:CompileThreshold=100"
       
        JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails -verbose:gc -Xloggc:/home/grzegorz/jvm"



      at now tomcat uses 2gb ram,
      After 1 or 2 day, my swap get over 4-5gb and still grow until I get


      java.lang.OutOfMemoryError: Java heap space



      To check what is in jvm I've added logging jvm status to jvm.log fle
      It seems GC after full GC cleans all data.




      9031.181: [GC [PSYoungGen: 479984K->2320K(474176K)] 1875918K->1398822K
      (1872320K), 0.0114430 secs]
      
      9031.193: [Full GC [PSYoungGen: 2320K->0K(474176K)] [PSOldGen:
      1396501K->295482K(1398144K)] 1398822K->295482K(1872320K) [PSPermGen:
      58204K->58204K(524288K)], 37.3468510 secs]
      
      9071.531: [GC [PSYoungGen: 471808K->4192K(469952K)] 767290K->305865K
      (1868096K), 0.0277160 secs] 



      but my cheap is growing and growing etc...


      After 3 days I must to restart my tomcat to clean swap and start app again.
      I don't know why jvm.log file show me that Full GC cleand all memory, but swap is getting bigger and bigger until tomcat throw this error.


      I've profiled my app by Yourkit Profiler, and it looks after logout, user session is completly cleaned, and If I force full gc in compiler all memory is cleaned too.


      I use jvm 1.5.10 sdk, tomcat 6.


      I thougth this swap can full fill my postmaster of postgres, but it is not.
      Today after tomcat restart, my swap was cleand from 4,8gb to 1.01 gb. Ram in server top was cleaned to free 3.x gb, and tomcat at now works fine.


      Can anybody help me what this could be, or how or where check anythink to find this leak.
      I've no idea what to check else.


        • 1. Re: Seam app on tomcat and growing swap...
          Stuart Douglas Master

          This is very odd, according to those settings, tomcat should never use more that 2Gb of ram. Have you ran top or similar to make sure it is actually the java process itself that is using up the memory and not some other related process? If it really is java then it is a bug in the JVM.


          The only thing that I can think of is that maybe tomcat is leaking external resources such as database connections? Although even then postgres should hit max connections before it starts chewing ram to much.


          I would update to the latest JVM, and confirm that it is actually the java process that is using up all the ram.

          • 2. Re: Seam app on tomcat and growing swap...
            Gregory Nikle Apprentice

            Hi Stuart,


            I've checked top in centos, java has between 50-60% of mem usage.


            My connector is configured to handle 300 requests. Postgres maxconnections is set to 1000.
            Production server is used over 3000 users per day.


            Tomcat is configured to removeAbandonedConnections


            <Resource name="jdbc/one" auth="Container"
                           type="javax.sql.DataSource" driverClassName="org.postgresql.Driver"
                           url="jdbc:postgresql://localhost:5432/one"
                             removeAbandoned="true" removeAbandonedTimeout="300" logAbandoned="true"
                           username="postgres" password="root" maxActive="10000" maxIdle="30" maxWait="-1"/>



            It is possible if I made some connection leak, but If tomcat remove that connection, whole resource should be cleaned.


            I'll try to upgrade to jvm 1.6.x, and I'll give u an answer what happen.
            I think also to move to jrockit, but my knowledge about it, is poor (newer used). Each time I read some posts about suns jvm i find some post about, how jrockit is cool, nice, best etc.


            Is it right? Have u got any experience with jrockit?

            • 3. Re: Seam app on tomcat and growing swap...
              Gregory Nikle Apprentice

              Sorry, my bad.
              My java version is:


              Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
              Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_06-b05, mixed mode)
              



              • 4. Re: Seam app on tomcat and growing swap...
                Stuart Douglas Master

                I have used jrockit, but moved back to hotpot when oracle bought out bea. It works pretty good, but when I said uprgade I would try 1.5.19 first, moving to java 6 could cause other problems.


                Are you using the tomcat native connector by any chance? Maybe if the native code was leaking it would not be restrained by the max heap space settings?

                • 5. Re: Seam app on tomcat and growing swap...
                  Gregory Nikle Apprentice

                  Stuart Douglas wrote on Jun 26, 2009 10:55:


                  Are you using the tomcat native connector by any chance?


                  You mean connector to db?
                  Its postgres connector:



                  postgresql-8.2-509.jdbc3.jar



                  Ok, I'll move to last 1.5.x java ver.



                  • 6. Re: Seam app on tomcat and growing swap...
                    Gregory Nikle Apprentice

                    Maybe if the native code was leaking it would not be restrained by the max heap space settings?

                    Yes, you're right.

                    • 7. Re: Seam app on tomcat and growing swap...
                      Stuart Douglas Master

                      No, i'm not talking about the postgres driver, it is 100% java so should not cause these issues. I was talking about the tomcat native connector


                      This may have come bundled with your version of tomcat, and considering you are serving 3000 users a day I figured there is a good chance they are in use.