11 Replies Latest reply on Jun 3, 2009 7:36 PM by gonorrhea

    Seam apps, jams server on perm gen error!

    sherkan777

      Hi, since couple weeks I try to fix my problem with JBoss 4.2.1 and Seam 2.0.1GA and perm gen memory leaks.


      After 4 days working, server gets down and throw perm gen space.
      Now I don't know how to fix it and clean my perm memory.


      Server has 4 GB ram on Centos OS.
      Here is my server conf



      f [ "x$JAVA_OPTS" = "x" ]; then
         JAVA_OPTS="-Xms256m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=256m"
         JAVA_OPTS="$JAVA_OPTS -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
         JAVA_OPTS="$JAVA_OPTS -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled"
         JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC -XX:ParallelGCThreads=2"
         JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails -verbose:gc -Xloggc:dupa"
         JAVA_OPTS="$JAVA_OPTS -Dorg.jboss.resolver.warning=true"
         JAVA_OPTS="$JAVA_OPTS -XX:CompileThreshold=100 -XX:MaxHeapFreeRatio=70"
         JAVA_OPTS="$JAVA_OPTS -XX:+UseFastAccessorMethods"
         JAVA_OPTS="$JAVA_OPTS -XX:-TraceClassLoading -XX:-TraceClassUnloading"
      # -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+StringCache -XX:+AggressiveOpts 
      fi



      On other java forum people told me to add

      JAVA_OPTS="$JAVA_OPTS -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled"



      but this not fixed my problem.


      Server works on JSK 1.5.09


      now i think to change my JDK to JRockit.


      That memory leaks happen only on my seam app, other apps (pure server apps, works fine).


      My app uses:


      JSF 1.2, facelets, Seam 2.0.1GA, postgres, quartz (implemented in seam), Rich Faces

        • 1. Re: Seam apps, jams server on perm gen error!
          luke.poldo.mailinator.com

          Our server has 4GB too...but we have 1GB with Permanent Memory, and 3GB for JBoss AS.


          JAVA_OPTS="-Xms1024m -Xmx3072m -XX:PermSize=1024m -XX:MaxPermSize=1024m"


          • 2. Re: Seam apps, jams server on perm gen error!
            sherkan777

            and your jboss works fine?

            • 3. Re: Seam apps, jams server on perm gen error!
              swd847

              only the sun vm has this problem, if you can handle the liscencing terms, using jrockit will solve your problem. The perm gen memory only leaks on hot deploys, so your server should need much less permgen than your development machine.

              • 4. Re: Seam apps, jams server on perm gen error!

                Until (if ever) the JVM becomes a Multi tasking JVM, this problem will haunt java forever, the only real solution now, is to program a reboot of you JVM every night (or wait until JOPR does it for you)... Or switch to JRockit...but I think that JRockit only seems to work because of its memory model is not segmented (but may also crash if the leak grows big enough)

                • 5. Re: Seam apps, jams server on perm gen error!
                  pgmjsd

                  Maybe the problem is simpler.   Have you turned off the deployment scanner? In production, we disable the deployment scanner in conf/jboss-service.xml.  This prevents other issues as well, and it's recommended by JBoss for production servers.


                  I have a Seam/JBoss application server running for weeks at a time, handling over 1 million HTTP requests a day, using -XX:MaxPermSize=96m.   This has been stable for over a year now.  We restart the JVMs once a week (over 7 million hits), just to be sure.   The only time we run out of perm gen space is on developer workstations, and then... only after several hot-deployments.






                  • 6. Re: Seam apps, jams server on perm gen error!
                    barakka

                    Hi,



                    Joshua Davis wrote on Mar 24, 2009 18:32:



                    I have a Seam/JBoss application server running for weeks at a time, handling over 1 million HTTP requests a day, using -XX:MaxPermSize=96m.   This has been stable for over a year now.  We restart the JVMs once a week (over 7 million hits), just to be sure.


                    Just out of curiosity: how do you afford the down time? Do you have a cluster and just shut down one node at a time?


                    Best,
                    Riccardo.

                    • 7. Re: Seam apps, jams server on perm gen error!
                      gonorrhea
                      The perm-gen area in the JVM can be monitored using jconsole.exe in the bin folder of your Sun JDK.  You will note that every time you re-deploy your EAR, for example, in a running JBOss AS, the perm-gen space will increase space allocation.  Eventually you may run out of perm-gen space if you re-deploy too many times in a row before restarting the JBoss server.

                      http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html

                      JRockit JVM does not have a perm-gen space so this is not an issue.

                      I have used JRockit JDK 1.5 and Sun HotSpot JDK 1.5 and experienced problems with both.  Don't recall the exact problem with JRockit but it only happened in dev with DEBUG mode in JBoss.  I have experienced perm-gen error many times.  Currently using HotSpot as JBoss tech sppt was unable to resolve the JRockit problem.  According to JBoss tech sppt, Sun JVM is the industry standard and most of their clients use that.  However, they do support JRockit as well and JRockit is supposed to have a 20% performance advantage.  I'm sure it really depends on your particular cluster and JVM tuning, etc.

                      from DAllen's book:

                      Sun JVM options when running JBoss AS

                      While you're on the task of setting up the JBoss AS runtime, I recommend a set of
                      JVM options to use when running JBoss AS with the Sun JVM. The default memory
                      allocation settings in Java are extremely conservative. On top of that, Sun’s JVM has
                      a concept of PermGen space, a separate allocated memory region from the heap.6
                      Even though the JVM garbage collector frees up memory automatically, there are certain
                      objects (such as class and method objects) that can evade removal because
                      they are in this isolated memory region. A flag must be provided that enables garbage
                      collection for the PermGen space. In addition, the garbage collector has trouble keeping
                      up when memory is tight. The limited resources quickly become a problem when
                      running a hefty application like JBoss AS that has high memory demands.
                      To avoid an untimely death of the JVM, and in turn, JBoss AS, you should supply the
                      following parameters in the VM options of the JBoss AS runtime configuration in your
                      IDE (or bin/run.conf in the JBoss AS home directory). I have never experienced an
                      out-of-memory error when using these settings.

                      -Xms128m -Xmx512m -Dsun.rmi.dgc.client.gcInterval=3600000
                      -Dsun.rmi.dgc.server.gcInterval=3600000
                      -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled
                      -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512m
                      -Xverify:none

                      There are ramifications of using the Concurrent Mark Sweep (CMS) garbage collector,
                      such as a longer startup time, so I wouldn’t blindly apply these settings to production.
                      However, I have found them to be sufficient for development purposes."
                      • 8. Re: Seam apps, jams server on perm gen error!
                        gonorrhea

                        Riccardo Serafin wrote on Mar 24, 2009 19:08:


                        Hi,


                        Joshua Davis wrote on Mar 24, 2009 18:32:



                        I have a Seam/JBoss application server running for weeks at a time, handling over 1 million HTTP requests a day, using -XX:MaxPermSize=96m.   This has been stable for over a year now.  We restart the JVMs once a week (over 7 million hits), just to be sure.


                        Just out of curiosity: how do you afford the down time? Do you have a cluster and just shut down one node at a time?

                        Best,
                        Riccardo.


                        JBOss in action states that the farmed deployment in AS 4.2 was unreliable and that it is no longer available in AS 5.  One of the advantages of using JBoss AS was the hot deployment feature of EAR/WAR and *-ds.xml in the deploy directory:


                        There are several different ways to do clustered deployments. JBoss provides a farming service, but it
                        has a bad track record and most people don’t rely on it in a production environment.
                        A popular way to handle cluster deployments is to define a deploy directory (as discussed in
                        Chapter 3) that points to a directory that is mounted on a network. On Windows this mount is usually
                        created using Server Message Block (also known as SMB or Microsoft Windows Network). On Unix,
                        Network File System (NFS) is often used to create the mount.



                        How many nodes in your cluster?  What are the logistics when you deploy a new app?  What web site is this that's running a Seam app with that many hits?

                        • 9. Re: Seam apps, jams server on perm gen error!
                          pgmjsd

                          Riccardo Serafin wrote on Mar 24, 2009 19:08:


                          Hi,


                          Joshua Davis wrote on Mar 24, 2009 18:32:



                          I have a Seam/JBoss application server running for weeks at a time, handling over 1 million HTTP requests a day, using -XX:MaxPermSize=96m.   This has been stable for over a year now.  We restart the JVMs once a week (over 7 million hits), just to be sure.


                          Just out of curiosity: how do you afford the down time? Do you have a cluster and just shut down one node at a time?

                          Best,
                          Riccardo.


                          Our users do not use the application at night and on weekends.  However, when it is up (7am to 5pm on business days) it really needs to be up.   We don't have to restart it on weekends, but we can so we do.


                          We do have a cluster, and we shut down one node at a time.



                          • 10. Re: Seam apps, jams server on perm gen error!
                            pgmjsd

                            Ron Ramirez wrote on Mar 24, 2009 22:06:


                            How many nodes in your cluster?


                            Two, currently.



                              What are the logistics when you deploy a new app?


                            We arrange downtime, usually on the weekends.  We pre-configure everything in new directories on the server, take the old instance down to migrate the database, start the new instance.   Our users are not online nights and weekends, so we can be down for a few minutes.



                            What web site is this that's running a Seam app with that many hits?


                            https://trade.parityenergy.com


                            It's an online commodity options platform for institutional traders.  The hits come from very fast Seam Remoting updates to a market data / order management screen.


                            • 11. Re: Seam apps, jams server on perm gen error!
                              gonorrhea

                              We are using EAP 4.3 CP01 with JavaService (not JavaService Wrapper, I know JavaService is not recommended).  There are currently two Seam apps deployed in the server/default/deploy directory.  This is a Windows 2000 box.  When I re-deploy a new version of an app on that instance, I stop the JBoss service first and then restart it after copying the EAR, foo-ds.xml, etc.


                              We experienced a strange permgen error in one of our envmts in the last couple days:


                              java.lang.OutOfMemoryError: PermGen space



                              As per the Jboss boot.log, the JDK being used yesterday and before (for several months most likely) was Jrockit 1.5.  JAVA_HOME envmt variable was pointing to Jrockit path.  I have uninstalled Jrockit and the current JDK is Sun jdk1.5.0_14JAVA_HOME is now pointing to Sun JDK path. 


                              After re-starting the Jboss windows service and running a few test cases, there are no exceptions in the server.log.  I was unable to connect to the JVM using jconsole when Jboss was running as a service.  I am able to connect to the processId using jconsole when I exec run.bat directly.


                              I don’t understand why/how the permgen OutOfMemoryError occurred twice in the last couple days in DV67 JVM.  There have been zero hot deployments, I always restart the JBoss AS via the JavaService.  I have not touched the configuration of that envmt in a long time (several months) and Jrockit JVM does not have a permgen space.


                              The only thing I can think of is that possibly the JAVA_OPTS is set as follows:


                              rem parameters as per Dan Allen in Seam in Action (boosted the min and max memory)
                              set |JAVA_OPTS|=%|JAVA_OPTS|% -Xms512m -Xmx1024m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512m -Xverify:none



                              I was using Sun JDK, switched to JRockit, now switched back to Sun.  Some (if not all?) of those options do not apply to the JRockit JVM obviously?


                              What does that JVM do then on startup when you pass unknown options like this?  Does is ignore them?  It was starting up fine...


                              What a mystery!


                              Isn't it impossible to get permgen error with JRockit JVM????!!!