14 Replies Latest reply on Nov 6, 2006 4:56 AM by pyanguas

    Memory leak in PermGen space in 4.0.4

    pmpm

      After switching from 4.0.3sp1 with EJB3 RC6 to 4.0.4 I'm having fast growing memory leak during my app activity.
      When I'm generating automated test requests to my server, it eats about 1 MB of memory in PermGen in several seconds. Even after setting -XX:MaxPermSize=256m it dies after about 10 minutes with OOM Error.

      On previous JBoss version, even if my app is tested for hours, memory usage in PermGen is about 33MB and does not increase.

      On thing I changed in JBoss distribution is applying a patch to javassist that is described here
      http://www.jboss.com/index.html?module=bb&op=viewtopic&t=83199
      because javassist cannot deal with my entities when they implement interfaces with generic types.

      I suspect this is a problem with javassist, but completely does not know how to further diagnose this problem.

        • 1. Re: Memory leak in PermGen space in 4.0.4
          bill.burke

          we are aware that perm gen is leaking. We its on the roadmap to fix.

          • 2. Re: Memory leak in PermGen space in 4.0.4
            pmpm

            Thanks for reply, Bill
            Is this EJBTHREE-551 issue on JIRA? If so, my case is not connected with redeployment, it also leaks on fresh deploy.

            • 3. Re: Memory leak in PermGen space in 4.0.4
              clebert.suconic

              That issue if for redeployment leakages.

              But if we add some comment it could become general memory leaks IMO.


              Anyway, at this point I'm creating a testcase for JMS, using JVMTIInterface from JBossProfiler.

              Basically I'm doing:


              // I have to pre-create every service.
              
              // Execute the testcase at least once (to force creation of everything is supposed to be created)
              
              Map inventory1 = jvmti.produceInventory();
              
              ..... execute the testcase again.
              
              
              // clear some stuff
              
              Map inventory2 = jvmti.produceInventory();
              
              assertTrue(jvmti.compareInventories(System.out,inventory1,inventory2,....);
              
              



              If nothing was created, compareInventories is going to pass, what would mean no memory leaks. This way we will avoid regressions of memory leaks on the future.

              I guess we could use the same idea with this testcase.

              • 4. Re: Memory leak in PermGen space in 4.0.4
                clebert.suconic

                 

                "clebert.suconic@jboss.com" wrote:
                I guess we could use the same idea with this testcase.


                I mean... create a similar testcase withing EJB3 project.

                • 5. Re: Memory leak in PermGen space in 4.0.4
                  jsb

                  Any update on this?

                  pmpm, did you ever discover exactly was was causing the leak? Did you work out a solution to the problem?

                  I recently moved an app from 4.0.3sp1+EJB3rc3 to 4.0.4.GA+EJB3rc7. It was deployed in the upgraded fashion last night and died with an OutOfMemoryError: PermGen space error this afternoon. This was definitely not a redeployment case.

                  The app differences are mostly bring the annotations/JNDI stuff up to date with the changes between the EJB3 release candidates, but we did add a couple new features too. One leverages the TimerService scheduled via a SLSB. I see mention of the TimerService in JIRA issue 3071:
                  http://jira.jboss.com/jira/browse/JBAS-3071

                  Is it possible this could be causing the memory leak without a redeployment involved?

                  • 6. Re: Memory leak in PermGen space in 4.0.4
                    clebert.suconic

                    I will be working on this on Monday.

                    However first thing I have to do is to produce a testcase, and chase the leakage.

                    So, in EJB3 what exactly you "feel" that might be the cause? Proxies or Timers?

                    I need to follow an initial feeling to produce a failing testcase scenario.


                    clebert

                    • 7. Re: Memory leak in PermGen space in 4.0.4
                      djr667
                      • 8. Re: Memory leak in PermGen space in 4.0.4
                        jsb

                        Thanks Clebert,

                        My initial feeling was that it might be the ServiceTimer because we just added the code that uses it and because it's not available in an @Service context we're using it creatively through a SLSB injected into an @Service. However, if this was the case, I think it would be fairly track-able because we set a TimerService event at a fairly consistent interval.

                        After the crash yesterday we upped the MaxPermSize from 128m to 256m and the max heap size from 512m to 1024m. The server hasn't hit the PermGen problem since and the RES size of java on the server has stabilized in the 650m range. At this point it looks like it is not a memory leak issue after all, but rather simply a change in memory requirements between to the two versions of JBoss AS/EJB3 and the changes to our app.

                        I'm keeping a close eye on it and will report more information when I have it, but I'm thinking I was raising a false alarm.

                        djr667 - I don't think our issues are related. We run JBoss with the -server setting. We have had trouble with CGLib/Hibernate when running in -client mode due to differences in bytecode optimization between the two with Sun's Linux JVM (not memory leak related), but -server has been fine for us.

                        Thanks again,
                        Jonn

                        • 9. Re: Memory leak in PermGen space in 4.0.4
                          jsb

                          One other change to note in our app is the introduction of Seam. We are only using it with a couple of JSF backing beans as a test at this point (and I'm really liking it, by the way). Noting this because Seam likely also effects memory requirements and class loading in some way.

                          - Jonn

                          • 10. Re: Memory leak in PermGen space in 4.0.4
                            pmpm

                            Clebert,
                            I eventually found some time to test this thoroughly and I think I isolated, what is causing the leak.

                            I found that leak increases in only one situation: when I send to the server entities which contain relations to other entities that I have received from the server in previous calls (as proxies) (I'm using a standalone java client that communicates with jboss via RMI.)

                            For example:
                            I have two entities: Product and Invoice. Invoice has a collection of Products.
                            1. When I'm sending (from client) Products constructed on the client side and persisting them on the server - there is no leak.
                            2. When client queries server for some Products, then creates a new Invoice, adds Products to the collection in Invoice and then calls server to persist Invoice - I observe a leak.
                            3. But when I, to further test it, wrote a method that receives a collection of Product id's and creates Invoice on the server side - there is no leak.

                            I'm also using TimerService, but I did all the above tests with and without timers and I observed that it does not affect memory leakage.

                            Hope that will help You find a hook point in solving this problem
                            Pawel Miniewicz

                            • 12. Re: Memory leak in PermGen space in 4.0.4
                              modoc

                              I'm seeing the same memory issues with Jboss 4.0.5 EJB3 profile running seam. I have a VERY small/simple app. It uses a timer to run a method on a Seam component which is application scoped. After 10-12 hours it starts throwing the permgen memory errors. Is there any solution other than upping the perm space? It's such a tiny application it seems unfortunate to have to give it even more memory....

                              • 13. Re: Memory leak in PermGen space in 4.0.4
                                maxandersen

                                try and run with cglib instead of javaassist and see if that helps.

                                • 14. Re: Memory leak in PermGen space in 4.0.4
                                  pyanguas

                                  Hi,

                                  I had the same problem, tried cglib and it was fixed.
                                  Another possibility is to use latest javassist 3.3 (for example, from JBoss 4.0.5 GA).

                                  Pablo