7 Replies Latest reply on Oct 31, 2011 12:00 AM by kragoth

    Fast Memory Leak

    romain.dev.easycity.com

      Hi guys,


      I have a fast memory leak with my application using Seam 2.2.1.GA and JBoss 4.3.2.GA.


      The free memory on my server is going down until 0 in approximately 2hours. Here is the output when I do a jmap :


      ns302166:~# jmap -histo -F 18914 | grep com.easycity
      Attaching to process ID 18914, please wait...
      Debugger attached successfully.
      Server compiler detected.
      JVM version is 11.2-b01
      Iterating over heap. This may take a while...
      Finding object size using Printezis bits and skipping over...
      1387920 17349   com.easycity.entity.i18n.CategoryDetails
      291840  570     com.easycity.ejb.pages.localsearch.PoiInfoManagerBean
      287856  5997    com.easycity.entity.i18n.ECLocale
      239880  5997    com.easycity.entity.i18n.Lang
      205288  383     com.easycity.entity.core.poi.LocalBusiness
      199728  438     com.easycity.entity.community.Mb
      197888  1546    com.easycity.entity.core.poi.Category
      142448  614     com.easycity.entity.i18n.POIDetails
      140200  701     com.easycity.ejb.AppIdentity_$$_javassist_seam_2
      134592  701     com.easycity.ejb.AppIdentity
      99360   621     com.easycity.entity.utils.embedded.NavstPOI
      80800   505     com.easycity.entity.utils.embedded.LocalBusinessSales
      77280   420     com.easycity.entity.utils.embedded.FacebookUserData
      68640   390     com.easycity.ejb.pages.UniversalSearchManagerBean
      64416   732     com.easycity.entity.utils.Comment
      58384   82      com.easycity.ejb.util.PropertiesManagerBean
      57536   124     com.easycity.entity.core.poi.AppPOI
      42152   479     com.easycity.entity.utils.embedded.MbStats
      41664   651     com.easycity.entity.utils.embedded.PoiStats
      41664   651     com.easycity.entity.utils.embedded.Address
      41664   651     com.easycity.entity.utils.embedded.PoiGoogleStreetViewData
      36256   412     com.easycity.entity.i18n.Country
      34464   359     com.easycity.entity.core.poi.Revision
      28800   400     com.easycity.entity.i18n.City
      25560   213     com.easycity.entity.community.Review
      23112   107     com.easycity.entity.utils.embedded.TwitterUserData
      21888   171     com.easycity.entity.utils.embedded.NavstCountry



      If I run jmap again 1 minute later, more instances of each class and more space are used :


      ns302166:~# jmap -histo -F 18914 | grep com.easycity
      Attaching to process ID 18914, please wait...
      Debugger attached successfully.
      Server compiler detected.
      JVM version is 11.2-b01
      Iterating over heap. This may take a while...
      Finding object size using Printezis bits and skipping over...
      2172720 27159   com.easycity.entity.i18n.CategoryDetails
      431520  8990    com.easycity.entity.i18n.ECLocale
      366080  715     com.easycity.ejb.pages.localsearch.PoiInfoManagerBean
      359600  8990    com.easycity.entity.i18n.Lang
      326528  2551    com.easycity.entity.core.poi.Category
      310880  580     com.easycity.entity.core.poi.LocalBusiness
      252624  554     com.easycity.entity.community.Mb
      208336  898     com.easycity.entity.i18n.POIDetails
      188200  941     com.easycity.ejb.AppIdentity_$$_javassist_seam_2
      180672  941     com.easycity.ejb.AppIdentity
      148480  928     com.easycity.entity.utils.embedded.NavstPOI
      120320  752     com.easycity.entity.utils.embedded.LocalBusinessSales
      96968   527     com.easycity.entity.utils.embedded.FacebookUserData
      96624   549     com.easycity.ejb.pages.UniversalSearchManagerBean
      80736   174     com.easycity.entity.core.poi.AppPOI
      67640   95      com.easycity.ejb.util.PropertiesManagerBean
      64768   736     com.easycity.entity.utils.Comment
      62272   973     com.easycity.entity.utils.embedded.PoiStats
      62272   973     com.easycity.entity.utils.embedded.Address
      62272   973     com.easycity.entity.utils.embedded.PoiGoogleStreetViewData
      53152   604     com.easycity.entity.i18n.Country
      52624   598     com.easycity.entity.utils.embedded.MbStats
      48960   510     com.easycity.entity.core.poi.Revision
      41256   573     com.easycity.entity.i18n.City
      35040   292     com.easycity.entity.community.Review



      The EJB related to my web pages are Stateful and Conversation scope with @Begin(join true) on my init function and no @End. Performances are very poor.


      Please help me to find a solution and use seam as it should to prevent this. It hurts :-)


      Romain


      NB: com.easycity.ejb.pages.localsearch.PoiInfoManagerBean is an EJB ; com.easycity.entity.i18n.CategoryDetails is an entity bean.

        • 1. Re: Fast Memory Leak
          kragoth

          It's pretty hard to pin point issues with this amount of information.


          During your 2 hours what is going on? How many sessions are being created. How many are destroyed over these 2 hours? If you have


          my web pages are Stateful and Conversation scope with @Begin(join true) on my init function and no @End
          



          what are you relying on to end your conversations? Are you just relying on session timeout? What's the average number of connections during the 2 hour window?


          Seam isn't going to be creating instances of entities just for fun. This is your code that is doing this.


          You need to do more investigation then just a memory dump of your entities. All this does is show how many of your entities are in memory. This proves nothing about a memory leak.


          • 2. Re: Fast Memory Leak
            romain.dev.easycity.com

            Hi Tim,


            Thank you for your help. There is about 1 cnx /second. I don't know the way to end a conversation when the user leave the page. So my first guess was conversations and memory keep creating each time a user visits a page.


            Maybe stateful conversation scope is not appropriate. I just need to load single pages with forms and ajax calls in it. I tried to change some of them to Stateless PAGE scope but I get a very strange behaviors: page loads with the data of the previous visitor when I load the page !!!


            When a do a top command on my hosting server, I see that the free memory space decreases continuously. I have to reboot jboss every 1 hour now.


            Thanks again,
            Romain


            • 3. Re: Fast Memory Leak
              kragoth

              Well, it's really hard to know exactly what to do without seeing your code base. But, because you do not end your conversations my first reaction would be to shorten the session timeout to something quite small. When a session is destroyed it will/should clean up all conversations that were in that session.


              However, realistically the problem probably lies in your code somewhere. Maybe you have an application scoped bean that is referencing things so they can't be Garbage Collected.


              At the end of the day though. One connection per second although it is higher then the app I'm working on (business app) it is not an unreasonable amount as during peak hours we probably experience similar amounts. But we still need more information from you. Each connection what is it doing? How long does it take to do what it does. Is it common or desired that these connections do many things before these connections are abandoned by the client. (logging out or just no longer used or whatever they do). Do people log into your system and then log out?


              Could you provide maybe a small paragraph describing the common operation (from a technical perspective) of what the general use case is of your app. Maybe then we can start understanding where things can be improved. Because as you have already thought maybe conversation scope isn't really what you need.


              Maybe stateful conversation scope is not appropriate. 
              I just need to load single pages with forms and ajax calls in it.
              I tried to change some of them to Stateless PAGE scope but I get a very strange behaviors: 
              page loads with the data of the previous visitor when I load the page !!!
              



              This problem could be a number of things. First it could purely be browser caching. IE8 is especially bad for this as the session object is shared between many tabs and windows. And although you may have 3 different windows up they all share the same session object. So be sure to check that this isn't the simple reason.


              The next obvious cause would be that you are storing state somewhere in an application scoped bean that's causing the issue.


              Last but not least if you are dealing with a mixture of conversation and page scoped beans it is very easy to get confused about the real scope of the data you are looking at.



              So, I guess the next step is to give a technical rundown of what you are doing. Why did you think conversation scope was appropriate? If you do not have @End for any of your conversations and are relying on session timeout, what is that session timeout? Seeing an app that does not have @End would immediately make me think that long running conversations is probably not the right solution. A long running conversation should have a begin and an end. It is like a unit of work that at the end you would hit save/finish and that would end your conversation.


              Our ability to help you with your issue is going to be directly related to how much information you give us. Carefully go through this post and my last and answer every question as best you can.


              Do you have a test environment where you can replicate the problem without the outside influence of other users?

              • 4. Re: Fast Memory Leak
                romain.dev.easycity.com

                Thank you Tim. I'm in vacation until August, 23. I really appreciate your help and I'll investigate as soon as I'm back.


                Best regards,
                Romain

                • 5. Re: Fast Memory Leak
                  jeanluc

                  (funny, I can simply paste a reply I just posted to another thread)


                  Use a profiler. You'll find out exactly what's leaking. If you don't have a profiler, download the trial version of YourKit (it's an excellent profiler, by the way). Once you connect to the server, watch the instances and use the delta function to show what objects are created. You may not even need to look at the heap to find out.

                  • 6. Re: Fast Memory Leak
                    romain.dev.easycity.com

                    Here is the output with YourKit :
                    - Memory usage
                    - Memory object stack


                    And the server monitoring (with reboot every hours):
                    - Server ressources usage


                    Any clue ?


                    Thanks a lot for your help

                    • 7. Re: Fast Memory Leak
                      kragoth

                      Unfortunately the sites you have linked are blocked by my work's proxy rules. sigh