9 Replies Latest reply on Aug 21, 2013 6:11 AM by fredfrancisco

    Serious memory leak in AS7.1.1

    k_benary

      Hi,

       

      currently we are facing a blocking memory leak in AS 7.1.1.

       

      In a test application I read 12000 EJB 2.1 EntityBeans. Hereafter,  there are about 5 million (!!) Interceptor[] arrays firmely bound in the heap.

      I made a heap dump and hat a look at where they are bound. A typical reference chain is this one:

       

      GC Root: ServiceContainerImpl$ServiceThread

      ServiceContainerImpl$ServiceThread.container = ServiceContainerImpl

      ServiceContainerImpl.registry = UnlockedReadHashMap

        UnlockedReadHashMap.table = AtomicReferenceArray

         AtomicReferenceArray.array = Object[16384]

          Object[6209] = UnlockedReadHashMap$Item[3]

           UnlockedReadHashMap$Item[1] = UnlockedReadHashMap$Item

             UnlockedReadHashMap$Item.value = ServiceRegistrationImpl

              ServiceRegistrationImpl.instance = ServiceControllerImpl

               ServiceControllerImpl.serviceValue = ImmediateValue

                ImmediateValue.value = TimedObjectInvokerImpl

                  TimedObjectInvokerImpl.timeoutInterceptors = HashMap

                    HashMap.table = HashMap$Entry[128]

                     HashMap$Entry[123] = HashMap$Entry

                      HashMap$Entry.value = ChainedInterceptor

                        ChainedInterceptor.interceptors = Arrays$ArrayList

                         Arrays$ArrayList.a = Interceptor[6]

       

      It looks as nearly all of them are tied to the ServiceThread of the ServiceContainerImpl.

       

      Is there any known issue covering this? We urgently need information on this since this makes EJB2.1 usage on AS7.1.1 impossible.

       

      Thanks

      Klaus

        • 1. Re: Serious memory leak in AS7.1.1
          jaikiran

          Looking at the code I can see a possible improvement. But is this really a leak? Do those instances stay behind after the application is undeployed? I understand 5 million interceptors shouldn't be around (and there's a way to improve that), but I want to understand the extent of this issue.

           

          P.S: Can you also check this against latest nightly build https://community.jboss.org/thread/167590

          • 2. Re: Serious memory leak in AS7.1.1
            k_benary

            Yes, it is a leak. If I read another 12000 beans I have 10 millions of them and so on. Eventually the AS goes OOM if I jut keep it going long enough. And they stay in memeory even after undeploying the applcation. So, IMO it looks serious.

             

            I will try to check this against the latesr build asap and post the results here.

            • 3. Re: Serious memory leak in AS7.1.1
              k_benary

              Now I managed to run the test on the latest 7.1.2 build. Unfortunately the situation has not changed. I used jmap with a preceeding full GC for counting the Interceptor arrays:

               

                  190762   (after startup and deployment)

                5507902   (after reading 12000 beans)

              10790063   (after reading the same 12000 beans again)

               

              At exactly the same magnitude there are java.util.Arrays$ArrayList instances growing. Some interceptors follow behind. Here are the top scorers:

               

              num     #instances         #bytes  class name
              ----------------------------------------------
                 1:      10790299      258967176  java.util.Arrays$ArrayList
                 2:      10790063      244139632  [Lorg.jboss.invocation.Interceptor;
                 3:       7059524      112952384  org.jboss.invocation.WeavedInterceptor
                 4:       3517332       84415968  org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor
                 5:       3517332       84415968  org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1
                 6:        227890       64023408  [Ljava.lang.Object;
                 7:       3730539       59688624  org.jboss.invocation.ChainedInterceptor

               

               

               

               

               

               

               

              • 4. Re: Serious memory leak in AS7.1.1
                safetytrick

                This looks strikingly similar to this (not mine, just helped me out): http://lists.jboss.org/pipermail/jboss-jira/2012-February/208418.html

                 

                In my application jsp tags were holding references to my objects long after they were no longer being used.

                 

                The fix is to add this to your system properties.

                 

                    <system-properties>

                        <property name="org.apache.jasper.Constants.USE_INSTANCE_MANAGER_FOR_TAGS" value="false"/>

                    </system-properties>

                 

                This is just a guess since you don't mention anything jsp tag related. The jmap results just look very similar to my own symptoms.

                 

                Michael

                • 5. Re: Serious memory leak in AS7.1.1
                  k_benary

                  I am afraid this is not the point. 

                  Firstly because this is no Servlet/JSP container issue, but en EJB container one. Second, the issue you mentioned was solved with AS7.1.0 Final.

                   

                  Thanks anyway

                  Klaus

                  • 6. Re: Serious memory leak in AS7.1.1
                    jaikiran

                    Klaus, I haven't been able to get to this yet. But please file a JIRA https://issues.jboss.org/browse/AS7 so that we don't forget about this one.

                    • 7. Re: Serious memory leak in AS7.1.1
                      k_benary

                      Done. https://issues.jboss.org/browse/AS7-4675

                       

                      Besides: There is another issue coming from EJB2.1 CMP/CMR which is completely blocking us: https://community.jboss.org/message/731706#731706

                      There is no qualified answer to this so far. Should I file an issue for that too?

                      • 8. Re: Serious memory leak in AS7.1.1
                        fredfrancisco

                        https://issues.jboss.org/browse/AS7-4675

                         

                        This is marked as resolved, and the fix is in my version 7.2.0.Final, but I am hitting this big time.

                         

                        I am calling a session bean (tx required) that fetches a lot of data, 45000 rows, using read-ahead. In less than a minute my old generation is exhausted and begins a cycle of Full GCs. I tried running on EAP 6.1.0, the result was the same.

                         

                        I did a heapdump, debugged Jboss7 code and jprofiled There is an enormous amount of interceptors being created, for every getter I call on my bean 500 are created, that hold the memory for the duration of the transaction so no space can be freed. Why are so many interceptors being created? Can anyone give me any pointers?

                         

                        Am am running Jboss-7.2.0.Final after migrating from Jboss-4.0.4.GA and the performance regression is significant but still acceptable. This problem, however, kills my server. My plan is to move away from ejb2 but that cannot be done at the moment.

                         

                        In Jboss 4, from what I understand, transation synchronizations and certain interceptors where bypassed if we called a get* method. Is this possible in Jboss7?

                         

                        OS: Fedora 18 (x86_64)
                        Jboss: 7.2.0.Final

                        • 9. Re: Serious memory leak in AS7.1.1
                          fredfrancisco

                          Hello jaikiran pai.

                           

                          Could you elaborate on the improvements you mentioned?

                           

                          Thanks.