4 Replies Latest reply on Jan 20, 2015 10:24 AM by hoffmeij

    OutOfMemory with Infinispan 5.3.0 in glassfish v3 cluster

    infinispan_user0815

      There was an memory leak in AbstractJBossMarshaller (see issue https://issues.jboss.org/browse/ISPN-2372) which should already been fixed with version 5.2.0 Beta2 of Infinispan.

       

      Currently we are using Infinispan Version 5.3.0 final, but still we are running out of memory with our application. We have a Java Enterprise Archive Application (EAR) deployed on a glassfish v3 (Oracle GlassFish Server 3.1.2.6 (build 3)) cluster with 2 servers.

      The JDK is jdk1.7.0_25_iaik_5.1.

       

      The application uses Infinispan as an embedded cache solution without any persitent store configurations. As I already mentioned we are constantly running out of memory. We already created some heap dumps where we could find the following in the dominator tree.

      As you can see from the screenshot there are many ThreadLocal Objects that hold refrences to AbstractJBossMarschaller objects which seems to be growing until the server runs out of memory.

      The high memory usage is mostly on the first server where we update a large cache once a day and synchronize it with the 2nd server.

       

      We couldn't find any solution for this problem yet, so maybe you can help us figure it out and fix it?

      Please find attached our Infinispan configuration.

        • 1. Re: OutOfMemory with Infinispan 5.3.0 in glassfish v3 cluster
          dan.berindei

          From your question I understand that you use a single cache and you don't restart it, but from the snapshot you attached, it looks like thread __ejb-thread-pool13 accessed at least 20 different caches. If it would be possible to keep using the same cache, without restarting it, I think that should solve your problem.

           

          Note that ISPN-2372 is only relevant when the cache is restarted often: on restart, a new AbstractJBossMarshaller instance was created, but the thread-local PerThreadInstanceHandler instances were not discarded.

           

          On the other hand, it looks like the fix for ISPN-2372 was indeed incomplete. We now clear the references from the thread-local marshallers to the cache itself, so it can be garbage-collected, but we don't clear the marshallers or their instance caches. Looks like your values' object graph is very large, and the cached marshallers are using up a lot of memory that could be garbage-collected (assuming the cache was indeed restarted and only one cache instance is running now). I've created [ISPN-3659] Cache stop should clear thread-local ExtendedRiverMarshaller or their instance caches to address this.

          • 2. Re: OutOfMemory with Infinispan 5.3.0 in glassfish v3 cluster
            infinispan_user0815

            Thank you for your help and sorry for the late answer. Our application which uses Infinispan as embedded cache solution uses not only one cache but multiple caches (in this case at least 20 different caches). These caches are not restarted so if I understood everything right, then ISPN-2372 is not the same issue that we are facing in our case, right? Do you know any other points that we can check in our application or our implementation where we are using the caches/Infinispan so that we can get a solution for our problem?

            • 3. Re: OutOfMemory with Infinispan 5.3.0 in glassfish v3 cluster
              infinispan_user0815

              Since I still have to analyse and fix the problem, but couldn't find any soltions or further hints, I will ask again if nobody is out there who has probably the same problem or maybe could help me or give me further hints about what I can do to get a solution or find the cause of my issue?

              • 4. Re: OutOfMemory with Infinispan 5.3.0 in glassfish v3 cluster
                hoffmeij

                Hi,

                 

                Did you find a solution for this? Did you figure out if this is a error in infinspan or a design error?

                 

                Im having the exact same problem in my application and im although still using Infinispan Version 5.3.0 final and glassfish 3.1.2.

                 

                Any help with this would be great!!