1 2 Previous Next 16 Replies Latest reply on Sep 4, 2013 2:20 PM by mlinhard

    Infinispan 5.2.1:  Very slow memory leak

    kevinhermann

      Hi,

       

      At my company we recently upgraded from our 1st Infinispan version (5.1.6) to version (5.2.1).  We use Infinispan in the capacity of a Hotrod client-server usage and we use Infinispan as a remote cache cluster.  We support a total of 9 caches and 6 of the 9 caches are not persistent - they are distributed using the following syntax in the XML configuration file.  We find that from a day-to-day basis there is a small memory leak and the heap space available on the Infinispan JVM servers is little-by-little growing.  This means that the objects on the heap are not being garbage collected approximately equal to the rate the are created over tine.   Please help us uncover why this could be occurring and the resolution to solve this problem. 

       

      Details:

       

      <namedCache name="ourCacheName">

            <expiration

               wakeUpInterval="500"

               reaperEnabled="true"

               lifespan="-1"

               maxIdle="3600000"

            />

       

            <clustering mode="distribution">

               <sync/>

               <hash numOwners="2"/>

            </clustering>

      </namedCache>

       

      The 3 remaining caches are "persistent" and we use the following XML format for all 3:

       

      <namedCache name="ourCache">

            <loaders passivation="false" shared="false" preload="true">

               <loader class="org.infinispan.loaders.file.FileCacheStore"

                  fetchPersistentState="true"

                  ignoreModifications="false"

                  purgeOnStartup="false">

                  <properties>

                     <property name="location" value="${infinispan.data.dir}"/>

                  </properties>

               </loader>

            </loaders>

            <clustering mode="distribution">

               <sync/>

               <hash numOwners="2"/>

            </clustering>

         </namedCache>

       

      Overall there are not 2 many entries in the non-persistent caches.  In the persistent caches we do have 1 case where the cache has about 43,000 entries and they are loaded as static data (the hit ration is typically 100%).  

       

      Infinispan Graph - JVM Heap Memory Usage.png

       

       

      When we restart the Infinispan servers in our production environment the JVM Heap space is restored.  We have 2 separate servers dedicated for Infinispan and there are 2 in production.  Our cache hit rations are generally 90%+ and most of the caches have a hit ration of 99% - 100%. 

       

      However, as a retail company, our traffic will increase immensely in November and December.  So we need to figure out the problem as soon as possible and correct it.   

       

      Can you help us narrow down how the memory leak could potentially be occurring?  We are using JBoss 5.1.0 and our Infinispan server cluster (2 of them) sit on separate physical machines.   

       

      Here are images of the heap dumps as analyzed by Visual VM:

       

      Screenshot 1 of 2: The highest occurring # of instances on the heap by % and #:

       

      Visual VM - heap dump containing largest number of instances on the heap.png

       

      Screenshot 2 of 2:  Instances that are consuming the most amount of space on the heap. 

       

      Visual VM - heap dump containing instances taking up the most amount of space on the heap.png

        1 2 Previous Next