1 Reply Latest reply on Jul 8, 2015 12:02 PM by Carolina Contiu

    Memory leak with 5.3.0 Final

    Carolina Contiu Newbie

      We are using an Infinispan Standalone Server version 5.3.0-Final.

       

      The memory setting are:

      -Xms2048m -Xmx2048m -XX:MaxPermSize=256m

       

      The client application is using  hotrod to communicate with the Infinispan Server.

       

      We are experiencing a slow memory leak. The old gen displays an increasing trend: the lower limit increases after each major collection.

      After around 2 months of up-time the old gen reaches the cap  and fails to go down after any major collections

      (resulting in high CPU,  etc. ) and imposing a restart.

       

      This server is hosting around 10 caches, with at most 5000 keys each.  The overall size of the information held in this

      caches should not exceed several megs.

       

      We managed to perform a heap dump when the old gen was 1.2G out of 1.4G and the offender seems to be:

       

      org.infinispan.util.concurrent.locks.containers.ReentrantPerEntryLockContainer

       

      Please check the attached image.

       

      InfinispanMem.JPG

      Our caches are defined in one of the following manners:

       

                      <local-cache name="Cache1" start="EAGER">

                           <locking isolation="NONE" acquire-timeout="30000" concurrency-level="1000" striping="false"/>

                           <eviction strategy="LRU" max-entries="2000"/>

                           <expiration max-idle="60000"/>

                           <transaction mode="NONE"/>

                     </local-cache>

                      <local-cache name="Cache2" start="EAGER">

                           <locking isolation="NONE" acquire-timeout="30000" concurrency-level="1000" striping="false"/>

                           <transaction mode="NONE"/>

                      </local-cache>

                     <local-cache name="Cache3" start="EAGER">

                           <locking isolation="NONE" acquire-timeout="30000" concurrency-level="1000" striping="false"/>           

                           <transaction mode="NONE"/>

                           <string-keyed-jdbc-store datasource="java:jboss/datasources/JdbcDS" passivation="false" preload="true" purge="false">

                               <property name="databaseType">ORACLE</property>

                               <property name="fetchPersistentState">true</property>

       

                               <string-keyed-table prefix="IS6">

                                   <id-column name="id" type="VARCHAR2(500)"/>

                                   <data-column name="datum" type="BLOB"/>

                                   <timestamp-column name="version" type="NUMBER(19)"/>

                               </string-keyed-table>                   

                  </string-keyed-jdbc-store>

                       </local-cache>

       

       

       

      Does anyone experienced a similar issue? Is there a know scenario/or misconfiguration on our side that could

      lead to this problem?

       

      I would appreciate any advice/hints.