1 2 Previous Next 17 Replies Latest reply on Nov 28, 2011 6:15 PM by _rob_jboss

    Infinispan not evicting entities from named cache

    _rob_jboss

      I have recently migrated from JBoss 4.2 to 6 and as a result from JBoss Cache to Infinispan.  I followed this article - https://docs.jboss.org/author/display/ISPN/Using+Infinispan+as+JPA-Hibernate+Second+Level+Cache+Provider to set caching up. 

       

      I'm getting problems when it comes to infinispan evicting/updating cached entities.  This is best described in steps:

       

      1) Request is made for all incident entities

      2) Incidents are placed into the entity-repeatable cache (NumberOfEntries=17 in the JMX infinispan stats)

      3) One query is placed into the local-query cache (NumberOfEntries=1 in the JMX infinispan stats)

      4) Another request is made for all incident entities (which are retrived from the cache)

      5) 5 seconds pass and the entities are still in the cache (according to the JMX infinispan stats)

      6) 10 seconds pass and the query is removed from the cache (according to the JMX infinispan stats)

      7) All incidents are updated within the database

      8) Another request is made for all incident entities

      9) New incidents are returned.

      10) Another request is _immediately_ made and the old incidents are returned. (Hits=17 in the JMX infinispan stats)

       

      The new incidents are only returned after the cached query is evicted and therefore a call is made to the database.

       

      No matter what value I define for the expiration life span (under the named cache entity-repeatable), the entities are never evicted.  At the same time the named cache local-query evicts queries as expected.  Weirdly, if I place the entities in the local-query cache, neither the entities nor the queries are evicted!

       

      I'm using:

       

      Java 1.6

      JBoss-6.0.0.Final

      Hibernate 3.6.0.Final

      Hibernate Commons Annotations 3.2.0.Final

      Infinispan 4.2.0.FINAL

      Single JBoss node running on Windows XP Pro

       

      My questions are:

       

      1) Why aren't the entitles being evicted/updated?

      2) Is it possible to turn on extra infinispan logging to see what evictions are taking place, if any?

       

      Any help in resolving this would be very much appreciated.

       

      Thanks,

       

      Rob.

       

      Persistence.xml

       

      <?xml version="1.0" encoding="UTF-8"?>

      <persistence xmlns="http://java.sun.com/xml/ns/persistence"

                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                   xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd"

                   version="2.0">

        <persistence-unit name="PersistenceUnit" transaction-type="JTA">

          <provider>org.hibernate.ejb.HibernatePersistence</provider>

          <jta-data-source>java:/MSSQLDS</jta-data-source>

          <jar-file>Project6-ejb.jar</jar-file>

          <properties>

            <property name="hibernate.cache.use_second_level_cache" value="true"/>

            <property name="hibernate.cache.use_query_cache" value="true"/>

            <property name="hibernate.cache.region.factory_class" value="org.hibernate.cache.infinispan.JndiInfinispanRegionFactory"/>

            <property name="hibernate.cache.infinispan.cachemanager" value="java:CacheManager/entity"/>

            <property name="hibernate.cache.region_prefix" value="Project6"/>

            <property name="hibernate.show_sql" value="false"/>

            <property name="hibernate.dialect" value="org.hibernate.dialect.SQLServer2008Dialect"/>

            <property name="hibernate.cache.infinispan.statistics" value="true"/>

          </properties>

        </persistence-unit>

      </persistence>

       

      Entity example with cache region references

       

      @Entity

      @Table(name = "Incident")

      @NamedQueries({

          @NamedQuery(name = "Incident.findAll",

          query = "SELECT i FROM Incident i",

          hints = {

              @QueryHint(name = "org.hibernate.cacheable", value = "true"),

              @QueryHint(name = "org.hibernate.cacheRegion", value = "local-query")})

      })

      @Cache(usage = CacheConcurrencyStrategy.READ_ONLY, region = "entity-repeatable")

      public class Incident implements Serializable {

       

        // usual entity code here

      }

       

      infinispan-config.xml (hibernate part only)

       

        <!-- Hibernate second-level cache -->

        <infinispan-config name="hibernate" jndi-name="java:CacheManager/entity">

          <infinispan xmlns="urn:infinispan:config:4.2">

            <global>

              <transport clusterName="${jboss.partition.name:DefaultPartition}-Hibernate" distributedSyncTimeout="17500">

                <properties>

                  <property name="stack" value="${jboss.default.jgroups.stack:udp}"/>

                </properties>

              </transport>

              <globalJmxStatistics enabled="true"/>

              <shutdown hookBehavior="DONT_REGISTER"/>

            </global>

            <default>

              <!-- Used to register JMX statistics in any available MBean server -->

              <jmxStatistics enabled="true"/>

              <!--transaction transactionManagerLookupClass="org.infinispan.transaction.lookup.JBossTransactionManagerLookup"/-->

            </default>

           

            <!-- Default configuration is appropriate for entity/collection caching. -->

            <namedCache name="entity-repeatable">

              <clustering mode="invalidation">

                <stateRetrieval fetchInMemoryState="false" timeout="20000"/>

                <sync replTimeout="20000"/>

              </clustering>

              <!-- Note: REPEATABLE_READ is only useful if the application evicts/clears entities

                from the Hibernate Session and then expects to repeatably re-read them in

                the same transaction. Otherwise, the Session's internal cache provides a

                repeatable-read semantic. Before choosing this config, carefully read the docs

                and make sure you really need REPEATABLE_READ.

               -->

              <locking isolationLevel="REPEATABLE_READ" concurrencyLevel="1000"

                       lockAcquisitionTimeout="15000" useLockStriping="false"/>

              <!-- Eviction configuration.  WakeupInterval defines how often the eviction thread runs, in milliseconds. 

                   0 means the eviction thread will never run.  A separate executor is used for eviction in each cache. -->

              <eviction wakeUpInterval="1000" maxEntries="10000" strategy="FIFO"/>

              <expiration maxIdle="-1" lifespan="5000"/>

              <lazyDeserialization enabled="true"/>

            </namedCache>

           

           

            <!-- A config appropriate for query caching.  Does not replicate queries. -->

            <namedCache name="local-query"><!-- Non-replicating query cache -->

              <locking isolationLevel="READ_COMMITTED" concurrencyLevel="1000"

                       lockAcquisitionTimeout="15000" useLockStriping="false"/>

              <!--Eviction configuration.  WakeupInterval defines how often the eviction thread runs, in milliseconds.  0 means

                 the eviction thread will never run.  A separate executor is used for eviction in each cache. -->

              <eviction wakeUpInterval="1000" maxEntries="10000" strategy="FIFO"/>

              <expiration maxIdle="-1" lifespan="10000"/>

            </namedCache>

       

           

            <!-- Optimized for timestamp caching. -->

            <!-- A clustered timestamp cache is required if query caching is used, even if the query cache itself is configured with CacheMode=LOCAL. -->

            <namedCache name="timestamps">

              <clustering mode="replication">

                <stateRetrieval fetchInMemoryState="true" timeout="20000"/>

                <async/>

              </clustering>

              <locking isolationLevel="READ_COMMITTED" concurrencyLevel="1000"

                       lockAcquisitionTimeout="15000" useLockStriping="false"/>

              <!--  Don't ever evict modification timestamps -->

              <eviction wakeUpInterval="0" strategy="NONE"/>

              <lazyDeserialization enabled="true"/>

            </namedCache>

       

       

          </infinispan>

        </infinispan-config>

        • 1. Re: Infinispan not evicting entities from named cache
          _rob_jboss

          Update:

           

          I have added the following two lines to the <default> part of the infinispan-configs.xml file:

           


          <eviction wakeUpInterval="1000" maxEntries="10000" strategy="FIFO"/>

          <expiration maxIdle="-1" lifespan="5000"/>

           

          so the following is now in place:

           


          <default>

          <!-- Used to register JMX statistics in any available MBean server -->

          <jmxStatistics enabled="true"/>

          <!--transaction transactionManagerLookupClass="org.infinispan.transaction.lookup.JBossTransactionManagerLookup"/-->

          <eviction wakeUpInterval="1000" maxEntries="10000" strategy="FIFO"/>

          <expiration maxIdle="-1" lifespan="5000"/>

          </default>

           

          This evicts the entities after five seconds!

           

          My questions now are:

           

          1) Why are these entities being cached under entity-repeatable but being removed by the eviction defined under default?

          2) Why doesn't the eviction defined under default remove the queries being cached under local-query?

           

          Thanks,

           

          Rob.

          • 2. Re: Infinispan not evicting entities from named cache
            _rob_jboss

            After further reading:

             

            1) Because all named caches inherite from default.

            2) Because the named cache overrides the default eviction/expiration settings.

             

            So, why doesn't the eviction/expiration defined in entity-repeatable override the default eviction setting as in local-query?

             

            Thanks,

             

            Rob.

            • 3. Re: Infinispan not evicting entities from named cache
              galder.zamarreno

              Because there's no link between named caches. Named caches inherit from default cache, but not from other named caches.

              • 4. Re: Infinispan not evicting entities from named cache
                galder.zamarreno

                Btw, if you want to do is set eviction/expiry settings for all entities or per entity type, it's much easier to do it via the Hibernate properties as explained in: https://docs.jboss.org/author/display/ISPN/Using+Infinispan+as+JPA-Hibernate+Second+Level+Cache+Provider#UsingInfinispanasJPA-HibernateSecondLevelCacheProvider-AdvancedConfiguration

                • 5. Re: Infinispan not evicting entities from named cache
                  _rob_jboss

                  Hi Galder and thanks for your response.

                   

                  I understand that name queries can only inherit from the default cache, my question wasn't very clear apologies for that.

                   

                  I still do not understand however, why the expiration does not work for entity-repeatable, but does work for local-query.  When testing, queries are evicted from local-query after 10 seconds, but the entities are never evicted for entity-repeatable even though they should be evicted after 5 seconds.

                   

                  Ultimately, I would like to put the entities in to four different named caches which will have lifespans of 1 minute, 5 minutes, 1 hour and 10 hours.  Once I have a single named cache evicting entities, I will then create the others.

                   

                  There are quite a few entities, so defining an eviction policy for each would be overkill.  However, in the interest of getting this working I’ll define the eviction/expiry settings in the hibernate properties and see where I get with that.

                   

                  Thanks,

                   

                  Rob.

                  • 6. Re: Infinispan not evicting entities from named cache
                    _rob_jboss

                    I have tested the following, all of which do not evict entities:

                     

                    1) Assigned incident to named cache

                     

                    persistence.xml

                     

                    <property name="hibernate.cache.infinispan.com.acme.Incident.cfg" value="incident-entity"/>

                     

                    infinispan-configs.xml

                     

                    <namedCache name="incident-entity">

                    <clustering mode="invalidation">

                      <stateRetrieval fetchInMemoryState="false" timeout="20000"/>

                      <sync replTimeout="20000"/>

                    </clustering>

                    <locking isolationLevel="REPEATABLE_READ" concurrencyLevel="1000"

                         lockAcquisitionTimeout="15000" useLockStriping="false"/>

                    <eviction wakeUpInterval="1000" maxEntries="10000" strategy="FIFO"/>

                    <expiration maxIdle="-1" lifespan="5000"/>

                    <lazyDeserialization enabled="true"/>

                    </namedCache>

                     

                    2) Assigned incident to named cache and set eviction in persistence.xml

                     

                    persistence.xml

                     

                    <property name="hibernate.cache.infinispan.com.acme.Incident.cfg" value="incident-entity"/>

                    <property name="hibernate.cache.infinispan.incident-entity.eviction.strategy" value= "FIFO"/>

                    <property name="hibernate.cache.infinispan.incident-entity.eviction.wake_up_interval" value= "1000"/>

                    <property name="hibernate.cache.infinispan.incident-entity.eviction.max_entries" value= "10000"/>

                    <property name="hibernate.cache.infinispan.incident-entity.expiration.lifespan" value= "5000"/>

                    <property name="hibernate.cache.infinispan.incident-entity.expiration.max_idle" value= "-1"/>

                     

                    infinispan-configs.xml

                     

                    <namedCache name="incident-entity">

                    <clustering mode="invalidation">

                      <stateRetrieval fetchInMemoryState="false" timeout="20000"/>

                      <sync replTimeout="20000"/>

                    </clustering>

                    <locking isolationLevel="REPEATABLE_READ" concurrencyLevel="1000"

                         lockAcquisitionTimeout="15000" useLockStriping="false"/>

                    <eviction wakeUpInterval="1000" maxEntries="10000" strategy="FIFO"/>

                    <expiration maxIdle="-1" lifespan="5000"/>

                    <lazyDeserialization enabled="true"/>

                    </namedCache>

                     

                    3) Created named cache and set eviction in persistence.xml

                     

                    persistence.xml

                     

                    <property name="hibernate.cache.infinispan.com.acme.Incident.eviction.strategy" value= "FIFO"/>

                    <property name="hibernate.cache.infinispan.com.acme.Incident.eviction.wake_up_interval" value= "1000"/>

                    <property name="hibernate.cache.infinispan.com.acme.Incident.eviction.max_entries" value= "10000"/>

                    <property name="hibernate.cache.infinispan.com.acme.Incident.expiration.lifespan" value= "1000"/>

                    <property name="hibernate.cache.infinispan.com.acme.Incident.expiration.max_idle" value= "-1"/>

                     

                    JMX Console

                     

                    I can view the stats under the named cache Project6.com.acme.Incident(local).

                     

                    Conclusion:

                     

                    It is very strange how queries are cleared from the cache, but entities are not.  Could this in some way be related to Hibernate not making a call to Infinispan to evict entities? 

                     

                    Is eviction solely down to Infinispan to manage or JBoss/Hibernate to make a call to Infinispan to initiate eviction?

                     

                    Thanks,

                     

                    Rob.

                    • 7. Re: Infinispan not evicting entities from named cache
                      galder.zamarreno

                      Hmm, that looks odd. At first glance I don't see anything wrong with that configuration. What's the fullly qualified name of your entity, is it Project6.com.acme.Incident or com.acme.Incident?

                       

                      I'd suggest you repeat the test for one of those configurations, for example the first one, and enable TRACE on org.infinispan and org.hibernate.infinispan packages and you attach the log here.

                      • 8. Re: Infinispan not evicting entities from named cache
                        _rob_jboss

                        Thanks for your response.

                         

                        The fully qualified name of my entity is com.acme.Incident.  I have tried included the region prefix (Project6) in the named cache, so Project6.com.acme.Incident instead of com.acme.Incident, but unfortunately with no luck.

                         

                        The latest changes are as follows:

                         

                        1) Updated logging (deploy\jboss-logging.xml) to include the following

                         

                        jboss-logging.xml

                         

                        <logger category="org.infinispan">

                          <level name="TRACE" />

                        </logger>

                         

                        <logger category="org.hibernate.infinispan">

                          <level name="TRACE" />

                        </logger>

                         

                        2) Assigned incident to named cache

                         

                        persistence.xml

                         

                        <property name="hibernate.cache.infinispan.com.acme.Incident.cfg" value= "incident-entity"/>

                         

                        infinispan-configs.xml

                         

                        <namedCache name="incident-entity">

                         

                        <clustering mode="invalidation">

                         

                          <stateRetrieval fetchInMemoryState="false" timeout="20000"/>

                         

                          <sync replTimeout="20000"/>

                         

                        </clustering>

                         

                        <locking isolationLevel="REPEATABLE_READ" concurrencyLevel="1000"

                         

                             lockAcquisitionTimeout="15000" useLockStriping="false"/>

                         

                        <eviction wakeUpInterval="1000" maxEntries="10000" strategy="FIFO"/>

                         

                        <expiration maxIdle="-1" lifespan="5000"/>

                         

                        <lazyDeserialization enabled="true"/>

                         

                        </namedCache>

                         

                        3) Logging attached

                         

                        Conclusion:

                         

                        Unexpectedly, I can see no output from org.hibernate.infinispan.  I can see a few references to local-query (with the region prefix Project6, so Project6.local-query), but I see no references to incident-entity.  I can also see a reference from hibernate adding entity to second level cache, but with no reference to which named cache it is adding it to.  Referring to the JMX console, the entities have appeared under Project6.com.acme.Incident(local) and not incident-entity as I would have expected.  The incident-entity named cache status is defined as 'not created'

                         

                        Thanks,

                         

                        Rob.

                         


                        • 9. Re: Infinispan not evicting entities from named cache
                          galder.zamarreno

                          *Attaching* a log is a more sensible option than pasting - use the advanced editor to do so

                          • 10. Re: Infinispan not evicting entities from named cache
                            _rob_jboss

                            Apologies.  I have amended the original post accordingly.

                             

                            Rob.

                            • 11. Re: Infinispan not evicting entities from named cache
                              galder.zamarreno

                              Hmmm, I can see how the entity cache is not been given the right lifespan settings. I'm not sure about the cause of it though.

                               

                              Couple of suggestions:

                               

                              1. This could be some sort of bug that we might have fixed, so I'd suggest trying to with the latest Hibernate 3.6.x version which is 3.6.8.Final, That comes with Infinispan 4.2.1.CR1, but you can safely use 4.2.1.Final.

                               

                              2. If that's not working, repeat the test and provide TRACE on org.hibernate.cache.infinispan package too (I made a mistake earlier, it should have been org.hibernate.cache.infinispan instead of org.hibernate.infinispan)

                              • 12. Re: Infinispan not evicting entities from named cache
                                _rob_jboss

                                I downloaded infinispan-4.2.1.FINAL (http://www.jboss.org/infinispan/downloads), but it only contained hibernate-core-3.5.4-Final.jar.  So, I downloaded it from http://sourceforge.net/projects/hibernate/files/hibernate3/3.6.8.Final/ instead.

                                 

                                I then removed common/lib/hibernate-core.jar and replaced with hibernate3.jar (3.6.8.Final).  On starting JBoss I can see the following line in the logs confirming this:

                                 

                                INFO  [org.hibernate.cfg.Environment] (Thread-2) Hibernate 3.6.8.Final

                                 

                                I assume this is the only file that needs to be replaced?

                                 

                                The logging in deploy\jboss-logging.xml has been updated to the following:

                                 

                                <logger category="org.infinispan">

                                  <level name="TRACE" />

                                </logger>

                                 

                                <logger category="org.hibernate.cache.infinispan">

                                  <level name="TRACE" />

                                </logger>

                                 

                                I then rerun the test:

                                 

                                1) Assigned incident to named cache (test1-trace.log attached)

                                 

                                persistence.xml

                                 

                                <property name="hibernate.cache.infinispan.com.acme.Incident.cfg" value= "incident-entity"/>

                                 

                                infinispan-configs.xml

                                 

                                <namedCache name="incident-entity">

                                <clustering mode="invalidation">

                                  <stateRetrieval fetchInMemoryState="false" timeout="20000"/>

                                  <sync replTimeout="20000"/>

                                </clustering>

                                <locking isolationLevel="REPEATABLE_READ" concurrencyLevel="1000"

                                     lockAcquisitionTimeout="15000" useLockStriping="false"/>

                                <eviction wakeUpInterval="1000" maxEntries="10000" strategy="FIFO"/>

                                <expiration maxIdle="-1" lifespan="5000"/>

                                <lazyDeserialization enabled="true"/>

                                </namedCache>

                                 

                                Outcome:

                                 

                                No attempt is made to assign the Incident to the incident-entity named cache.  In fact, it creates the named cache "Project6.com.acme.Incident(local)" instead and assigns a lifespan of -1:

                                 

                                2011-11-15 22:57:36,306 TRACE [org.infinispan.interceptors.InvocationContextInterceptor] (WorkerThread#0[192.168.199.1:3235]) Invoked with command PutKeyValueCommand{key=com.acme.Incident#a317, value=CacheEntry(com.acme.Incident)[,2011-11-14 00:00:00.0,,,Northbound,,null,,null,null,null,null,null,1,Accident,2011-11-14 09:54:23.717,49.766827,-7.557148,London Test,1,2,,null,null,null,null,aid,2011-11-14 09:53:00.0,null], putIfAbsent=true, lifespanMillis=-1, maxIdleTimeMillis=-1} and InvocationContext [NonTxInvocationContext{flags=[ZERO_LOCK_ACQUISITION_TIMEOUT, FORCE_ASYNCHRONOUS, FAIL_SILENTLY, PUT_FOR_EXTERNAL_READ]}]

                                 

                                Also from the logs I can see that the local-query named cache is created and assign the correct lifespan:

                                 

                                2011-11-15 22:57:35,415 INFO  [org.jboss.ha.ispn.DefaultCacheContainerFactory] (WorkerThread#0[192.168.199.1:3235]) Started "Project6.local-query" cache from "hibernate" container

                                2011-11-15 22:57:35,415 TRACE [org.infinispan.interceptors.InvocationContextInterceptor] (WorkerThread#0[192.168.199.1:3235]) Invoked with command PutKeyValueCommand{key=org.hibernate.cache.infinispan.util.CacheHelper$EvictAll@1bea7076, value=INIT, putIfAbsent=false, lifespanMillis=5000, maxIdleTimeMillis=-1} and InvocationContext [NonTxInvocationContext{flags=[CACHE_MODE_LOCAL]}]

                                2011-11-15 22:57:35,415 TRACE [org.infinispan.interceptors.CallInterceptor] (WorkerThread#0[192.168.199.1:3235]) Executing command: PutKeyValueCommand{key=org.hibernate.cache.infinispan.util.CacheHelper$EvictAll@1bea7076, value=INIT, putIfAbsent=false, lifespanMillis=5000, maxIdleTimeMillis=-1}.

                                 

                                2) Create named cache in persistence.xml and assign entity to this region (test2-trace.log attached)

                                 

                                persistence.xml

                                 

                                <property name="hibernate.cache.infinispan.RegionA.eviction.strategy" value= "FIFO"/>

                                <property name="hibernate.cache.infinispan.RegionA.eviction.wake_up_interval" value= "1000"/>

                                <property name="hibernate.cache.infinispan.RegionA.eviction.max_entries" value= "5500"/>

                                <property name="hibernate.cache.infinispan.RegionA.expiration.lifespan" value= "5000"/>

                                <property name="hibernate.cache.infinispan.RegionA.expiration.max_idle" value= "-1"/>

                                 

                                com.acme.Incident

                                 

                                @Entity

                                @Table(name = "Incident")

                                @NamedQueries({

                                  @NamedQuery(name = "Incident.findAll",

                                  query = "SELECT i FROM Incident i where i.incidentId = 'a317'",

                                  hints = {

                                    @QueryHint(name = "org.hibernate.cacheable", value = "true"),

                                    @QueryHint(name = "org.hibernate.cacheRegion", value = "local-query")})

                                })

                                @Cache(usage = CacheConcurrencyStrategy.READ_ONLY, region="RegionA")

                                public class Incident implements Serializable {

                                  // usual code here

                                }

                                 

                                Outcome:

                                 

                                The named cache (RegionA) is created and the entities are placed into it, but are not evicted after 5 seconds.

                                 

                                Conclusion:

                                 

                                It appears that the approach used to setup query named caches and to evict queries from these caches works and possibly needs to be replicated for entity named caches.

                                • 13. Re: Infinispan not evicting entities from named cache
                                  _rob_jboss

                                  Hi Galder,

                                   

                                  As a final test I downloaded JBoss 6 again and replaced the currently installed version.  Unfortunately the problem was still present.  If possible can you try a similar test on your development environment to see if this can be replicated so that we may move forward?  Due to client constraints, we are unavailable at this time to migrate to JBoss 7 so need a fix/work around for JBoss 6.

                                   

                                  Thanks very much,

                                   

                                  Rob.

                                  • 14. Re: Infinispan not evicting entities from named cache
                                    galder.zamarreno

                                    Rob, can you try something? Can you set CacheConcurrencyStrategy to CacheConcurrencyStrategy.TRANSACTIONAL ?

                                     

                                    How are you testing this read only scenario? I mean, you have a database preloaded with entities and trying to read from there?

                                     

                                    I've tried to replicate a similar scenario but the lifespan is picked correctly, but I was using a transactional cache. Trying to find out what might be different between my case and yours.

                                    1 2 Previous Next