JBoss AS 6 Hibernate L2C, Infinispan RemoteCacheStore

[matt_filion]

I've been toying with JBoss AS 6 a bit and can get all of my variations of our cache strategies to work except when we need to store the entities in a RemoteCacheStore. For some reason hibernate simply does not call any of the methods on the cache store to load and persist the data.

What I'm basically trying to accomplish is a "Distant" cache for our entities that will be frequently used across all nodes in the cluster. This will reduce the memory footprint of our JBoss instances and reduce the amount of effort spent marshalling/demarshalling inforamtion to and from the database since everything stored in cache is a bit more "native" to the applications environment. This is the cache configuration that I have inside my "infinispan-configs.xml" file in JBoss AS 6 (CR1) but i never see my RemoteCacheStore being utilized.

Has anyone have gone down this route yet or have any knowledge if CacheStores work within the infinispan-configs.xml configurations?

<!-- *************************************** -->

<!-- *          100% Remote Cache          * -->

<!-- // infinispan-configs.xml // -->

<!-- *************************************** -->
<!-- *          100% Remote Cache          * -->
<!-- *************************************** -->
<!-- 
          This configuration passes all cache gets and puts to a remote server and never
          stored locally within the VM.


          passivation="false" Must be set to false otherwise the cache loader 
          will not be used for each transaction, it ill only update the cache store 
          when it triggers a "passivation" event at the end of a transaction or during 
          eviction. 

          shared=true Setting this to true avoids multiple cache instances 
          writing the same modification multiple times. If enabled, only the node where 
          the modification originated will write to the cache store. 

          preload=false 
          If true, when the cache starts, data stored in the cache store will be pre-loaded 
          into memory. You can set this attribute to true if we want to establish some 
          sort of "warm" cache that loads from a remote source.  
-->
<!-- *************************************** -->
<namedCache name="entity-remote">
     <loaders preload="false" shared="true" passivation="false">
          <loader 
               class="org.infinispan.loaders.remote.RemoteCacheStore"
               fetchPersistentState="false" 
               ignoreModifications="false"
               purgeOnStartup="false" 
               purgeSynchronously="false">
               <properties>
                    <property name="hotRodClientPropertiesFile"     value="${jboss.common.base.url}cluster/infinispan-cache-registry.sar/hotrod-client.properties" />
                    <property name="useDefaultRemoteCache" value="false" />
                    <property name="remoteCacheName" value="dist" />
               </properties>
          </loader>
     </loaders>
</namedCache>



<!-- // persistence.xml // -->
<property name="hibernate.transaction.manager_lookup_class" value="org.hibernate.transaction.JBossTransactionManagerLookup"/>
<property name="hibernate.cache.region.factory_class"       value="org.hibernate.cache.infinispan.JndiInfinispanRegionFactory"/>
<property name="hibernate.cache.infinispan.cachemanager"    value="java:CacheManager/entity"/>

<!-- 
     To be able to define custom cache for a specific entity or entities collection use the following.
     When adding new class names to the custom definitions the "name" attribute must start with 
     'hibernate.cache.infinispan' and end with 'cfg'. Its easy to miss the last part. The middle is
     the full package and class name of the entity that is currently managed in cache. 
-->
<property name="hibernate.cache.infinispan.com.model.Entity.cfg"            value="entity-remote"/>
<property name="hibernate.cache.infinispan.com.model.Entity.collection.cfg" value="entity-remote"/>

<!--  Default Configurations for Entities that dont have explicit definitions. -->
<property name="hibernate.cache.infinispan.entity.cfg"      value="entity-remote"/>
<property name="hibernate.cache.infinispan.collection.cfg"  value="entity-remote"/>
<property name="hibernate.cache.infinispan.query.cfg"       value="entity-remote"/>
<property name="hibernate.cache.infinispan.timestamp.cfg"   value="timestamps"/>

<!-- *************************************** -->

<!--

This configuration passes all cache gets and puts to a remote server and never

stored locally within the VM.

passivation="false" Must be set to false otherwise the cache loader

will not be used for each transaction, it ill only update the cache store

when it triggers a "passivation" event at the end of a transaction or during

eviction.

shared=true Setting this to true avoids multiple cache instances

writing the same modification multiple times. If enabled, only the node where

the modification originated will write to the cache store.

preload=false

If true, when the cache starts, data stored in the cache store will be pre-loaded

into memory. You can set this attribute to true if we want to establish some

sort of "warm" cache that loads from a remote source.

-->

<!-- *************************************** -->

<namedCache name="entity-remote">

<loaders preload="false" shared="true" passivation="false">

<loader

class="org.infinispan.loaders.remote.RemoteCacheStore"

fetchPersistentState="false"

ignoreModifications="false"

purgeOnStartup="false"

purgeSynchronously="false">

<properties>

<property name="hotRodClientPropertiesFile"     value="${jboss.common.base.url}cluster/infinispan-cache-registry.sar/hotrod-client.properties" />

<property name="useDefaultRemoteCache" value="false" />

<property name="remoteCacheName" value="dist" />

</properties>

</loader>

</loaders>

</namedCache>

[matt_filion]

I was able to get past this by not using jboss's supplied cache manager and instead defining the configurations in my application. However, i would prefer to offload these kinds of configurations the container so I consider this a work around and not necessarily the right answer to the problem.

<!-- // persistence.xml // -->
<!-- *************************************************** -->
<!--  Use the application provided configurations.       -->
<!-- *************************************************** -->
<property name="hibernate.cache.region.factory_class"  value="org.hibernate.cache.infinispan.InfinispanRegionFactory"/>
<property name="hibernate.cache.infinispan.cfg"        value="ati-infinispan-configs.xml"/>

<property name="hibernate.cache.region.factory_class"  value="org.hibernate.cache.infinispan.InfinispanRegionFactory"/>

<property name="hibernate.cache.infinispan.cfg"        value="ati-infinispan-<!-- // persistence.xml // -->

<property name="hibernate.cache.region.factory_class"  value="org.hibernate.cache.infinispan.InfinispanRegionFactory"/>

<property name="hibernate.cache.infinispan.cfg"        value="app-infinispan-configs.xml"/>configs.xml"/>

[matt_filion]

Had to turn on debug logging on the bootup but figured out that the problem was my hotrod-client.properties file was not getting picked up by jboss and therefore never being configured. I ended up hard coding an absolute path to the \server\all\cluster\infinispan-cache-registry.sar\infinispan-configs.xml and that fixed my issue.

<namedCache name="entity-remote">
     <loaders preload="false" shared="true" passivation="false">
          <loader 
               class="org.infinispan.loaders.remote.RemoteCacheStore"
               fetchPersistentState="false" 
               ignoreModifications="false"
               purgeOnStartup="false" 
               purgeSynchronously="false">
               <properties>
                    <property name="hotRodClientPropertiesFile"     value="/absolute/file/path/hotrod-client.properties" />
                    <property name="useDefaultRemoteCache" value="false" />
                    <property name="remoteCacheName" value="dist" />
               </properties>
          </loader>
     </loaders>
</namedCache>

[galder.zamarreno]

Matt, two things:

1. Did you have to enable debugging to see that the file was not being picked up? According to the code this should be logged at error level, see https://github.com/infinispan/infinispan/blob/4.2.x/cachestore/remote/src/main/java/org/infinispan/loaders/remote/RemoteCacheStoreConfig.java

2. I'm not sure about your use case. The whole point of a 2LC is that is fast to local and it's fast to access, much faster than having to go via TCP/IP to a database. Now, by adding a remote cache store, you're potentially adding further delays cos get() calls that are empty in the local cache now will go remotely to the remote Hot Rod server to double check. So, you're adding unnecessary delays there.

Sure, there're situations when using a cache loader makes sense, for example a cluster cache loader instead of enabling state transfer, but I have doubts about your remote cache store.

   public void setHotRodClientPropertiesFile(String hotRodClientProperties) {

      FileLookup fileLookup = new FileLookup();

      InputStream inputStream = fileLookup.lookupFile(hotRodClientProperties);

      try {

         this.hotRodClientProperties.load(inputStream);

      } catch (IOException e) {

         log.error("Issues while loading properties from file", e);

         throw new CacheException(e);

      }

[matt_filion]

Galder,

1. I didn't see any error messages, this isn't to say that it doesn't exist but I didn't notice it. I turned on debugging and saw it as an info statement saying that it couldn't find it on the classpath and trying to look it up as a file.
2. In some cases we have entities that need to have their state persisted across restarts of the servers, but don't need super fast access. Which is ultimately the perfect fit for a RemoteCacheStore setup like this where it simply gets the data from the RemoteCacheStore in all instances.
   
   We also have a very large collection of entities that we need to be fast and persist across restarts as well. So in these 2 scenarios we have different strategies in place which uses the RemoteCacheStore for a second "tier" of caching when its expired from the in-vm cache. The in-vm cache of course has a much shorter eviction strategy than the remote cache store because the volume of entities that could potentially get loaded over time is enormous. The data grid gives the data the protection we need it to have because with the volume of reads that are done the db gets fairly heavily hit.

The main benefit of having the RemoteCacheStore is that we don't have to have to programatically declare different loading for each entity, its defined by the underlying Level 2 Cache strategy. Gives our engineers a bit less to worry about. Also, since the infinispan server cluster is as close as you declare the server it can be maintained on the same machine and lead to much faster access of entities than a DB since that will always have to live on a remote server.

[galder.zamarreno]

Re 1. An info statement? Hmmm, sounds like it should be pointed rather as an ERROR or WARN message at least. Would you mind opening a jira in https://issues.jboss.org/browse/ISPN with the debugging that you saw and the configuration you used? Thanks

Re 2. The only advantage I could see RemoteCacheStore giving you here is precisely quicker access to those entities as opposed to having Hibernate retrieve it from the database, but then again, I think you'd be losing out in cache queries that return nothing, cos they'd incur a cost of having to go to the RemoteCacheStore and check.

You also have to remember that entity cache, which is communicates synchronously, is configured with state transfer. So, this means that when a server is restarted, it will try to get the entity cache contents from another node in the cluster. So, unless you do full cluster restarts, you shouldn't find yourself in situation where the cache needs to be completely repopulated out of the db. Instead, if you use rolling restart strategies, you might be able to always keep several nodes with entity caches in the cluster, while other nodes get restarted.

I think what you're trying to do is interesting, but I really think you should test both scenarios, first with default configurations provided by the Infinispan 2LC jar, and then with a RCS, and see which one performs better. I don't think the decision is that clear cut.

[matt_filion]

Sorry for the long delay, the holidays and an up comming release at work has left me with little spare time. I'll get to the docs within the next week or so.

[galder.zamarreno]

Added a section using Infinispan as remote second level cache to http://community.jboss.org/docs/DOC-14105