Clustered WarmCache with Persistence

[brenuart]

Dear all,

We'd like to make use of Infinispan as our data grid solution. Our (first) requirements are:

- cluster of at least two nodes to provide high availability

- local persistence on each node to survive to a restart of all nodes.

Basically, this setup should be seen as a shared persistent memory - it should behave "as if" a database would be used instead...

I came to a configuration that seems to match our needs. However, I'd like to post it here to gather feedback from others with more experience on the subject than me...

This post starts with a description of my understanding of some of the Infinispan configuration options. Please, tell me if I'm correct...

Then follows our sample configuration.

Don't hesitate to comment and share your experience on similar configuration!

Thanks a lot.

/bertrand

CacheLoader configuration options

First of all, I want to make sure I understand correctly some of the different configuration options described in http://community.jboss.org/wiki/CacheLoaders (Configuration).

<loaders preload=”true”>

Enable this option in order to instruct the cache to populate itself with content provided by the declared cache loaders at startup.

If not enabled, the cache will start empty and cache loaders will be consulted only on cache miss (request for a key not already in the cache).

This option is disabled by default.

Question: what happens if a ClusterCacheLoader is configured ?

Apparently - nothing, a ClusterCacheLoader cannot be used to preload the cache…

{color:green}Is this correct?{color}

<loader fetchPersistentState="true">

If I understand correctly, this option influences how the cache is preloaded (warm cache) when:

1. the cache is started and is the only cluster member;
2. the cache is started and joins an already established cluster

This option is set per CacheLoader. It is used to indicate whether or not content from this CacheLoader should be used to populate the cache when joining an existing cluster (scenario 2 above).

{color:green}Is this correct?{color}

<loader purgeOnStartup="false">

This option empties the cache loader on startup.

If I understand correctly, the content is purged when the cache loader itself is started. Consequence is it cannot be used to preload the cache anymore... This option seems to be incompatible with a “warm cache” configuration.

The only usage scenario that comes to my mind is when passivation and ignoreModifications are enabled. The local storage is used to offload the memory only, kind of swap feature. When the cache is started, you cleanup your swap first.

{color:green}Is this correct?{color}

Clustered WarmCache with Persistence

Here is our sample configuration - partial - only the part relevant to CacheLoaders and Clustering.

What do you think of it ?

<clustering mode="replicated">

  <sync />

    <!--

       When joining an existing cluster, the cache state will be initialized from other cluster

       members.

     -->

    <stateRetrieval

            timeout="20000"

       fetchInMemoryState="true"

       alwaysProvideInMemoryState="false"

    />

</clustering>

<!--

  preload=true

    warm-cache-> attempt to preload cache content from CacheLoaders at startup.

->

<loaders preload="true" >

  <!--

    fetchPersistentState=false

      This property controls the behavior of the CacheLoader when the cache joins anexisting node (ie.

      when not the first member of the cluster).

      In this case, setting the value to false prevents from preloading the cache withthe local content.

      On theother hand, we rely on the stateTransfer feature to warm our cache and synchronize it with the

      rest of the cluster.

    purgeOnStartup=false

      Set to false otherwise persisted content will be deleted before the cache is preloaded !

    ignoreModifications=false

      We want the entire cache content to be mirrored on disk. The FileCacheStore must

      take modifications into account and write them on disk.

  -->

  <loader class="org.infinispan.loaders.file.FileCacheStore"

    fetchPersistentState   ="false"

    purgeOnStartup         ="false"

    ignoreModifications    ="false"

  >

    <properties>

      <property name="location"value="${data.dir}/${cliid}"/>

    </properties>

    <!--

       write-behind

          Updates to the cache are asynchronously written to the cache store. This way the

          performance of a cache operation does not get affected by the update of the underlying

          store. On the other hand, since the update happens asynchronously, there's a time window

          during the which the cache store can contain stale data compared to the cache.

    -->

    <async enabled="true"threadPoolSize="10"/>

  </loader>

</loaders>

[mircea.markus]

1. ClusterCacheLoader

yes you are right. I've also updated Cache loader doc with that.

2. FetchPersistentState.

yes, you are right.

3. you can have both purge on startup and fetchPersistentState enabled on one cache in order to warm it up with data stored in the cache loader of a remote cache. 

[mircea.markus]

regarding the configuration, looks good to me.

[brenuart]

3. you can have both purge on startup and fetchPersistentState enabled on one cache in order to warm it up with data stored in the cache loader of a remote cache.

If you purgeOnStartup it means you discard everything you have persisted on node restart (whether it joins an existing cluster or creates a newone).

Ultimately, what's the point in persisting anything ?

[galder.zamarreno]

The only reason I could think this might be useful is if you have one cache with a shared cache loader, and you have N caches that have a local cache store. This could happen because only one node has access to a remote database for example. In this case, the other N caches, when restarted, they might wanna get the persisted data, and fetch persistent state basically brings that to their local cache stores.

fetchPersistentState does not force the persistent store to be cleared before applying the state (this could be expensive to do), and so purgeOnStartup on these nodes would make sure no leftover, or data removed by the node that has access to the shared cache store remains in the local cache store.

I agree that this is an edge case and not the usual set up, but could make sense.