Infinispan cluster with shared file cache store.

[c0ff75]

Hello everybody!

I want to create Infinispan small cluster with 3-4 nodes. This cluster should service simple but big cache between database and web-clients.

I don't want to lose cached data  after reboot of cluster. It means that cache store is needed for data persistense. But which type of cache store is better in my case?

My wishes:

1. I don't want create difficult design with separate database with access over JDBC. Database in this case - single point of failure. And I should spend resources for run it.

2. All nodes in my cluster must be same. I can stop/start their in any time.

May be good way is create a shared file system for all nodes and just place on it files of cache store? But which file system store can be used in my case?  This is not easy choiche:

1. Single File Store can be fragmented and have no tools for compaction

2. Soft-Index File Store is experimental

3. LevelDB can't work in shared mode from separate processes (nodes)

4. BerkeleyDB isn't supported after version 5.3 of Infinispan

People, how you build Infinispan clusters with cache data store?

Best regards,

Eugeny

[nadirx]

Hi Eugeny, none of the file-based stores can be configured as shared, so you really need to rely on one of: JDBC, Cassandra, Redis.

[c0ff75]

Hi, Tristan Tarrant

If I will use any JDBC cache store then I will create single point of failure in my configuration. Plus I should spend additional hardware for running some database (processors and RAM). It isn't good.

My current Infinispan cluster configuration may be isn't beatyful, but it works:

- each node have own file store (LevelDB) in separate disk folder

- all nodes synchronized via JGroups

Good sides of this solution:

- I spend only disk space, but today this isn't expensive

- Each node of my configuration is exactly same as others. I can copy it, start or stops in any time.

Best regards,

Eugeny

[wdfink]

Bad side of the solution:

- node crashed >= numOwner for DIST caches and you loose data

- complete cluster shutdown must be done carefully to not loose data or have stale data / inconsistent cluster nodes

This has been solved by "graceful shutdown" in later Infinispan versions

[c0ff75]

Hi, Wolf-Dieter Fink !

Thanks for answer. It has very useful information for me!

> - node crashed >= numOwner for DIST caches and you loose data

I have no distributed caches in my configuration, I use only replicated caches.

> - complete cluster shutdown must be done carefully to not loose data or have stale data / inconsistent cluster nodes

Thanks. I have configured persistency of global state now.  All my cluster nodes uses same directory for it:

        <global-state>

            <persistent-location path="/mnt/shared/cache_global" />

            <temporary-location path="/mnt/shared/cache_global/tmp"/>               

        </global-state>

Is it correct? In documentation I can't find exact explanation about it. I think that global state must be shared between all nodes.

Best regards,

Eugeny

[wdfink]

FileStores are not designed to be shared. You can't sync the access to the file (only lock the complete file).

So use the same file does not mean you have a correct working shared persistence. As Tristan said you need to have a persistence which can handle this correctly

[nadirx]

Graceful shutdown is not merged yet, but will be soon.