1 Reply Latest reply on May 13, 2014 8:49 AM by sannegrinovero

Infinispan as datomic storage

fmjrey May 7, 2014 6:27 AM

Hello,

I'm new here and with infinispan, yet I'm considering it's usage as storage for datomic. The convenience of infinispan integration with datomic and immutant makes me wonder if we need something more than just that for durable storage (more would mean a more complicated setup). This is almost like the question of using infinispan as a database despite it's origin from being a data grid.

While datomic supports infinispan as storage, the creator of clojure and datomic, Rich Hickey, said in a post from a year and a half ago:

I think you need to be very careful if attempting to use Infinispan durably. Having grown up from being a cache, it has some interesting and possibly fatal options when treating cache loading as durability.

We only recommend Infinispan when your requirements are satisfied by a redundant memory cluster. In any case, be very certain of the semantics of the configuration settings you choose.

This was in october 2012 when infinispan 5.1 was released and 5.2 was in beta.

Since then I was wondering if new developments could mitigate this cautionary advice and if so what would be a proper configuration so that infinispan may become a reliable and durable storage, ie ACID even in case of hardware/datacenter failure. Reading infinispan doc it seems as if there could be some configuration where it can become a robust and durable store, part of such setting would certainly be the sync write-through mode, perhaps also the FULL_XA mode, but then it's unclear to me what would be the best store for such scenario and how the store would integrate with transactions (LevelDB doesn't do transaction from what I gather). Also unclear is if datomic, perhaps due to its way of using infinispan, would be able to take advantage of such configuration. This discussion indicates hotrod is used by both the datomic peers (readers only) and transactor (single writer).

A little help from more experienced infinispan developers/users would be appreciated either here or in the corresponding datomic google group discussion.

Thanks,

François

1. Re: Infinispan as datomic storage

sannegrinovero May 13, 2014 8:49 AM (in response to fmjrey)
Hi,
Rich's advice is still very valid, and probably will always be as there are configuration options made available to you which allow to favor performance over safety.
Of course you could avoid those options, but this implies understanding them carefully first. The point of running at least in a cluster is also very true: otherwise just shutting down or losing power / crashing of the single node would imply a data loss.

Since Infinispan 5.1 a lot of stability improvements where made making data loss less likely, still its very nature is to store data in memory. Generally when using it as a storage I'd recommend to use a CacheStore: for example to ultimately write-through (or write-behind if you think that's safe enough) to a traditional database, to Cassandra or to LevelDB. These ones don't do transactions, but having Infinispan "on top" you get transactions, and data is at least safely replicated to the memory of additional nodes in case the underlying storage fails.

To clarify a couple of the points you made:
Hot Rod doesn't support transactions
When Infinispan writes to a CacheStore, this operation is NOT included in the current transaction: it doesn't take advantage of transactionality of the CacheStore
There is no self-healing strategy to resync the CacheStore contents in case the transaction was accepted but then the CacheStore operation fails; it should eventually get fixed by further operations on the CacheStore but usage of certain options or flags could prevent this (I'm not sure of the full details here, just that there are some edges to consider).

Thanks for the pointers to datomic, very interesting.
Actions