Thanks hchiorean

After spending more time researching this and looking through both the Infinispan source code and issue tracker including the links above (which are very helpful) I've confirmed the cache stores are never enlisted in the cache transaction. The call to the store is done in the commit phase cache entry that is added in the transactions is added as a separate call to the store and no attempt to rollback the store is made if any of the writes fail.

This issues is only relevant if using the JDBC Cache Store as none of the other cache store implementations support transactions, and I can see why in the majority of cases it will not work (the cache store resource and JTA transaction manger are on different nodes). The one scenario where this could work if cache stores enlisted in transactions is using a shared JDBC Cache Store which is what I do.

Given that Modeshape's persistence implementation uses Infinispan to persist nodes then even when using transactions to ensure all of the changes within the transaction are either committed or rolled back there is no guarantee that the writes to the underlying cache store are committed/rolled back correctly and may have a different sent of entries to the in-memory cache. The in-memory which will mask the inconsistencies until the application is restarted. While this may not happen frequently it will eventually happen which means you can not store data in modeshape with any confidence that it will be there after the application is restarted.

It appears that Infinispan does provide a manual recovery mechanism as discussed in the above link, there is some controversy around it's usability with cache stores. The above link says it should work as does JIRA issue but the official documentation (see quote in first post) says it does not. Either way this is not an ideal solution when scaling to 100's writes a second and billions of nodes which is our long term goal.

This is kind of deal breaker when it comes to using Modeshape as a database for business critical data that needs to survive an application restart and really limits it's uses which I'm sure is not the intention of the ModeShape developers. I understand that this is not something that can be fixed by the modeshape developers directly but give Infinispan limitation's around this is there any intention to look at more reliable persistence approaches in the future?

rhauch, I've asked the questions in the Infinispan forum but based on past experience they are not as responsive as you and Horia which is why I was hoping to engage your help.

Can you please elaborate on

Randall Hauch wrote:

 

I don't think it's a foregone conclusion that the cache stores have to be XA resources in order for everything to work properly in a fault-tolerant manner.

I've spent some time going through both the Cache Store framework and StringBasedJDBCStore implementation and from what I can see is:

1. The call to store the entry is done in the commit phase

2. Each entry is stored in a separate call to the store which can either succeed or fail

3. If an error occurs a Persistence Exception is thrown, an previous calls to the store are not rolledback

4. The commit fails and the cache is not updated. At this point I'm a little hazy on what happens to other XA resources in the transaction but I believe those which have already committed are left committed.

Given this I'm not sure how a cache-store can be fault tolerant.

Part of the problem is that there is no batching in the Cache Store, if the framework passed all of the entries that needed to be persisted to the cache store at the same time then there is at least an opportunity to perform updates in batch and attempt to clean up if something goes wrong.

rhauch, I've now had some feedback from the Infinispan community both on their forum and stack overflow. The general gist is that currently cache stores are not fault tolerant and can overtime become out of sync with the cache

From Re: Enlist Cache Store in a Cache Transaction?

The short answer is: no.

We are thinking of allowing this for a very narrow case: with shared cachestores, where only the node which initiates the tx has the onus of writing all changes. However, because of the way the data is distributed in the cluster, the initiator node will need to retrieve all entries modified in the tx from the respective owners, and persist them.

From http://stackoverflow.com/questions/28127428/enlisting-a-infinispan-cache-store-in-a-cache-transaction/28146994#28146994

It applies to writes as well, failure to write to the store does not affect rest of the transaction.

The reason for this is that the actual persistence API is not transactional. Therefore, with 2-phase commits (in first phase - prepare - all locks are acquired, in second one - commit - the write is executed) the write to the store is executed in the second phase. Therefore, the failure cannot rollback changes on different nodes.

Saying that there are potential plans to support enlisting cache stores in XA Tx in certain uses cases. I've also put forward a proposal that could potential allow a Cache Store implementation to batch writes and attempt to rollback when participating in an XA Tx is not available.

Given this do you think it might be useful to update the Modeshape documentation to include this information so that potential users evaluating Modeshape as a solution are aware of the limitations?

hchiorean I'm not sure I understand your previous reply. Given the information provided in the links above by the Infinispan community I don't believe you can assume:

Horia Chiorean wrote:

 

I think the main scenario where this problem manifests itself (potentially) is when running a cluster with multiple cache stores

Horia Chiorean wrote:

 

in effect, just 1 cache store which should always be in-sync with whatever node made the changes last.

The current issue with cache stores in that Infinispan currently only does a best effort to ensure they remain in-sync with the cache (transactional or not).

Currently when using transactions infinispan updates the cache store and cache as follows:

1. A lock on the key is obtained either at write time (pessimistic) or prepare time (optimistic)

2. Infinispan then attempts to update the cache store for each key being modified in the transaction. This is done in the commit stage of the transaction (or via a one phase commit if using pessimistic locking). This step NEVER participates in a transaction and each modification is done as a separate call to the store.

3. If all of the modifications to the cache store where successful then the changes to the cache are marked as successfully committed, otherwise the changes to the cache are rolled back. No attempt is ever made to clean up an modifications to the cache store that occurred in the transaction prior to a failure.

This means that if an error occurs writing to a cache store (and where Networking and IO are involved errors always occur) then any modifications made prior to the error will remain in the cache store but NOT be reflected in the cache. At this point if the cache is destroyed (either the single node in a local cache or all nodes in the cluster in a replicated cache are shutdown) and the node(s) are restarted the cache will be rebuilt using the cache store which may NOT actually represent the expected state of the application because it became out of sync with cache prior to the cache being destroyed.

This behaviors should be expected when using a non-transactional cache store e.g. file system as you can not reasonably expect file system updates to be rollback but when you use a cache store that can support transactions e.g. JDBC it is not unreasonable to expect it to participate in a transaction. Given that this is not the case there is currently no way to configure ModeShape to ensure that the cache store will always correctly represent the applications state.

While this may be expectable in some JCR use cases where consistency is not an issue, there are many cases where a JCR does need to be at least eventually consistent if not strongly consistent and currently Modeshape can not gaurantee this.

As per the links above the Infinispan community is aware of this and is looking at supporting shared XA cache stores (e.g. JDCB) enlisting in the transaction but if this is implemented it will most likely be part of 7.x or 8.x where as Modeshape currently uses 6.0.2.Final and is unlikely to move forward for a while given Wildfly 8 and most likely the next EAP also uses 6.0.2.Final.

Horia Chiorean wrote:

 

If something fails - i.e. the transaction does not complete successfully - both cache & cache store are rolled back and effectively kept in-sync.

When running in invalidation mode with a shared cache store, I assumed (perhaps wrongly) that since there is only 1 cache store instance in the environment, tx recovery would work automatically (not using synchronizations but as part of 2PC), keeping the cache store & caches in sync.

 

The critical issue here is that the store is not enlisted in the transaction 1PC or 2PC and is never rolled back. If you look at the code in Infinispan that does this it actually suspends the Tx prior to writing to the store and resumes it afterwards (CacheWriterInterceptor.java) This also confirmed in the Infinispan documentation and the links I posted above.

Given this how do you feel about me raising a JIRA issue to track this. I understand that this is not something that can be fixed in the modeshape code base as it depends on new features in infinispan being implemented or a significant redesign of modeshapea persistance tier but it would be good to be aware of the issue and mathe address it in the future.

I am also looking at patching Infinispan 6.0.2.Final to see if I can get a cache store to participate in the transacroon. I'll let you know how it goes.

Raised issue. [MODE-2420] Modeshape can potentially lose data because Infinispan Cache Stores do not participate in transactions - JBo…