Modeshape Consistency Concerns

[ma6rl]

I've been in the process of evaluating and prototyping an application that uses Modeshape for the last several months. During this time I've discovered some issues with using Modeshape as a consistent store and wanted to bring them to the attention of the community in order to see if and how they can best be addressed.

The consistency issues all stem from the use of Infinispan as a persistence store. While Infinispan is a powerful distributed cache and is moving in the direction of becoming a strongly consistent in-memort database it does not have currently provide the consistency guarantees that you would expect from a persistence store.

The two issues that I have encountered so far our:

[MODE-2420] Modeshape can potentially lose data because Infinispan Cache Stores do not participate in transactions - JBo…

Cache stores do not participate in the Infinispan Tx and are not rolled back if an error occurs writing to the store. The cache does not get updated if an error occurs and this results in the store having different values to the cache.

[MODE-2427] Infinispan cache commit failures do not propagate to Modeshape - JBoss Issue Tracker

If an error occurs during the commit-phase of the Infinispan Tx Modeshape does not see this error and considers the Tx successful even though the cache is not updated. This issue is amplified by the fact that cache store writes happen during the commit-phase and our prone to failure (IO, Network, Resources etc).

The first issue is the most concerning because it means the store will never become consistent. The second issue could be argued as a limitation of using a NoSQL database and that the store does become eventually consistent and the application needs to handle this but if this is the case this should be well documented.

Given these issues along with the inflexibility to adopt new versions of Infinispan because Modeshape is tied to using the same version as Wildfly would it make sense to look at alternative persistence options for Modeshape?

[hchiorean]

[MODE-2427] Infinispan cache commit failures do not propagate to Modeshape - JBoss Issue Tracker should not be an issue with the latest codebase (upcoming 4.2). See my JIRA comment on the subject.

Given these issues along with the inflexibility to adopt new versions of Infinispan because Modeshape is tied to using the same version as Wildfly would it make sense to look at alternative persistence options for Modeshape?

While something that may be worth investigating, my personal view on the matter is that this is something highly unlikely to happen at least for the current major version of ModeShape - the 4.x series. It's more realistic to hope that a newer version of Wildfly will come along with a version of ISPN that will have the issues that you mention fixed.

Of course if anyone in the community feels like forking off ModeShape and attempt at providing an alternate persistence solution, that'd be great. At a minimum that should support clustering (maybe just replication to begin with) and JDBC, FS and in-memory storage IMO.

[ma6rl]

My initial thoughts on how moving persistence out of Infinispancould be achieved are:

1) Continue to use Infinispan as a replicated/distributed cache which would allow modeshape nodes to be clustered as they are today

2) No longer rely on Infinispan cache stores.

3) Implement a persistence api in Modeshape that abstracts writing to file/jdbc etc that correctly participates in a transaction

4). Continue to use the Schematic DB and persist data as key/value pairs

5) Have modeshape co-ordinate the interaction between the persistence store and infinispan cache e.g. Reads would attempt to retrieve the entry from the cache and if not found use the persistence api to retrieve the entry. Writes would updated the persistence store and cache in the same Tx and roll both back if a problem occurs.

This is obviously very high level and needs a lot more work but any initial feedback would be appreciated.

[hchiorean]

Regarding the exception propagation, this seems to be directly related to the fact that Infinispan is using synchronizers to enroll in the active transaction. However, this is not the only option: did you try running with the FULL_XA or NON_DURABLE_XA transaction modes ? That should make Infinispan participate in the transaction as an XA resource.

[ma6rl]

Horia Chiorean wrote:

Regarding the exception propagation, this seems to be directly related to the fact that Infinispan is using synchronizers to enroll in the active transaction. However, this is not the only option: did you try running with the FULL_XA or NON_DURABLE_XA transaction modes ? That should make Infinispan participate in the transaction as an XA resource.

hchiorean Thanks for the suggestion. I have previously tested FULL_XA with the Modeshape 4.0 code base and seen similar issues. The exception gets swallowed somewhere in the XARecovery code. I have not tired NON_DURABLE.

I will retest both with the latest 4.2.0-SNAPSHOT and let you know.

[ma6rl]

hchiorean I've restested with FULL_XA and NON_DURABLE_XA and the good news is NON_DURABLE_XA works.

FULL_XA still fails to propagate the exception, I believe it is swallowed somewhere around here

NON_DURABLE_XA does propagate the exception to the application tier and my transaction is being rolled back as required.

So it looks like NON_DURABLE_XA is the way to go at least for my use case. It may be worth adding this to the Modeshape documentation so that users are aware of the consequences of the different transaction modes.

Thanks four your help in finding a solution to this.

[ma6rl]

For anyone looking for a solution to the issues described in the original post we are doing the following:

1. Using NON_DURABLE_XA as the Infinispan Tx mode for the modeshape cache (configured via the Wildfly subsystem). This ensures the exception is caught by the container and the Tx rolled back.

2. Using a patched version of Infinispan 6.0.2.Final that wraps the call to the cache stores in a Local Tx and rolls it back if an exception is thrown (see here). This ensures if a cache store write fails to a JDBC store all writes are rolled back.

Dependencies:

Wildfly 8.2

Modeshape 4.2-SNAPSHOT (hopefully a final version will be available soon)

Infinispan 6.0.2.Final Patched (built using this branch. This is experimental and should be used at your own risk).

This has been only been tested using EJB Container Managed transactions.