Configuring transactions in Modeshape

[mkotsur]

We're building a clustered standalone Java application with Modeshape 4.4 and it's not completely clear how we should configure transactions in Infinispan. One thing I understood: Modeshape requires transactions in order to be able to commit.

Several times I've encountered in the docs that this configuration is good as a default choice:

            <transaction mode="NON_XA" locking="PESSIMISTIC"/>

However, what concerns us is that if we don't specify transaction manager lookup explicitly, it will use built-in GenericTransactionManagerLookup which will mean that DummyTransactionManager is used which works within a single JVM and can not coordinate actions of different nodes in the cluster.

In our application:

• We don't do JTA transactions, only session.save();
• It can happen that a node is modified concurrently from different nodes as well as the same node;

Does modeshape, or any components inside it require more sophisticated transaction manager to work properly?

Thanks in advance for your replies.

[hchiorean]

First, if you're planning on clustering, I strongly suggest you look at Clustering - ModeShape 4 - Project Documentation Editor for the different topologies and known Infinispan issues and limitations.

Regarding transactions: first and foremost you must make sure that the Infinispan cache is enrolled in a transaction (either XA or NON_XA). The normal approach for that is, like you mention above, to configure it to use a GenericTransactionManagerLookup instance. This in turn, does not necessarily mean that a DummyTransactionManager is used. It means that Infinspan will look at runtime at its running environment and will attempt to locate an existing transaction manager implementation (e.g. JBossAS, Weblogic etc) and only if it doesn't find one, it will fall back to the DummyTransactionManager.

If you're not planning on running your application in an enviroment (i.e. container) where a transaction manager is present by default and which can be detected by the GenericTransactionManagerLookup instance, you have to use a transaction manager implementation and make sure Infinispan is able to detect it. This example: modeshape-examples/modeshape-spring-example at master · ModeShape/modeshape-examples · GitHub shows how you can use Atomikos as a custom Tx manager and plug it into ISPN.

Regarding XA versus NON_XA: ModeShape can run with Infinispan configured either as NON_XA or some XA flavor. When running in cluster, using XA is not a requirement unless you hit issues like [MODE-2427] Infinispan cache commit failures do not propagate to Modeshape when synchronizations are used - JBoss Issue …)

In our application:

• We don't do JTA transactions, only session.save();

That's fine, you don't have to explicitly use Transactional APIs in your application (ModeShape & ISPN will do that behind the scenes). However, as explained above, you must have a JTA implementation available in your runtime environment which Infinispan can integrate with.

• It can happen that a node is modified concurrently from different nodes as well as the same node;

ModeShape relies on Infinispan PESSIMISTIC locking in its attempt to provide strong consistency (which is essentially required by the JCR spec). This means that when running in a cluster, global cluster locking will be used (via ISPN) so that concurrent updates should not be a problem (theoretically )

[mkotsur]

Horia, thanks for your reply. We are going to try to use either atomicos, or narayana. Any idea which one would work better, or be simpler to configure in our case?

Regarding optimistic/pessimistic locking we've conducted some experiments and for some reason, we're not getting consistent reads with either of configurations (on a single node). So far we've tried all possible combinations of :

• NON_XA/FULL_XA
• PESSIMISTIC/OPTIMISTIC
• DummyTransactionManager/com.arjuna.ats.jta.TransactionManager

In essence, the experiment is:

when:

setting property test_1 to n;

setting property test_2 to n;

session.save();

then:

read property test_1 and property test_2;

they should be equal.

This block is executed concurrently up to 10000 times with thread pool consisting of 10 threads. It fails with the difference that amount of failed checks with OPTIMISTIC locking is way greater than with PESSIMISTIC. Let's say 250 vs 5.

The code is available in this github repo: mkotsur/modeshape-transactions-poc · GitHub. I can translate it into maven/java if it's a problem :-)

[hchiorean]

Horia, thanks for your reply. We are going to try to use either atomicos, or narayana. Any idea which one would work better, or be simpler to configure in our case

Naryana should be simpler to configure, because if it's present in your runtime classpath ISPN can pick it up out-of-the-box, without the need for any additional configuration by using: transaction-manager-lookup="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup" (see https://github.com/ModeShape/modeshape/blob/master/modeshape-jcr/src/test/resources/config/infinispan-persistent-jbosstxn-pessimistic.xml). Atomikos on the other hand will require a bit of custom code (see the example from my previous post).

Regarding optimistic/pessimistic locking we've conducted some experiments and for some reason, we're not getting consistent reads with either of configurations (on a single node). So far we've tried all possible combinations of :

• NON_XA/FULL_XA
• PESSIMISTIC/OPTIMISTIC
• DummyTransactionManager/com.arjuna.ats.jta.TransactionManager

You should always use PESSIMISTIC locking with READ_COMMITTED isolation, as documented here: Configuration - ModeShape 4 - Project Documentation Editor. For POC purposes, the tx manager doesn't matter (you can try with Dummy) and also NON_XA/FULL_XA should not matter in a non-clustered POC. Looking at your code, you've configured the ISPN cache to use write-behind, making it asynchronous. ModeShape will never work correctly with this type of configuration, as it requires the cache to be synchronous.

If you haven't already tried, you should remove the write-behind setting and try running with PESSIMISTIC locking & READ_COMMITTED isolation. If your POC still fails, you should try running another cache store (e.g. FS) just to make sure it's not related to the LevelDB cache store. In general, we have extensive unit tests for these type of concurrent scenarios and all of them work fine.

[mkotsur]

First of all, thanks a lot for your help!

I've updated the code with suggested values (also using FS store), the test still fails, however with much lower number: around 4 out of 100000. I can imagine that the problem is not caught by the unit-tests because of high level of repeats which has to happen before it appears.

What I want to do now is make sure that transaction manager does its job correctly and extend this POC for 2 nodes. Is there any kind of scenario you could advice for testing that, or I could use the same?

Also, one thing I'm still missing in the whole picture is: how 2 instances of transaction manager on different nodes coordinate their work? Do they rely on Jgroups configuration?

[rhauch]

Mike Kotsur wrote:

In essence, the experiment is:

when:

setting property test_1 to n;

setting property test_2 to n;

session.save();

then:

read property test_1 and property test_2;

they should be equal.

This block is executed concurrently up to 10000 times with thread pool consisting of 10 threads. It fails with the difference that amount of failed checks with OPTIMISTIC locking is way greater than with PESSIMISTIC. Let's say 250 vs 5.

Even if the "when" block works consistently 100% of the time (and it should with pessimistic locking), it is possible under high concurrency for the two reads in the "then" block to still read different values. This is because the nodes in ModeShape sessions that have not been modified are dynamically updated when other sessions save changes to those nodes. So, one reason your test might fail is that after a session reads a value from "test_1", the node is updated to reflect the changes saved by another session, and then your test reads a now-different value from "test_2". These kinds of failures appear because of how the test is written, not because ModeShape behaves inconsistently.

[hchiorean]

What I want to do now is make sure that transaction manager does its job correctly and extend this POC for 2 nodes. Is there any kind of scenario you could advice for testing that, or I could use the same?

Not sure what you mean. We have some clustering quickstarts for JBoss AS (quickstart/modeshape-clustering at master · ModeShape/quickstart · GitHub) which you can use to see the ISPN & JGroups configuration files. Other than that, you can also look at some the clustering configuration for the local unit tests: https://github.com/ModeShape/modeshape/tree/modeshape-4.4.0.Final/modeshape-jcr/src/test/resources/config

Also, one thing I'm still missing in the whole picture is: how 2 instances of transaction manager on different nodes coordinate their work? Do they rely on Jgroups configuration?

JGroups is the backbone for all the remote/cross node communication. Since the only recommended clustering setup for ModeShape is using a shared cache store and since ModeShape relies on ISPN's pessimistic locking (cluster-wide), there isn't any specific "communication" that needs to take place between the transaction managers (which is also why NON_XA works in a cluster). All those aspects are handled by Infinispan really (not ModeShape) so if you want you can read more on those topics here: Consistency guarantees in Infinispan · infinispan/infinispan Wiki · GitHub