9 Replies Latest reply on Feb 9, 2015 12:45 PM by ma6rl

    XAResources and Pessimistic Locking

    ma6rl

      I currently have Infinispan configured to enlist as a XAResource (FULL_XA) and use pessimistic locking.

       

      <cache-container name="modeshape" default-cache="repo" module="org.modeshape">
          <transport lock-timeout="60000"/>
          <replicated-cache name="repo" mode="SYNC">
              <locking isolation="READ_COMMITTED" striping="false"/>
              <transaction mode="FULL_XA" locking="PESSIMISTIC"/>
              <string-keyed-jdbc-store shared="true" preload="false" passivation="false" purge="false" datasource="java:jboss/datasources/MyDS">
                  <string-keyed-table prefix="modeshape">
                      <id-column name="id" type="VARCHAR(200)"/>
                      <data-column name="datum" type="LONGBLOB"/>
                      <timestamp-column name="version" type="BIGINT"/>
                  </string-keyed-table>
              </string-keyed-jdbc-store>
          </replicated-cache>
      </cache-container>
      

       

      The application uses FULL_XA as the I also have JMS and JDBC XA Resources enlisted in the same transaction as Infinispan.

       

      The application uses PESSIMISTIC locking (writeSkew=disabled) because the application often sees concurrent modification of cache entries is and is the recommended setting when configuring and infinispan cache for use in modeshape which is my current use case.

       

      Configuration - ModeShape 4 - Project Documentation Editor

      Applications that may be concurrently updating the same nodes should use Infinispan configured to use pessimistic locking with the READ_COMMITTED isolation level. By default Infinispan will use optimistic locking; this is more efficient for applications that don't update the same nodes, but concurrently updating the same nodes with optimistic locking may very well cause some updates to be lost. If you're not sure, use pessimistic locking.

       

      After reading the Infinispan documentation and performing some tests it appears that if I uses PESSIMISTIC locking (writeSkew=disabled) then the Infinispan transaction is performed as an independent one-phase-commit separate from the other XAResources in the transaction.


      Infinispan User Guide


       

      If the write skew check is not enabled, then all the transaction are committed in one phase (independently if Infinispan is enlisted as Synchronization or XaResource).

       

       

      Does this mean that the other resources enlisted in the transaction may not be rollback if they have already been committed and and error occurs during in the execution of the infinispan transaction?

       

      If this is the case what if any options do I have to ensure all resources are co-ordinated in the same transaction while guarding against high contention on keys?

       

      Many thanks in advance.

        • 1. Re: XAResources and Pessimistic Locking
          rvansa

          If the write skew check is not enabled, then all the transaction are committed in one phase (independently if Infinispan is enlisted as Synchronization or XaResource).

           

          The citation from Infinispan docs relates to Total Order Protocol - this is not used in Infinispan by default. With optimistic locking, transactions are executed in two phases (in first phase the locks for modified keys are acquired and write skew is checked, in second phase the modifications are applied). With pessimistic locking, the locks are acquired before each modification method call, and then during the 1-phase commit the write skew is checked and all modified keys (already locked) are applied.

           

          but concurrently updating the same nodes with optimistic locking may very well cause some updates to be lost

           

          Besides bugs, with optimistic locking AND write skew check enabled there can't be a lost update - if the entry was modified (by another transaction) after reading it, write skew checks would rollback the transaction. Maybe the ModeShape revealed a bug (I remember that few of them related to WSC) and considered this a feature.

          On the other hand, pessimistic locking without WSC could lead to update loss, because if you do

           

          int x = cache.get('x'); cache.put('x', x + 1);

           

          the lock is acquired only before the put() call. The fix for this would be to use write skew check (that's optimistic approach), or pessimistically:

           

          cache.getAdvancedCache().lock('x'); int x = cache.get('x'); cache.put('x', x + 1);

           

          or even better:

           

          int x = cache.getAdvancedCache().withFlags(FORCE_WRITE_LOCK).get('x'); cache.put('x', x + 1);

           

          Regarding pessimistic case and other XA resources: I think that Infinispan prepares as the last XA resource with 1pc commit, therefore, other resources are already prepared and the transaction is safe. nadirx would know the correct answer.

          • 2. Re: XAResources and Pessimistic Locking
            pruivo

            Just a couple of things about the pessimistic transaction mode:

             

            * internally, Infinispan commits the transaction in one-phase, independently if the transaction manager asks Infinispan to commit in one or two phase. If the transaction manager ask to commit in two phase, in the first phase (the prepare), Infinispan does not perform anything and reports OK (locks are already acquired and the transaction is safe to commit)

            * the writeSkew configuration is not allowed in pessimistic transactions. This is a problem when optimistic transactions is used.

             

            rvansa already replied to the rest, so if you have others questions let us know.

            • 3. Re: XAResources and Pessimistic Locking
              ma6rl

              rvansa, pruivo thanks for the information above. I believe I now understand how Infinispan participates as an XA resource in a transaction with Pessimistic locking but I just want to confirm that I have it straight.

               

              The following assumes FULL_XA with PESSIMISTIC locking:

               

              1. At the start of the transaction Infinispan enlists as an XA Resource along with any other XA Resources in the transaction e.g. JMS

              2. A cluster wide lock is created for each key prior to the prepare stage at the point in time the key is first written to. If a lock already exists the current transaction is blocked until either the lock becomes available or times out and the transaction fails.

              3. The transaction is the prepared. If all the other XA Resources in the transaction prepare successfully Infinispan then performs a one phase commit (there is no need to prepare as the locks have already been acquired).

              4. If a cache store is configured each modification in the transaction is applied to the cache store. The cache store is NOT enlisted in the transaction.

              5. If all of the cache store modifications are successful the one phase commit completes and the other XA Resources in the transaction are committed.

              6. If a cache store modification fails the one phase commit fails, the cache is not updated an the other XA Resources in the transaction are rolled back. Any modifications to the cache store that were made prior to the error will remain in the cache store and the store will no longer be in-sync with the cache.

               

              Could you confirm if the above is correct or if it is not let me know what I've misunderstood?

               

              Many thanks in advance.

              1 of 1 people found this helpful
              • 4. Re: XAResources and Pessimistic Locking
                pruivo

                In 2) you can acquire the lock explicit or using FORCE_WRITE_LOCK (as in the Radim's example).

                 

                In 6) it depends on the TransactionManager implementation that you are using. If anything fails in Infinispan, it reports to the TransactionManager and it marks the transaction as "in doubt". In this state, the locks are not released and it requires a system administrator intervention to commit/rollback the transaction. You can know about it in the user guide: http://infinispan.org/docs/7.0.x/user_guide/user_guide.html#_transaction_recovery

                • 5. Re: XAResources and Pessimistic Locking
                  rvansa

                  pruivo are you sure that failure to write a value into cache store really fails the transaction? I've seen in JDG documentation that this failure is logged but ignored.

                  • 6. Re: XAResources and Pessimistic Locking
                    pruivo

                    yes, I'm sure, for sync cache store (except if we are swallowing the exception somewhere, but I'm not aware of anything like that).

                    For async, the failure is logged as you said.

                    • 7. Re: XAResources and Pessimistic Locking
                      ma6rl

                      I've done some testing on 6.0.2.Final and can confirm that the transaction is left 'in doubt' failed if a cache store write fails.

                       

                      A CacheException is thrown by the CacheWriter while executing the CommitCommand. This is caught by the TransactionCoordinator#commitInternal and passed to TransactionCoordinator#handleCommit which throws an XAException indicating a heuristic rollback. The transaction is then marked as 'in doubt/failed' by the TransactionManager (in my case the JBoss JTA transaction manager shipped with Wildfly 8.2) and the error is logged. Control is then returned to Infinispan by the TxManager by invoking SynchronizationAdapter#afterCompletion which re-throws the original CacheException.

                       

                      Unfortunately that's where it stops as the CacheException is caught by com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple#afterCompletion and swallowed which means the exception never propagates up to the application tier that invoked the cache write and as far as the application logic is concerned it was successful although neither the store or the cache were actually updated.

                       

                      I currently workaround this by querying the cache in a new Tx after the write Tx completes to see if my modifications are available if not I invoke my error handling logic.

                      • 8. Re: XAResources and Pessimistic Locking
                        rvansa

                        If SynchronizationImple class is involved, seems like XA is not used underneath. Sychronization (NON_XA transaction mode) really does not propagate the exception. And the configuration example you've posted in the beginning is 7.0-style, so you could have misconfigured it in the test with 6.0.2.

                        • 9. Re: XAResources and Pessimistic Locking
                          ma6rl

                          rvansa, thanks for the comments. The configuration above is taken from my Wildfly standalone.xml where Infinispan is registered a configured as a Wildfly sub-system. This uses different configuration to a standalone 6.0.2.Final Infinispan instance and I believe was the basis from the 7.0 configuration changes.

                           

                          With regards to your comments about  SynchronizationImple, I believe the problem is not Infinispan as I original thought but instead Modeshape. I do not interact with Infinispan directly but instead use Modeshape which in turn uses Infinispan. The following is taken from the org.modeshape.jcr.txn.Transactions.

                           

                          * Note that when distributed (XA) transactions are used, ModeShape properly integrates and uses the XA transaction but does not
                          * register itself as an {@link XAResource}. (Note that the Infinispan cache <i>is</i> enlisted as a resource in the transaction.)
                          * Therefore, even when XA transactions only involve the JCR repository as the single resource, ModeShape enlists only a single
                          * resource, allowing the transaction manager to optimize the 2PC with a single resource as a 1PC transaction. (Rather than
                          * enlisting the repository as an XAResource, ModeShape registers a {@link Synchronization} with the transaction to be notified
                          * when the transaction commits successfully, and it uses this to dictate when the events and session's changes are made visible
                          * to other sessions.

                          So it looks like it is Modeshape's interacting with the TxManager that results in the exception being swallowed.

                           

                          I plan on doing some tests using Infinispan directly to see how it behaves in XA vs NON-XA mode but in the mean time I will talk to the Modeshape community to see if there is anything I can do.

                           

                          Also out in interest what is the reason for this?

                          Sychronization (NON_XA transaction mode) really does not propagate the exception.