5 Replies Latest reply on Feb 16, 2017 11:40 AM by Radim Vansa

    Transactional behavior when using CACHE_MODE_LOCAL flag

    José Dâmaso Newbie

      When updating an invalidation cache (transactional - NON_XA) using the CACHE_MODE_LOCAL flag, I noticed a distributed transaction (and locks) are still performed.

      Is it possible to configure the cache to participate in a running JTA transaction locally for a single put operation (same behavior as a local cache), or disable transactions for a single operation?

       

      Just to provide some background for my question:

      I am using an invalidation cache in a 2-node JBoss cluster to minimize database access.

      Entries are loaded from the database and stored in the cache for future use. This can be done individually by each node, no need for a replicated cache (using put with CACHE_MODE_LOCAL).

      However, if the underlying database tables are changed, the cache entries must be invalidated cluster-wide (normal put operation).

      The database change and cache removal are done in a single transaction, to avoid reloading stale data on the interval between the cache entry removal and database commit.

       

      So, summing up, is there a way to maintain the transaction guarantees while avoiding distributed transactions in certain put operations (local only transaction or no transaction at all)?

        • 1. Re: Transactional behavior when using CACHE_MODE_LOCAL flag
          Radim Vansa Master

          The flag should be honored, and if there's no non-local operation in the transaction, it should not go to remote nodes. How have you observed that the transaction goes remote? And what version do you use?

          • 2. Re: Transactional behavior when using CACHE_MODE_LOCAL flag
            José Dâmaso Newbie

            I am load testing my application (only read scenarios, but both nodes are

            trying to load the same entries on their cache concurrently) and noticed a

            lot of replication timeout errors and poor performance. As soon as I change

            the cache to a local cache or disable transactions, the errors disappear

            and performance goes back to normal.

             

            Some technical details (I am using Infinispan 8.2.6 - infinispan-core dependency):

             

            Infinispan configuration (the "catalog.version" cache is not being used in my load test scenario):

             

            <?xml version="1.0" encoding="UTF-8"?>
            <infinispan xmlns="urn:infinispan:config:8.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:infinispan:config:8.2 http://infinispan.org/schemas/infinispan-config-8.2.xsd">
            
              <jgroups>
                <stack-file name="jgroups-udp" path="jgroups-udp.xml"/>
              </jgroups>
            
              <cache-container default-cache="default">
                <transport stack="jgroups-udp"/>
            
                <jmx domain="sds.infinispan"/>
            
                <invalidation-cache name="default">
                  <transaction mode="NON_XA"/>
                  <store-as-binary/>
                </invalidation-cache>
                
                <distributed-cache name="catalog.version"/>
                <invalidation-cache name="catalog.materializations"/>
                <invalidation-cache name="catalog.tariff.subscription.specificity"/>
                <invalidation-cache name="oidc.clients"/>
                <invalidation-cache name="preference.preferences"/>
                <invalidation-cache name="preference.values"/>
                <invalidation-cache name="sap.sysConf"/>
              </cache-container>
            </infinispan>
            

             

            Put (load) operation (using a cache wrapper to have a friendlier interface):

             

            package pt.link.sc.sds.cache;
            
            import org.infinispan.cache.impl.AbstractDelegatingCache;
            
            import java.util.function.Supplier;
            
            import static org.infinispan.context.Flag.*;
            
            public class Cache<K, V> extends AbstractDelegatingCache<K, V> {
            
                public Cache(org.infinispan.Cache<K, V> cache) {
                    super(cache);
                }
            
                public V getOrLoad(K key, Supplier<V> loader) {
                    V value = get(key);
            
                    if (value == null) {
                        value = loader.get();
            
                        getAdvancedCache()
                                .withFlags(CACHE_MODE_LOCAL, FAIL_SILENTLY, ZERO_LOCK_ACQUISITION_TIMEOUT)
                                .put(key, value);
                    }
            
                    return value;
                }
            
            }
            
            • 3. Re: Transactional behavior when using CACHE_MODE_LOCAL flag
              Radim Vansa Master

              It seems that you're right, care to create a JIRA and link to this forum thread? It seems that the PrepareCommand is sent with single modification: InvalidateCommand, with empty set of keys.

               

              PS: you can even contribute a fix

               

              Here is a test (in o.i.invalidation.BaseInvalidationTest) that reproduces it:

               

                 public void testLocalModeTxPutDoesNotGoRemote() throws Exception {
                    AdvancedCache cache1 = cache(0,"invalidationTx").getAdvancedCache();
                    AdvancedCache cache2 = cache(1,"invalidationTx").getAdvancedCache();
              
                    RpcManager originalRpcManager = TestingUtil.extractComponent(cache1, RpcManager.class);
                    TestingUtil.replaceComponent(cache1, RpcManager.class, new AbstractControlledRpcManager(originalRpcManager) {
                       @Override
                       protected Object beforeInvokeRemotely(ReplicableCommand command) {
                          throw new IllegalStateException("Should not invoke remote operations");
                       }
                    }, true);
              
                    TransactionManager tm1 = TestingUtil.getTransactionManager(cache1);
              
                    tm1.begin();
                    cache1.withFlags(CACHE_MODE_LOCAL).put("key", "value");
                    tm1.commit();
              
                    assertEquals("value", cache1.get("key"));
                    assertNull(cache2.get("key"));
                 }
              
              
              
              • 4. Re: Transactional behavior when using CACHE_MODE_LOCAL flag
                William Burns Expert

                I personally think the way this behaves in general should be reevaluated.  To me an Invalidation cache has only purpose and that is to send InvalidateCommand messages to the cluster. No other commands should be sent. In this case the lock command, prepare, commit, rollback etc should never be sent across. Maybe people envisioned this being able to be used to to control locking behavior when using a tx?  Unfortunately without a primary owner we will run into deadlocks like Infinispan DIST used to and exactly what Jose ran into.

                 

                The InvalidateCommand for a Tx shouldn't be sending an empty set of keys either like in Radim's test

                • 5. Re: Transactional behavior when using CACHE_MODE_LOCAL flag
                  Radim Vansa Master

                  I agree with Will - the locks seemingly don't work with invalidation cache. But just broadcasting invalidate command is not enough: usually you want to start invalidating the entry during prepare phase, block (ignore) any writes to that during the DB commit and stop invalidating after commit. XA transactions can't work well, since you can't set up the order of XA resources during prepare/commit and the invalidation has to "wrap" the DB commit.

                   

                  This comes up pretty often, and I have an itch that we should accomodate some generic tier (similar cache mode, but likely with different API) that will cover both ORM 2LC use case, as well as other uses as cache (e.g. what Keycloak uses).