4 Replies Latest reply on Apr 13, 2016 3:33 AM by pruivo

    is there 'Lock Reordering' for 'Optimistic locking' on 'Local' mode (ISPN 6.X)

    ofer.baranes

      Hello,

      According to this post:

          

            Lock Reordering For Avoiding Deadlocks · infinispan/infinispan Wiki · GitHub

       

      optimistic locking should enjoy locks reordering in order to avoid deadlocks.

       

      I suspect it is not working properly on ISPN 6.0.2 for 'Local' mode.

       

      So the question: is that feature supported on ISPN 6.x for 'Optimistic locking' on 'Local' mode?

      If so, is there a ready test on ISPN for reference?

       

      Thanks

      Ofer

        • 1. Re: is there 'Lock Reordering' for 'Optimistic locking' on 'Local' mode (ISPN 6.X)
          pruivo

          Hi Ofer,

           

          Yes lock ordering is supported in local mode for optimistic transactions in 6.x. However, it does not support clear operation, i.e., lock ordering will not be used if clear() is invoked.

           

          You can find a test in here: infinispan/LocalLockReorderingTest.java at 6.0.2.Final · infinispan/infinispan · GitHub

           

          Cheers,

          Pedro

          • 2. Re: is there 'Lock Reordering' for 'Optimistic locking' on 'Local' mode (ISPN 6.X)
            rvansa

            Note that the ordering operation is based on hashCode() of keys; if you happen to have collision on hashCodes, it's possible that the keys will be locked in different order in different transactions. Is this your case?

            • 3. Re: is there 'Lock Reordering' for 'Optimistic locking' on 'Local' mode (ISPN 6.X)
              ofer.baranes

              Hello,

               

              I ensured that the keys are different from the MurmurHash2 perspective for the deadlock case.

              Unlike ISPN LocalLockReorderingTest , on my case, there are 2 caches involved.


              Please find a test simulating the issue (sorry for its ISPN clumsy code..)

               

              To run the test, please implement its abstract startTransaction & endTransaction methods. (i have a JTA transaction manager plugged)


              TestInfinispanLocalLockReorderingBug.java:


              import java.net.URL;

              import java.util.ArrayList;

              import java.util.List;

              import java.util.concurrent.Callable;

              import java.util.concurrent.ExecutorService;

              import java.util.concurrent.Executors;

              import java.util.concurrent.Future;

              import junit.framework.TestCase;

              import org.infinispan.Cache;

              import org.infinispan.commons.hash.MurmurHash2;

              import org.infinispan.manager.DefaultCacheManager;

              import org.infinispan.util.concurrent.TimeoutException;

              import org.slf4j.Logger;

              import org.slf4j.LoggerFactory;

               

              public abstract class TestInfinispanLocalLockReorderingBug extends TestCase {

                private static final Logger logger = LoggerFactory.getLogger(TestInfinispanLocalLockReorderingBug.class);

               

                DefaultCacheManager cacheManager;

                Cache cache1;

                Cache cache2;

               

                ExecutorService service;

               

                public TestInfinispanLocalLockReorderingBug(String theName) {

                super(theName);

                }

               

                abstract void startTransaction();

                abstract void endTransaction();

               

                @Override

                protected void setUp() throws Exception {

                service = Executors.newFixedThreadPool(10);

               

                ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader() ;

                URL configFileURL = contextClassLoader.getResource("infinispan.xml"); 

                cacheManager = new DefaultCacheManager(configFileURL.openStream());

                cache1 = cacheManager.getCache("cache1");

                cache2 = cacheManager.getCache("cache2");

                }

               

                @Override

                protected void tearDown() throws Exception {

                service.shutdown();

                cache1.stop();

                cache2.stop();

                cacheManager.stop();

                }

               

                public void testDeadlock() throws Exception {

                /** this is what's used for inducing ordering */

                MurmurHash2 hashFunction = new MurmurHash2();

                assertTrue(hashFunction.hash("key1") !=  hashFunction.hash("key2"));     

               

                try {

                class CacheAccess implements Callable<TimeoutException> {

                int id;

               

                public CacheAccess(int id) {

                this.id = id;

                }

               

                @Override

                public TimeoutException call() throws Exception {

                TimeoutException failure = null;

               

                try {

                startTransaction();

               

                if (id % 2 == 0) {

                cache1.put("key1","value1");

                cache2.put("key2","value2");

                } else {

                cache2.put("key2","value2");

                cache1.put("key1","value1");

                }

                } finally {

                long start = System.currentTimeMillis();

                try {

                endTransaction();

                } catch (TimeoutException e) {

                failure = e;

                logger.error("end tx failed",e);

                } finally {

                logger.debug("tx ending {} millis",System.currentTimeMillis()-start);

                }

               

                return failure;

                }

                }

                }

               

                int CUNCURRENCY_LEVEL = 10;

                int TESTS_COUNT = 3;

               

                List<CacheAccess> tasks = new ArrayList<CacheAccess>();

               

                for (int i=0;i<CUNCURRENCY_LEVEL;i++)

                tasks.add(new CacheAccess(i));

               

                endTransaction();

               

                for (int i=0;i<TESTS_COUNT;i++) { //test 10 times

                startTransaction();

                cache1.clear();

                cache2.clear();

                endTransaction();

               

                logger.debug("invoking test #{}",i);

               

                List<Future<TimeoutException>> results = service.invokeAll(tasks);

               

                for (int j=0;j<results.size();j++) {

                TimeoutException failure = (TimeoutException)results.get(j).get();

                assertNull(failure);

                }

                }

                } catch (Error e) {

                logger.error("test fail {}",e.toString());

                throw e;

                }

                }

               

              }

               

              infinispan.xml

              <?xml version="1.0" encoding="UTF-8"?>

              <infinispan xmlns="urn:infinispan:config:6.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >

                <global>

                <globalJmxStatistics enabled="true" cacheManagerName="testDeadlock"/>

                </global>

                <default>

                <clustering mode="local"/>

                <transaction transactionMode="TRANSACTIONAL" transactionManagerLookupClass="org.infinispan.transaction.lookup.GenericTransactionManagerLookup" lockingMode="OPTIMISTIC" />

                <locking lockAcquisitionTimeout="20000"/>

                </default>

                <namedCache name="cache1"/>

                <namedCache name="cache2"/>     

              </infinispan>

               

               

              Thanks

              Ofer

              • 4. Re: is there 'Lock Reordering' for 'Optimistic locking' on 'Local' mode (ISPN 6.X)
                pruivo

                Hi Ofer,

                 

                The lock reordering is per-cache mechanism and it has no info about others caches locks. We have plans to improve the multi-cache transactions but it will probably only be available in the next major release.

                 

                In your version, you can solve the problem by acquiring the lock manually using pessimistic locking. In this case, it allows you to manually lock the keys using AdvancedCache.lock()). To enable it, change lockingMode to PESSIMISTIC.

                 

                Cheers,

                Pedro