7 Replies Latest reply on Mar 23, 2011 7:00 AM by nfilotto

    Are lock striping and Tx compatible?

    nfilotto

      Hi all,

       

      There is something that I cannot understand so far, I hope that someone can help me to understand. In order to reduce the foot print, I would like to set the parameter "useLockStriping" to "true", unfortunatelly it looks like it is nearly impossible to use it when we apply changes to our ISPN cache instance within a Tx without getting unpredictable deadlocks (aka TimeoutException "Unable to acquire lock after [xxx] on key [yyyy] for requestor [zzzz]! Lock held by [wwww]")

       

      When we don't use the lock striping, we know that if we don't sort the keys, we can face deadlocks, for example if the first thread does:

       

      tm.begin()
      // We acquire the lock on "A"
      cache.put("A","A");
      // We acquire the lock on "B"
      cache.put("B","B");
      // Here all the acquired locks will be released
      tm.commit();
      

       

      Then in second thread, it does:

       

      tm.begin()
      // We acquire the lock on "B"
      cache.put("B","B");
      // We acquire the lock on "A"
      cache.put("A","A");
      // Here all the acquired locks will be released
      tm.commit();
      

       

      We will then create a deadlock, to fix this issue we have to sort the keys within the Tx which can easily be done.

       

      In case of lock stripping it seems that we have no way to predict and avoid the deadlocks since the way the application get the locks is implementation-dependent so it cannot be predicted by definition. For example this following TestCase fails:

       

       

      public class TestDeadlock extends TestCase
      {
      
         /**
          * We get an unpredictable deadlock because apart if we know the implementation
          * of the ReentrantStripedLockContainer, we cannot know that the keys
          * 8 * "a" and 13 * "a" are mapped to the same lock and 11 * "a" and 14 * "a" are
          * both mapped to the same lock that is different from the first one.
          * In the use case below the keys are sorted but we still get a deadlock
          */
         public void testUseLockStripingTrueUnpredictableDeadLock() throws Exception
         {
            DefaultCacheManager manager = new DefaultCacheManager();
            manager.getDefaultConfiguration().setInvocationBatchingEnabled(true);
            manager.getDefaultConfiguration().setLockAcquisitionTimeout(1000);
            final Cache<String, String> cache = manager.getCache();
            final CountDownLatch startSignal = new CountDownLatch(1);
            final CountDownLatch endSignal = new CountDownLatch(2);
            final AtomicReference<Exception> exception = new AtomicReference<Exception>();
            new Thread()
            {
               /**
                * @see java.lang.Thread#run()
                */
               @Override
               public void run()
               {
                  try
                  {
                     cache.startBatch();
                     startSignal.await();
                     // 8 * "a"
                     cache.put("aaaaaaaa", "Value");
                     // 14 * "a"
                     cache.put("aaaaaaaaaaaaaa", "Value");
                     cache.endBatch(true);
                  }
                  catch (Exception e)
                  {
                     exception.set(e);
                  }
                  finally
                  {
                     endSignal.countDown();
                  }
               }
            }.start();
            new Thread()
            {
               /**
                * @see java.lang.Thread#run()
                */
               @Override
               public void run()
               {
                  try
                  {
                     cache.startBatch();
                     startSignal.await();
                     // 11 * "a"
                     cache.put("aaaaaaaaaaa", "Value2");
                     // 13 * "a"
                     cache.put("aaaaaaaaaaaaa", "Value2");
                     cache.endBatch(true);
                  }
                  catch (Exception e)
                  {
                     exception.set(e);
                  }
                  finally
                  {
                     endSignal.countDown();
                  }
               }
      
            }.start();
            startSignal.countDown();
            endSignal.await();
            assertNull(exception.get());
         }
      }
      

       

      My question is: Can you confirm that lock stripping and Tx are not compatible? If not what is the trick to use?

       

      Thank you in advance for your answers,

      BR,

      Nicolas

       

      The use case was incorrect

        • 1. Are lock striping and Tx compatible?
          mircea.markus
          Can you confirm that lock stripping and Tx are not compatible? If not what is the trick to use?

          Lock stripin increases the chance of lock colisions (e.g. the keys "k1" and "k2" might use the same underlying lock object), and hence a higher chance of deadlock to happen. There are some things you can do: e.g. use good hash fubnctions on your objects, increase the concurrencyLevel for locking, enable deadlock detection (or even better order the keys)

          • 2. Are lock striping and Tx compatible?
            nfilotto

            Thx for your answer. If I understand it right, having a better hash function and increasing the concurrencyLevel will only help to reduce the risk but the risk still exists. And enabling the deadlock detection will help to rollback only one Tx instead of two. In other words, we limit the risk and the side effects but we don't prevent the deadlocks, am I correct?

             

            Final questions, is-it really interesting to use the lock striping in term of memory footprint (and/or perfs?) ? What is the ratio compared to disabling it?

             

            Thank you in advance for your answer,

            Nicolas

            • 3. Are lock striping and Tx compatible?
              mircea.markus

              All the points I've mentioned are to reduce the chance, yes.

               

              Final questions, is-it really interesting to use the lock striping in term of memory footprint (and/or perfs?) ? What is the ratio compared to disabling it?

              I'm not aware of such data, perhaps others can help? If you run some tests would be great if you make this public.

              • 4. Are lock striping and Tx compatible?
                dan.berindei

                You can create your own instance of org.infinispan.util.concurrent.locks.containers.ReentrantStripedLockContainer and call hashToIndex(key) method to obtain the lock index that Infinispan will use for a key (provided you have the same concurrency level).

                 

                If you can sort your keys based on this lock index, then you would ensure that you always obtain the locks in the same order even with lock striping.

                • 5. Are lock striping and Tx compatible?
                  nfilotto

                  Thank you for your help, it could be indeed a potential workaround, the risk is that it would tie my code base to on an internal component of ISPN that could change from one version to another. So why not, but I still need to know if it's worth it

                  • 6. Re: Are lock striping and Tx compatible?
                    manik

                    Agreed that it is an implementation detail, however I don't see why this cannot be made into a public API of sorts.  E.g., cache.getAdvancedCache().getLockManager().getLockId(key) which would return an int.  Essentially by internally calling hashToIndex(key).

                     

                    If you think this is useful to you, create a feature request in JIRA and vote on it. 

                     

                    Regarding memory footprint, lock striping will prevent creating a lock per entry, which can be several hundred bytes each.  This can become big.

                    • 7. Re: Are lock striping and Tx compatible?
                      nfilotto

                      Hi Manik,

                       

                      Thanks for your proposal, it should indeed help to fix the issue if we sort our keys according to the lock id, here is the related JIRA https://issues.jboss.org/browse/ISPN-993. BTW, in case we disable the lock striping what should return this method? the key itself?